Guides

Guide · Google and AI tools

Why your AI-built app isn’t on Google, and how to see what Google sees

Usually it’s one of a few things your site tells Google without you knowing: “don’t list me”, “don’t come in”, or “here’s an empty page”. Each one is visible from outside, so you can check it in a few minutes, and Google gives you a free tool to see your page the way it does.

Why it matters

Google has to crawl your page, render its JavaScript and then index it, and the rendering step waits in a queue that “can take longer” than a few seconds (Google Search Central). If one small tag says “don’t list me”, none of that happens. Google also says it may take months to revisit a page on its own, so a leftover from a preview build can quietly cost you months (Google Search Central).

What keeps an app off Google

A noindex tag or headerA <meta name="robots" content="noindex"> tag, or an X-Robots-Tag: noindex header, often left over from a preview build.Fix now
robots.txt blocks everythingUser-agent: * with Disallow: / shuts every search engine out of your whole site.Fix now
An empty page before JavaScriptGoogle can render it, later; most AI crawlers can’t (see that guide).Worth fixing
A canonical link to another addressPointing at a preview or platform address tells Google to rank that one instead.Worth fixing
Made-up pages answer “200 OK”Google can fill up on junk pages that all look like your home page.Worth fixing
No sitemapNot fatal, but Google finds new pages more slowly.Worth fixing

See it yourself

  1. Search Google for site:yourapp.com (your own address). No results means Google hasn’t indexed any of your pages yet.
  2. Open your site, press Ctrl+U (Cmd+Option+U on a Mac) and search the source for noindex. For the header, open your browser’s developer tools, go to Network, reload, click the first request and look for X-Robots-Tag under Response headers.
  3. Open yourapp.com/robots.txt. It should be plain text, not your app’s page, and it shouldn’t have Disallow: / under User-agent: *.
  4. Open yourapp.com/sitemap.xml, then an address that can’t exist, like yourapp.com/no-such-page-123. In the Network tab, that one should say 404, not 200.
  5. In Google Search Console, use URL Inspection. It shows whether Google has your page, lets you test the live address, and has a Request indexing button (with a daily limit) (Google’s help).

Fix it

Remove the noindex from your real domain. Keep it only on preview addresses. This is the same prompt our checker gives you:

Paste this into Lovable, Bolt, Cursor or Claude Code

My site tells search engines not to index it (noindex). Find where the noindex robots meta tag or X-Robots-Tag header is set and remove it for my production domain. Keep it only on preview or staging addresses.

Then ask Google to come back with Request indexing in URL Inspection, instead of waiting for it to revisit.

Open your robots.txt. If it blocks the whole site, use this one:

Paste this into Lovable, Bolt, Cursor or Claude Code

My robots.txt blocks all search engines with "Disallow: /". Change it so my public pages can be crawled, and only disallow private areas like /admin or /api if I have them.

Don’t use robots.txt to hide a page from Google. Google says it isn’t a way to keep a page out: a blocked address can still be listed, and Google can’t see a noindex on a page it isn’t allowed to read (Google Search Central).

The rest, in words:

  • Sitemap: serve one at /sitemap.xml listing your public pages, add a Sitemap: line to robots.txt, and submit it in Search Console’s Sitemaps report. Google calls it a hint, not a promise (Google Search Central).
  • Canonical: on each page, a <link rel="canonical"> pointing at that page on your main domain, never a preview address. Google treats it as a strong signal (Google Search Central).
  • Real 404s: addresses that don’t exist should answer 404. If your app sends every address to the same page, Google suggests a JavaScript redirect to a page that answers 404, or a noindex on the not-found page (Google Search Central).
  • An empty page: pre-render your public pages. The AI-assistants guide explains how.

Ready-made fix prompts for these four are part of Pro, coming soon. Every problem and what we found stays free.

What this can’t tell you

We check the address you give, its robots.txt and its sitemap, and one made-up address for the 404 test. We can’t see your Search Console, whether Google has indexed you, or where you rank: fixing all of these makes you listable, not first. We check that a sitemap exists, not that it lists the right pages. If your host serves Google a pre-rendered copy, as Lovable does for older apps, we can’t see that copy either.

Related

Sources