Google not indexing your pages: a debugging checklist

When a page won't show up in Google, the temptation is to hit "Request indexing" again and wait. Treat it like any other bug instead. Get the exact error, reproduce it locally, fix the cause, then verify.

This checklist is written for developers who own the code and the server. Most of the checks are a single curl command.

1. Read the exact status first

Open Search Console, paste the full URL into the bar at the top and read the result of the URL Inspection tool. Write down three things:

• the status under "Page indexing"
• the last crawl date
• the canonical Google selected

Then click "Test live URL". The live test fetches the page as it is now, which tells you whether you are debugging the current build or an old crawl.

For a site-wide view, the Page indexing report groups URLs by reason. Its example lists stop at 1,000 URLs, so drill into single pages with URL Inspection.

The status is your error message. Everything below maps a status to a check.

2. Status code: does the page return 200?

curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/blog/my-post | head -n 1

You want HTTP/2 200. Anything else explains a lot:

• 301/302: Search Console reports "Page with redirect". Google indexes the target. Update internal links and the sitemap to the final URL and stop submitting the old one.
• 401/403: often a WAF, bot protection or a staging password left on. Check your CDN rules for anything that challenges unknown bots.
• 404/410: the route is missing in production. Common after a framework migration.
• 5xx: check the Crawl Stats report and your logs for the crawl date Google shows. Google explains how each code is handled in HTTP status codes and network errors.

Spoofing the user agent only shows what your server returns to that string. Your firewall may treat the real Googlebot differently, so the live test in Search Console stays the final word.

3. Directives: is something saying noindex?

Two places to look. The response header:

curl -sI https://example.com/blog/my-post | grep -i x-robots-tag

And the HTML:

curl -s https://example.com/blog/my-post | grep -io '<meta[^>]*robots[^>]*>'

If either contains noindex, nothing else matters until it's gone. Usual suspects: an SEO plugin setting, a NODE_ENV check that ships the staging tag to production, or a reverse proxy adding a header to every route. See Block indexing with noindex.

4. robots.txt: is the path blocked?

curl -s https://example.com/robots.txt

Look for a Disallow that matches your path, including wildcard rules such as Disallow: /*?. The robots.txt report in Search Console shows the version Google last fetched. Remember that robots.txt controls crawling, not indexing: a blocked page can't be crawled, so Google can't see a noindex on it either. Reference: Introduction to robots.txt.

5. Canonical: are you pointing somewhere else?

curl -s https://example.com/blog/my-post | grep -io '<link[^>]*canonical[^>]*>'

The canonical should be the page itself, in its final form: same protocol, same host, same trailing-slash convention. Templating bugs are common here. A canonical hard-coded to the homepage, or built from a request header behind a proxy, turns every page into a duplicate. If Search Console says "Google chose different canonical than user", read Consolidate duplicate URLs.

6. Rendering: is the content in the HTML Google sees?

Compare raw HTML with what the browser shows:

curl -s https://example.com/blog/my-post | sed 's/<[^>]*>//g' | wc -w

A client-rendered page can return a few dozen words of shell. Google does render JavaScript, but rendering can fail on blocked API calls, scripts that time out or content that only loads after a click. Open "View crawled page" in the live test and check the rendered HTML for your main text. If it's missing, server-side rendering or static generation for that route is the reliable fix. Background: JavaScript SEO basics.

7. Discovery: can Google find the page at all?

If the URL isn't in the Page indexing report, Google doesn't know it exists yet. Check two things.

Internal links. At least one indexed page should link to it with a real anchor:

<!-- crawlable -->
<a href="/blog/my-post">My post</a>

<!-- not crawlable -->
<span onclick="router.push('/blog/my-post')">My post</span>

Google's rules are in crawlable links.

Sitemap. Confirm the final URL is listed:

curl -s https://example.com/sitemap.xml | grep -c "https://example.com/blog/my-post"

A result of 0 means your sitemap generator skipped it. Check the build filters and the lastmod value too.

8. The two "not indexed" statuses

These two cause the most confusion, because nothing is technically broken.

Discovered - currently not indexed. Google knows the URL but hasn't crawled it yet. Its help text says the crawl was rescheduled because it was "expected to overload the site". On real projects that usually means slow response times or thousands of low-value URLs (faceted filters, sort parameters, tag archives) competing for crawl time. Fix server latency and stop exposing parameter URLs that add nothing. Large sites should read the crawl budget guide.

Crawled - currently not indexed. Google fetched the page and decided not to keep it for now. No header or tag will change that. Look at the content: thin pages, templated pages that differ by one word, and near-copies of other URLs are the usual reasons. Merge or expand them, then add internal links from related pages. Google's helpful content guidance is the reference.

Soft 404 belongs here too. Your app returns 200 with an empty state ("No results", "Product not found"). Return a real 404 or 410, or put content on the page.

9. Verify, then wait on a schedule

After a fix, request indexing once in URL Inspection. Google states that repeated requests for the same URL don't speed anything up, and there is a daily quota.

Google's own timing: crawling a new page can take "a week or so", and "Validate fix" in the Page indexing report "typically takes up to about two weeks" (source). A practical loop:

The checklist, copy-paste version

[ ] URL Inspection status, crawl date and Google canonical noted
[ ] 200 OK, no redirect, no login, no WAF challenge
[ ] No noindex in X-Robots-Tag or meta robots
[ ] robots.txt does not match the path
[ ] Self-referencing canonical, exact final URL
[ ] Main content present in rendered HTML
[ ] Content is not a near-copy of another URL
[ ] Linked from an indexed page with a real <a href>
[ ] Listed in sitemap.xml under the final URL
[ ] Fast responses, no recent errors in Crawl Stats
[ ] Indexing requested once after the fix
[ ] Re-checked on day 7, 14 and 28

When an outside tool helps

An indexing service makes sense when Google hasn't discovered or crawled a page that is otherwise fine. It can't fix a noindex, a robots.txt block, a wrong canonical, an error page or thin content. Fix those first, or you pay to resubmit a page Google will reject again.

For pages on sites you don't own, such as backlinks, Search Console shows nothing. A bulk index checker is the only practical way to audit those.

Written by the IndexerPal team. We build tools for URL and backlink indexing. To check a list of URLs in Google at once, see the IndexerPal index checker.