Indexability and canonical checker

Why it matters

Google considers a page for indexing when it answers 2xx, though indexing is not certain even then. Google does not index an address that answers 4xx, and drops one that keeps answering 5xx.

How to check it

Run curl -IL with the address and read the status of the last answer. In Search Console, a live test of the final address in URL Inspection shows whether Google could fetch it.

Official documentation: How HTTP status codes affect Google’s crawlers

Why it matters

Google advises redirecting straight to the final address; failing that, ideally no more than 3 hops and fewer than 5. Its crawlers follow up to 10; Search Console reports a chain too long, or a loop, as a redirect error.

How to check it

Run curl -IL with the address: it prints each answer in turn, and only the last should be 200.

Official documentation: Redirects and Google Search

Why it matters

Google recommends HTTPS to protect visitors’ security and privacy, and when a page has both an http and an https address, Google prefers to index the https one.

How to check it

Open the page and check that its address starts with https and the browser shows no security warning. Search Console’s HTTPS report counts the indexed addresses still on http.

Official documentation: Understanding page experience in Google Search results

Why it matters

Google accepts a canonical link element only in the head. An element that does not belong there, such as an img or an iframe, ends the head early for Google, which ignores what follows.

How to check it

In the page source, check that the canonical sits inside the head, with no img, iframe or other body element above it.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google takes a canonical declaration as a signal that the address it names should become canonical. A value that is empty or is not an http or https address names no page to prefer.

How to check it

Read each canonical’s value in the page source, and any Link header with curl -I. Each should be a full address that opens the page.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google prefers https pages as canonical, and lists an https page whose canonical names the http version among the conflicting signals that can override that preference.

How to check it

In the page source, check that the canonical starts with https://. Search Console’s HTTPS report counts the indexed addresses still on http.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google lists the robots rules it supports and names others it ignores, such as noarchive. A word outside its list has no documented effect, and it may be a misspelling of the rule you meant.

How to check it

Read the robots meta tags in the page source and the X-Robots-Tag headers with curl -I, and compare each word with Google’s list.

Official documentation: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications

Why it matters

A canonical, in the head or a Link header, tells Google which address you prefer when a page can be reached at several. Google does not require one: without it, Google picks the version it judges best to show.

How to check it

View the page source and search for canonical. URL Inspection in Search Console shows the user-declared canonical and the one Google selected.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google supports relative canonical addresses but recommends full ones, because a relative address can cause problems later, for example if a test copy of the site gets crawled.

How to check it

In the page source, check that each canonical value starts with https://.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google advises against naming different canonical addresses for the same page, and warns that conflicting or multiple canonical tags may lead to unexpected results.

How to check it

Search the page source for canonical, and run curl -I with the address to read any Link header.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google Search shows the target of a permanent redirect in its results and takes the redirect as a signal that the target should be canonical. After a temporary redirect it shows the source address.

How to check it

Run curl -IL with the address and read the status of each redirect: 301 or 308 marks a permanent one.

Official documentation: Redirects and Google Search

Why it matters

Google treats a declared canonical as a hint. It chooses the canonical from several signals, and its search results usually point to the page it chose.

How to check it

URL Inspection in Search Console shows the user-declared canonical beside the one Google selected.

Official documentation: What is canonicalization

Why it matters

Google does not crawl or index content that robots.txt blocks. The address itself can still appear in search results, without a description, when other pages link to it.

How to check it

In Search Console, URL Inspection reports whether crawling is allowed for this address, and the robots.txt report shows the file Google last fetched.

Official documentation: How Google interprets the robots.txt specification

Why it matters

When Googlebot crawls a page and finds noindex, Google drops the page from its search results, even when other sites link to it. The rule none means noindex, nofollow.

How to check it

View the page source and search for noindex. URL Inspection in Search Console shows whether indexing is allowed.

Official documentation: Block Search indexing with noindex

Why it matters

The X-Robots-Tag header has the same effect as a robots meta tag, and it also covers files such as PDFs. When Googlebot finds noindex or none in it, Google drops the address from its search results.

How to check it

Run curl -I with the address and read any X-Robots-Tag lines. URL Inspection in Search Console shows whether indexing is allowed.

Official documentation: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications

Why it matters

A 4xx other than 429 means no crawl restrictions. After a 5xx, a 429 or a failed fetch, Google stops crawling for 12 hours, then uses its last good copy for up to 30 days. RFC 9309 says an unreachable file means everything is disallowed.

How to check it

Open the site’s /robots.txt in a browser, or run curl -I on it, and read the status. Search Console’s robots.txt report shows what Google last fetched.

Official documentation: How Google interprets the robots.txt specification

Why it matters

RFC 9309, the robots.txt standard, tells a crawler to obey the group of rules that matches its name, or the * group when none does. A crawler that follows it does not fetch what its group forbids. Blocking one search engine can be deliberate.

How to check it

Open the site’s /robots.txt, find the group for Bingbot, or the * group when there is none, and read the rule shown.

Official documentation: RFC 9309: Robots Exclusion Protocol

Why it matters

A canonical asks Google to show the address it names. Google does not index an address that answers 4xx or redirects elsewhere, and drops one that keeps answering 5xx.

How to check it

Run curl -I with the canonical address: it should answer 200, with no Location header.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

A sitemap helps discovery only when it can be fetched and read. Search Console reports such failures as Couldn’t fetch or Has errors, and one file may hold at most 50,000 addresses and 50 MB uncompressed.

How to check it

Open each sitemap address in a browser and check that it lists your pages. Search Console’s Sitemaps report shows whether Google could read the sitemaps you submitted.

Official documentation: Build and submit a sitemap

Why it matters

Google reads robots.txt up to 500 KiB and ignores everything after that point, so rules further down have no effect on Google.

How to check it

Download /robots.txt and check its size in bytes. Search Console’s robots.txt report also shows the size of the file Google fetched.

Official documentation: How Google interprets the robots.txt specification

Why it matters

A sitemap helps search engines discover a site’s pages, though Google may not crawl or index everything it lists. Google says a site of about 500 pages or fewer, well linked internally, might not need one.

How to check it

Open the site’s /robots.txt and look for a Sitemap line. Search Console’s Sitemaps report lists the sitemaps you submitted, but not those found only through robots.txt.

Official documentation: Build and submit a sitemap

Why it matters

Google recommends listing in a sitemap the addresses you want in its search results, written in full. A page left out can still be discovered through links.

How to check it

Open the sitemap and search for the page’s address. URL Inspection in Search Console names the submitted or robots.txt sitemaps that point to it.

Official documentation: Build and submit a sitemap

Why it matters

The page named as canonical is the one you want shown. With noindex, Google drops it from its results. If robots.txt forbids Googlebot to fetch it, Google may index the address without its content.

How to check it

Inspect the canonical address with URL Inspection in Search Console: it shows whether crawling and indexing are allowed.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google must fetch a page to see its noindex. While robots.txt forbids Googlebot to fetch it, Google never sees the rule, and the page can still appear in search results, for example when other pages link to it.

How to check it

After changing robots.txt, URL Inspection in Search Console shows whether crawling is allowed and whether the page allows indexing.

Official documentation: Block Search indexing with noindex

Why it matters

When the page you name as canonical names another address as its own, the two pages state different preferences. Google treats a canonical as a hint and chooses among all the signals it collects.

How to check it

Open the address named as canonical, view its source, and check that its own canonical names itself.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

A canonical asks Google to show another address in place of this one, and noindex asks for this page not to be shown at all. For duplicates within a site, Google recommends the canonical over noindex.

How to check it

Check the page source, and its headers with curl -I, for noindex and for a canonical naming another address, and keep only one of the two.

Official documentation: How to specify a canonical URL with rel="canonical" and other methods

Why it matters

Google treats every address in a sitemap as a suggested canonical, and advises against listing an address in a sitemap while its page names another as canonical. A noindex page asks not to be shown.

How to check it

After the sitemap is updated, open it and search for the page’s address.

Official documentation: Build and submit a sitemap