TULUS
Start a project
TULUS

AI crawler checker

Which AI crawlers may fetch this page under its robots.txt, and what is each of them for?

Baca dalam bahasa Indonesia

For example yourbusiness.com/services

The checker runs in your browser with JavaScript, which is off here. To have a person look at the page instead: Request a Search Presence Review

The report is kept for 24 hours, and only this browser can open it. How scans are handled

Want a person to read it? Request a Search Presence Review Know what needs changing? Discuss your project

What it checks

The checker reads the site’s robots.txt for every AI crawler on its list, each as its operator documents it, then the page’s own robots meta tags and headers: 13 checks in three groups. The report names each one with what it found, why it matters and how to check it yourself.

All 13 checks

Surface

How the address answers, and where it leads.

  • The answer the address gives If the page answers with an error, Google can drop it from its search results.
  • Redirects before the page Each redirect is a detour on the way to the page, and too many make Google give up.

Structure

The tags and headers a crawler reads.

  • What the page tells AI features in search What this page’s own tags let AI features in Google and Bing use.

Presence

Permission to crawl and index, and the sitemap.

  • The site’s robots.txt file robots.txt is the file of rules for search engines; if it answers with an error, Google can pause crawling the whole site.
  • The size of robots.txt Google reads only the first 500 KB of robots.txt and ignores the rest.
  • robots.txt for Googlebot If robots.txt blocks Google’s crawler, Google cannot read this page.
  • robots.txt for Bingbot If robots.txt blocks Bing’s crawler, Bing cannot read this page.
  • The robots meta tag A noindex tag asks Google not to show the page in its search results at all.
  • The X-Robots-Tag header A noindex in the page’s headers asks Google not to show it in its search results at all.
  • robots.txt for AI search and answer crawlers If AI search crawlers are blocked, AI services such as ChatGPT search may leave your pages out of their answers.
  • robots.txt for AI model training Whether AI companies may use your pages to train their models is your choice, made in robots.txt.
  • robots.txt for fetches a person asks for AI assistants fetch a page when someone asks them to, and robots.txt may not stop them.
  • Names of AI crawlers in robots.txt A misspelt or retired crawler name in robots.txt matches no crawler, so its rules apply to no one.

Example

A real check of tulus.studio, made on 16 October 2026.

The example report

Report: https://tulus.studio/

Checked
16 October 2026, 01:13 WIB

1 page inspected, with its site’s robots.txt. 9 applicable checks ran.

Nothing to fix.

The checks in each group

  • Surface:

    2 passed

    How the address answers, and where it leads.

  • Structure:

    No check applied

    The tags and headers a crawler reads.

  • Presence:

    7 passed

    Permission to crawl and index, and the sitemap.

Each piece of a ring is one check.

  • Passed
  • Warning
  • Error
Who may fetch this page

robots.txt https://tulus.studio/robots.txt answered 200

Found, with 1 group of rules.

  • Search and answers None kept out
  • Model training None kept out
  • Fetches a person asked for None kept out
Crawler robots.txt says The lines that decide
Search and answers None kept out Crawlers for search results and answers that show or link the pages they come from.
Claude-SearchBot Anthropic Crawls the web to improve the quality, relevance and accuracy of search results for users. Anthropic’s page on Claude-SearchBot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Applebot Apple Crawls for search in Spotlight, Siri and Safari, and what it crawls may also help train Apple’s AI models. Apple’s page on Applebot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
DuckAssistBot DuckDuckGo Crawls pages in real time for DuckDuckGo’s AI-assisted answers, which cite their sources. DuckDuckGo’s page on DuckAssistBot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Meta-WebIndexer Meta Crawls the web to improve the quality of Meta AI’s search results. Meta’s page on Meta-WebIndexer robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
MistralAI-Index Mistral AI Crawls the web to index content for Mistral’s search, which helps answer people’s questions in Vibe. Mistral AI’s page on MistralAI-Index robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
OAI-SearchBot OpenAI Used to show websites in the search results of ChatGPT’s search features. OpenAI’s page on OAI-SearchBot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
PerplexityBot Perplexity Used to show and link websites in Perplexity’s search results. Perplexity’s page on PerplexityBot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Googlebot Google Crawls for Google Search and all its features, AI Overviews and AI Mode among them. Google’s page on Googlebot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Bingbot Microsoft Bing’s standard crawler, whose crawling and indexing Microsoft says Copilot also relies on. Microsoft’s page on Bingbot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Model training None kept out Crawlers that collect content to train AI models, and tokens that say whether pages another crawler fetched may be used for it.
Amazonbot Amazon Used to improve Amazon’s products and services, and may be used to train Amazon’s AI models. Amazon’s page on Amazonbot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
ClaudeBot Anthropic Collects web content that could help train Anthropic’s generative AI models. Anthropic’s page on ClaudeBot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
CCBot Common Crawl Builds Common Crawl’s open repository of web data, a primary source of training data for many large language models. Common Crawl’s page on CCBot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Meta-ExternalAgent Meta Crawls the web for uses such as training AI models or improving products by indexing content directly. Meta’s page on Meta-ExternalAgent robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
MistralAI-Training Mistral AI Crawls web content to help build datasets for training Mistral’s generative AI models. Mistral AI’s page on MistralAI-Training robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
GPTBot OpenAI Crawls content that may be used to train OpenAI’s generative AI models. OpenAI’s page on GPTBot robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Applebot-Extended Apple Tells Apple whether content Applebot crawled may be used to train Apple’s foundation models. Apple’s page on Applebot-Extended Fetches nothing: its rules govern the use of pages another crawler fetched. robots.txt says: May be used The lines that decide: line 1 User-agent: * line 2 Allow: /
Google-Extended Google Tells Google whether content it crawls may train Gemini models and ground answers in Gemini Apps and Vertex AI. Google’s page on Google-Extended Fetches nothing: its rules govern the use of pages another crawler fetched. robots.txt says: May be used The lines that decide: line 1 User-agent: * line 2 Allow: /
Fetches a person asked for None kept out Fetches made because a person asked an AI assistant to open a page.
Amzn-User Amazon May fetch up-to-date information from the web when a person asks Alexa a question that needs it. Amazon’s page on Amzn-User Amazon says robots.txt may not apply to it. robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Claude-User Anthropic May access a website when a person asks Claude a question. Anthropic’s page on Claude-User robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Meta-ExternalFetcher Meta Fetches individual links at a person’s request, including to help AI navigate websites and complete tasks. Meta’s page on Meta-ExternalFetcher Meta says robots.txt may not apply to it. robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
MistralAI-User Mistral AI May visit a web page to help answer a person’s question in Mistral’s Vibe. Mistral AI’s page on MistralAI-User robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
ChatGPT-User OpenAI May visit a web page when a person asks ChatGPT or a custom GPT a question. OpenAI’s page on ChatGPT-User OpenAI says robots.txt may not apply to it. robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /
Perplexity-User Perplexity May visit a web page to help answer a person’s question in Perplexity. Perplexity’s page on Perplexity-User Perplexity says it generally ignores robots.txt. robots.txt says: May fetch The lines that decide: line 1 User-agent: * line 2 Allow: /

Each crawler as its operator describes it, read on 15 October 2026.

Surface

2 of 2 applicable checks passed

How the address answers, and where it leads.

2 checks with nothing to fix
The answer the address gives

Passed Observed

The address, after any redirects, answered with a success status (2xx).

More about this check
Why it matters
Google considers a page for indexing when it answers 2xx, though indexing is not certain even then. Google does not index an address that answers 4xx, and drops one that keeps answering 5xx.
How to check it
Run curl -IL with the address and read the status of the last answer. In Search Console, a live test of the final address in URL Inspection shows whether Google could fetch it.

Official documentation: How HTTP status codes affect Google’s crawlers

Evidence

  1. Answer https://tulus.studio/ 

    200 OK
Redirects before the page

Passed Observed

The address led to the page directly or through no more than three redirects.

More about this check
Why it matters
Google advises redirecting straight to the final address; failing that, ideally no more than 3 hops and fewer than 5. Its crawlers follow up to 10; Search Console reports a chain too long, or a loop, as a redirect error.
How to check it
Run curl -IL with the address: it prints each answer in turn, and only the last should be 200.

Official documentation: Redirects and Google Search

Evidence

  1. Answer https://tulus.studio/ 

    200 OK

Structure

No check of this kind was performed by this tool on this page.

The tags and headers a crawler reads.

1 check with nothing to fix
What the page tells AI features in search

Not applicable Observed

The page’s robots meta tags and X-Robots-Tag headers carry none of the directives Google or Microsoft document for the AI features of their search.

More about this check
Why it matters
Google says nosnippet keeps a page’s content from being used as a direct input for AI Overviews and AI Mode and removes its snippet from every search result, and max-snippet limits both. Microsoft says noarchive keeps content out of Copilot’s responses and nocache limits Copilot to the address, title and snippet.
How to check it
View the page source for its robots meta tags, and run curl -I with the address to read any X-Robots-Tag lines. In Search Console, URL Inspection shows the HTML Googlebot received.

Official documentation: AI features and your website

Evidence

Nothing to show: the finding rests on what is absent from the page.

Presence

7 of 7 applicable checks passed

Permission to crawl and index, and the sitemap.

10 checks with nothing to fix
robots.txt for AI model training

Information Observed

robots.txt was checked for 8 AI training crawlers and control tokens, to see which may fetch or use this address and which it keeps out.

More about this check
Why it matters
Whether a site’s content trains AI models is the owner’s choice, and each of these operators documents robots.txt as a way to make it. Google-Extended and Applebot-Extended fetch nothing: they govern how content Google and Apple crawl may be used, and both say keeping them out does not remove a page from their search results.
How to check it
Open the site’s /robots.txt and find the group that names each crawler or token, or the * group when none does. Each operator’s page, linked beside its row, gives the name to use.

Official documentation: RFC 9309: Robots Exclusion Protocol

Evidence

  1. robots.txt https://tulus.studio/robots.txt line 1 

    User-agent: *

  2. robots.txt https://tulus.studio/robots.txt line 2 

    Allow: /
robots.txt for fetches a person asks for

Information Observed

robots.txt was checked for 6 agents that fetch a page when a person asks an AI assistant to. For these, their operators say robots.txt may not apply: Amzn-User, Meta-ExternalFetcher, ChatGPT-User, Perplexity-User.

More about this check
Why it matters
These agents fetch a page because a person asked an AI assistant something, not to crawl the web. Some of their operators say robots.txt may not apply to such fetches, so a Disallow rule may not keep those agents out.
How to check it
Open the site’s /robots.txt and find the group that names each agent, or the * group when none does, and read the rule shown.

Official documentation: RFC 9309: Robots Exclusion Protocol

Evidence

  1. robots.txt https://tulus.studio/robots.txt line 1 

    User-agent: *

  2. robots.txt https://tulus.studio/robots.txt line 2 

    Allow: /
robots.txt for Googlebot

Passed Observed

Nothing in robots.txt forbids Googlebot to fetch this address.

More about this check
Why it matters
Google does not crawl or index content that robots.txt blocks. The address itself can still appear in search results, without a description, when other pages link to it.
How to check it
In Search Console, URL Inspection reports whether crawling is allowed for this address, and the robots.txt report shows the file Google last fetched.

Official documentation: How Google interprets the robots.txt specification

Evidence

  1. robots.txt https://tulus.studio/robots.txt line 1 

    User-agent: *

  2. robots.txt https://tulus.studio/robots.txt line 2 

    Allow: /
The robots meta tag

Passed Observed

No robots or googlebot meta tag in the page carries noindex or none.

More about this check
Why it matters
When Googlebot crawls a page and finds noindex, Google drops the page from its search results, even when other sites link to it. The rule none means noindex, nofollow.
How to check it
View the page source and search for noindex. URL Inspection in Search Console shows whether indexing is allowed.

Official documentation: Block Search indexing with noindex

Evidence

  1. HTML https://tulus.studio/ line 1, column 538 

    <meta name="robots" content="index, follow, max-image-preview:large">
The X-Robots-Tag header

Passed Observed

No X-Robots-Tag header in the answer carries noindex or none for all crawlers or for Googlebot.

More about this check
Why it matters
The X-Robots-Tag header has the same effect as a robots meta tag, and it also covers files such as PDFs. When Googlebot finds noindex or none in it, Google drops the address from its search results.
How to check it
Run curl -I with the address and read any X-Robots-Tag lines. URL Inspection in Search Console shows whether indexing is allowed.

Official documentation: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications

Evidence

Nothing to show: the finding rests on what is absent from the page.

The site’s robots.txt file

Passed Observed

The site serves a robots.txt file, and this scan read it.

More about this check
Why it matters
A 4xx other than 429 means no crawl restrictions. After a 5xx, a 429 or a failed fetch, Google stops crawling for 12 hours, then uses its last good copy for up to 30 days. RFC 9309 says an unreachable file means everything is disallowed.
How to check it
Open the site’s /robots.txt in a browser, or run curl -I on it, and read the status. Search Console’s robots.txt report shows what Google last fetched.

Official documentation: How Google interprets the robots.txt specification

Evidence

  1. Answer https://tulus.studio/robots.txt 

    200 OK, 88 bytes, 1 group
robots.txt for Bingbot

Passed Observed

Nothing in robots.txt forbids Bingbot to fetch this address.

More about this check
Why it matters
RFC 9309, the robots.txt standard, tells a crawler to obey the group of rules that matches its name, or the * group when none does. A crawler that follows it does not fetch what its group forbids. Blocking one search engine can be deliberate.
How to check it
Open the site’s /robots.txt, find the group for Bingbot, or the * group when there is none, and read the rule shown.

Official documentation: RFC 9309: Robots Exclusion Protocol

Evidence

  1. robots.txt https://tulus.studio/robots.txt line 1 

    User-agent: *

  2. robots.txt https://tulus.studio/robots.txt line 2 

    Allow: /
robots.txt for AI search and answer crawlers

Passed Observed

Nothing in robots.txt keeps any of the 7 AI search and answer crawlers checked from fetching this address.

More about this check
Why it matters
These crawlers fetch pages for the search results and answers of AI services, and OpenAI, for one, says a site that keeps OAI-SearchBot out is not shown in ChatGPT’s search answers, except as a navigational link. Being allowed to fetch a page is not the same as being shown.
How to check it
Open the site’s /robots.txt, find the group that names each crawler listed, or the * group when none does, and read the rule shown. Applebot follows the group for Googlebot when no group names it.

Official documentation: RFC 9309: Robots Exclusion Protocol

Evidence

  1. robots.txt https://tulus.studio/robots.txt line 1 

    User-agent: *

  2. robots.txt https://tulus.studio/robots.txt line 2 

    Allow: /
The size of robots.txt

Passed Observed

robots.txt is within Google’s limit of 500 KiB.

More about this check
Why it matters
Google reads robots.txt up to 500 KiB and ignores everything after that point, so rules further down have no effect on Google.
How to check it
Download /robots.txt and check its size in bytes. Search Console’s robots.txt report also shows the size of the file Google fetched.

Official documentation: How Google interprets the robots.txt specification

Evidence

  1. Answer https://tulus.studio/robots.txt 

    88 bytes
Names of AI crawlers in robots.txt

Not applicable Observed

robots.txt names no AI crawler, or there is no robots.txt, so there is no name to check.

More about this check
Why it matters
A crawler obeys the group whose User-agent line matches its name, whatever the case, and the * group when none does. A misspelt name, or one its operator no longer uses, matches no crawler, so the crawler it was meant for does not follow the rules under it.
How to check it
Open the site’s /robots.txt, go to each line listed, and compare the name with the token on its operator’s page.

Official documentation: RFC 9309: Robots Exclusion Protocol

Evidence

Nothing to show: the finding rests on what is absent from the page.

Checked in 0.7 s with 2 requests, 18 kB received.

What this check cannot see

Two things about this page stay out of a public check’s reach:

  • Whether an AI system shows or cites the page. robots.txt only says who may fetch it, and being fetched is not being shown.
  • Which crawlers actually came, and whether a firewall or a CDN turns some away whatever robots.txt says. The site’s own logs and settings show that.

Everything a public scan cannot see

Official documentation

The pages each check is based on.

  1. How HTTP status codes affect Google’s crawlers
  2. Redirects and Google Search
  3. How Google interprets the robots.txt specification
  4. RFC 9309: Robots Exclusion Protocol
  5. Block Search indexing with noindex
  6. Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
  7. AI features and your website
  8. Anthropic’s page on Claude-SearchBot, ClaudeBot, Claude-User
  9. Apple’s page on Applebot, Applebot-Extended
  10. DuckDuckGo’s page on DuckAssistBot
  11. Meta’s page on Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher
  12. Mistral AI’s page on MistralAI-Index, MistralAI-Training, MistralAI-User
  13. OpenAI’s page on OAI-SearchBot, GPTBot, ChatGPT-User
  14. Perplexity’s page on PerplexityBot, Perplexity-User
  15. Microsoft’s page on Bingbot
  16. Amazon’s page on Amazonbot, Amzn-User
  17. Common Crawl’s page on CCBot
  18. Google’s page on Google-Extended

Related tools