robots.txt for AI crawlers: allow, block or choose
Definitions in one minute
- robots.txt: a plain text file at the root of your site (for example
https://example.com/robots.txt) that tells automated clients which paths they may fetch. It is standardised as the Robots Exclusion Protocol in RFC 9309 (September 2022). - User-agent token: the name you write after
User-agent:, such asGPTBot. Each company publishes its own tokens. - Training crawler: collects pages that may be used to train AI models.
- AI search crawler: builds the index an assistant uses to find and cite pages in its answers.
- User-triggered fetcher: reads one page because a person asked an assistant a question just now.
- Product token: a robots.txt name that controls how content is used, but never visits your site itself (Google-Extended and Applebot-Extended are examples).
If you want to know who these bots are and how often they send visitors back, start with AI crawlers explained. This guide is the practical part: what to write in the file.
The names to use, by purpose (checked 12 November 2026)
Each company describes its own bots. We list only names that the company itself documents, and link to the page we read.
| Company | Training | AI search | Person asked for it | Follows robots.txt? (company's words) |
|---|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | For ChatGPT-User: "robots.txt rules may not apply" |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | Anthropic says its bots honour robots.txt, Claude-User included |
| Perplexity | none listed (PerplexityBot "is not used to crawl content for AI foundation models") | PerplexityBot | Perplexity-User | Perplexity-User "generally ignores robots.txt rules" |
| Google-Extended (product token) | Googlebot (Search, including AI Overviews and AI Mode) | n/a here | Google's common crawlers "always obey robots.txt rules when crawling automatically" | |
| Apple | Applebot-Extended (product token) | Applebot (Spotlight, Siri, Safari) | n/a here | Applebot respects robots.txt; Applebot-Extended "does not crawl webpages" |
| Common Crawl | CCBot (an open web archive anyone can download) | n/a | n/a | Common Crawl documents User-agent: CCBot with Disallow: / to opt out |
| Meta | Meta-ExternalAgent (training foundation AI models, also "improving products by indexing content directly") | Meta-WebIndexer (Meta AI search, citing and linking) | Meta-ExternalFetcher | Meta-ExternalFetcher "may bypass robots.txt rules" |
Three details that trip people up:
- Google-Extended is not about Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode are part of Search: Google's AI features guide says Googlebot rules are the control there, and
nosnippet,data-nosnippet,max-snippetornoindexlimit what is shown. - Applebot-Extended is not Applebot. Disallowing Applebot-Extended opts your content out of training Apple's foundation models, and Apple says such pages "can still be included in search results". To keep content out of Apple's AI-generated "broad world knowledge" answers, Apple points to the
nosnippetmeta tag instead. - Each setting is separate. OpenAI says "Each setting is independent of the others": you can allow OAI-SearchBot and disallow GPTBot. Anthropic and Perplexity describe the same per-bot choice.
We left out bots whose owners don't publish a robots.txt page we could check, such as Bytespider. You can still name them in your file; just know you are relying on unverified behaviour.
How robots.txt rules are read
A few rules from RFC 9309 matter a lot for AI bots:
- A bot uses the group that names it, and only that group. If a group names GPTBot, GPTBot ignores your
User-agent: *group. So if*blocks/admin/, repeat that line inside every named group, or the named bot won't see it. - Several names can share one group. Put multiple
User-agent:lines on top of one set of rules. Matching of names is case-insensitive. - The most specific path wins.
Allow: /blog/beatsDisallow: /for pages under /blog/. If an allow and a disallow are equally specific, allow should win. - Status codes count. If robots.txt returns a 4xx error, crawlers may fetch anything. If it returns a 5xx server error, they must assume everything is disallowed. A broken robots.txt can quietly block every bot, search engines included.
- Caching. Crawlers should not use a cached copy for more than 24 hours. OpenAI, Perplexity and Meta each say changes can take about 24 hours to apply.
- It is a request, not a lock. RFC 9309 says the rules "are not a form of access authorization" and that the protocol "is not a substitute for valid content security measures". Anything listed in robots.txt is also public, so don't use it to hide private paths.
Three robots.txt files you can copy
Replace example.com with your domain and keep any rules you already have for private paths.
1. Allow everyone (including all AI bots)
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
An empty file, or no file at all, means the same thing to crawlers that follow the standard. Writing it out makes the choice explicit.
2. Say no to training, yes to AI search and user fetches
# Training: please don't
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /
# AI search and fetches a person asked for: welcome
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Meta-WebIndexer
Allow: /
# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
What this does, per each company: you stay eligible for ChatGPT search, Claude's search, Perplexity results and Meta AI citations, and you stay in Google Search and Apple search. You ask OpenAI, Anthropic, Google, Apple, Common Crawl and Meta not to use your pages for model training. Two side effects to accept: Google-Extended also covers grounding in Gemini Apps, and Meta says Meta-ExternalAgent is also used for "improving products by indexing content directly".
3. Block all the AI bots listed above
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
This keeps Googlebot, Bingbot and Applebot, so your normal search presence stays. That also means Google can still show your pages in AI Overviews and AI Mode, because those run on Googlebot's crawl. Blocking Googlebot itself would remove you from Google Search, which is rarely what anyone wants. And remember: ChatGPT-User, Perplexity-User and Meta-ExternalFetcher may still fetch a page when a person asks for it.
Allow, block or choose: the trade-offs
There is no single right answer. It is a content and business decision.
- Allowing AI search bots keeps you eligible to be found and cited in ChatGPT search, Claude, Perplexity and Meta AI. OpenAI and Perplexity both recommend allowing their search bots if you want to appear there. Those citations are the links that bring AI referral visits; see What is AI traffic?
- Blocking AI search bots removes you from those answers. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links". Anthropic says disabling Claude-SearchBot or Claude-User "may reduce your site's visibility".
- Blocking training bots tells those companies not to use your future content for training. It doesn't send or cost you visitors directly, because training crawlers don't put links in answers. It also doesn't remove content that was collected before.
- Blocking user-triggered fetchers stops some assistants from reading your page when someone asks about it. Where the company says the fetcher may not follow robots.txt, the rule may not have much effect.
A sensible order of work: measure first, then decide. Look at which bots read which pages and whether the same assistants send you visitors, then change one thing at a time and watch the result for a couple of weeks.
How to test your robots.txt
The quickest check is our free AI readiness checker: enter your domain and it reads your robots.txt the way each AI crawler does, then checks llms.txt and your sitemap. To do it by hand:
- Open it in a browser. Visit
https://yourdomain/robots.txt. It should load as plain text with status 200. Check every host separately: a subdomain such as shop.example.com has its own file. If a CDN, host or plugin generates the file, check that the live file is the one you meant. - Read it as a bot would. For each bot you care about, find the group that names it. If none does, the
*group applies. Check that the group has every rule you intended, including private paths. - Use Google's tools for Google. The robots.txt report in Search Console shows the file Google fetched, parsing errors and warnings, and lets you request a recrawl after an urgent fix. The URL Inspection tool tests a single URL. Developers can run Google's open-source robots.txt parser locally. These tell you how Google reads the file, not how every other company does.
- Give it a day. Changes can take about 24 hours to reach OpenAI, Perplexity and Meta, and crawlers may cache your file for up to a day.
- Watch the bots. Bots that follow your rules should stop fetching blocked pages. Your server logs, or a crawler log such as Countnest's, show whether that actually happened.
Anthropic adds one warning: blocking its IP addresses instead of using robots.txt "may not work correctly or persistently guarantee an opt-out", because it stops the bot from reading your robots.txt. Bot names can also be faked, so if you need certainty, check requests against the IP lists OpenAI, Perplexity, Apple and Common Crawl publish.
A note on law and copyright
This section is general information, not legal advice. In the EU, the copyright exception for text and data mining in Article 4 of Directive (EU) 2019/790 applies only if rightholders have not reserved their rights "in an appropriate manner, such as machine-readable means in the case of content made publicly available online". Whether a given robots.txt rule is such a reservation for a given use, and how this works outside the EU, depends on national law and the courts. If it matters for your business, ask a lawyer who knows your situation. Our disclaimer explains how we write about rules like these.
A newer, more general signal is on the way. The IETF's AI Preferences (aipref) working group is standardising a shared vocabulary for usage preferences such as "no AI training". Anthropic says that since 30 October 2026 it has begun recognising such a signal, with rules that name ClaudeBot taking priority. Google says it intends to support the shared training preference signal, written with user-agent: __usage_ai-training, in early 2027. Until support is broad, naming each bot remains the dependable way.
How to see the effect in Countnest
Crawlers don't run JavaScript, so no script-based analytics tool (Countnest's tracker, Google Analytics or any other) sees them. Countnest shows them when your server reports them:
- Add the server-side snippet. Use the WordPress plugin, the Next.js middleware, a Cloudflare Worker or a simple API call from any server. See the install docs and the WordPress page.
- AI traffic tab, AI crawlers card. See which bots fetched your site, how many fetches and how many pages, plus your most crawled pages. Countnest logs only the bot name, the page path, the HTTP status and the time, and crawler fetches never count toward your session quota.
- Crawled vs. cited. The same card compares crawler fetches with AI visits per page, so you can see whether pages AI bots read also get visitors from AI answers.
- GEO tab. "AI crawler activity" adds when each bot was last seen, which is a quick way to check that a bot you blocked has stopped. "AI visits by platform" and "Pages AI sends visitors to" show the human side: real people clicking through from ChatGPT, Perplexity, Gemini, Claude and others.
- Health tab. The site check warns if your robots.txt is missing or unreachable.
A before-and-after check: note each bot's fetches for two weeks, change robots.txt, then compare the next two weeks with your AI visits by platform. If you block training bots and keep search bots, your AI referral visits should not drop because of that change. The guides on tracking ChatGPT traffic, tracking Perplexity traffic and GA4's AI Assistant channel explain the visitor side. A product overview is on AI traffic analytics.
Honest limits: Countnest recognises bots by the name in their user agent and does not check IP ranges, so a fake bot using a real name is counted under that name. It lists Claude-SearchBot and Claude-User separately, and Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer separately. Bots that don't identify themselves are not shown at all.
What we do on countnest.com
Our own robots.txt allows AI search and training bots on public pages and keeps private app paths off limits for everyone. We want Countnest to be found and cited in AI answers, and our pages are written to be quoted. Your choice can be different, and that's fine.
Official sources
- IETF: RFC 9309, Robots Exclusion Protocol and AI Preferences (aipref) working group
- OpenAI: Overview of OpenAI crawlers
- Anthropic: How site owners can block or limit Anthropic's bots
- Perplexity: Perplexity crawlers
- Google: Google's common crawlers, including Google-Extended, AI features and your website and robots.txt report
- Apple: About Applebot
- Common Crawl: CCBot
- Meta: Meta web crawlers
- EU: Directive (EU) 2019/790 on copyright in the Digital Single Market
Questions and answers
Can I block AI training but still appear in AI answers?
Yes, for the companies that separate the two. Disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent, and allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer. OpenAI, Anthropic and Perplexity each say their robots.txt settings work independently.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search or ranking. AI Overviews and AI Mode are part of Search and follow Googlebot rules; to limit what is shown, Google points to nosnippet, data-nosnippet, max-snippet or noindex.
Why do I still see ChatGPT-User or Perplexity-User after blocking them?
Because a person asked for the page. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta says Meta-ExternalFetcher may bypass it. Anthropic says Claude-User follows robots.txt.
How long does a robots.txt change take to work?
Usually within a day. OpenAI, Perplexity and Meta each mention about 24 hours, and RFC 9309 says crawlers should not use a cached copy for more than 24 hours.
Is robots.txt legally binding?
This is general information, not legal advice. RFC 9309 says robots.txt rules are not a form of access authorization; well-behaved bots follow them voluntarily. In the EU, Article 4 of Directive (EU) 2019/790 lets rightholders reserve text and data mining rights by machine-readable means for content online. How that applies to you depends on national law, so ask a lawyer if it matters.
Should I block AI bots by IP address instead?
Usually not as your first step. Anthropic says IP blocking may not work reliably because it stops its bot from reading robots.txt. Use robots.txt for your preferences and the published IP lists to verify that a bot is genuine.
Will blocking AI crawlers change my visitor numbers?
Not directly. Crawlers don't run JavaScript, so they were never counted as visitors. Blocking AI search bots can reduce future visits from AI answers; blocking training bots shouldn't change them.
30-day free trial, no card required. Then from $14 a month. Cancel anytime.
Start free for 30 daysSources
- IETF: RFC 9309, Robots Exclusion Protocol (September 2022)
- OpenAI: Overview of OpenAI crawlers (read 12 Nov 2026)
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (read 12 Nov 2026)
- Perplexity: Perplexity crawlers (read 12 Nov 2026)
- Google: Google's common crawlers (last updated 30 Oct 2026)
- Google Search Central: AI features and your website (last updated 10 Dec 2025)
- Search Console Help: robots.txt report
- Google: open-source robots.txt parser
- Apple: About Applebot (published 4 Sep 2026)
- Common Crawl: CCBot
- Meta for Developers: Meta web crawlers (read 12 Nov 2026)
- EUR-Lex: Directive (EU) 2019/790, Article 4 (text and data mining)
- IETF: AI Preferences (aipref) working group