Countnest

AI crawlers explained: GPTBot, ClaudeBot, PerplexityBot and how to see them

By Adrian Ababei · Published · Facts checked 10 November 2026

Want these numbers for your own site? Start a 30-day free trial of Countnest, no card required, or see pricing.

Definitions in one minute

  • AI crawler: a bot run by an AI company that fetches web pages automatically, for training data or for an AI search index.
  • AI user agent (fetcher): a bot that fetches one page because a person asked an assistant something right now. It is not crawling the web on its own.
  • AI referral visit: a human who clicks a link in an assistant and lands on your site. This is a real visitor. See What is AI traffic?
  • Product token: a name you can use in robots.txt to set a preference, even though no bot visits under that name (for example Google-Extended).

The main AI bots, by company (checked 10 November 2026)

CompanyNameWhat it does, according to the company
OpenAIGPTBotCrawls content that may be used to train OpenAI's generative AI foundation models
OpenAIOAI-SearchBotSurfaces websites in ChatGPT's search features
OpenAIChatGPT-UserVisits a page when a user asks ChatGPT or a custom GPT something; not used for automatic crawling
AnthropicClaudeBotCollects web content that could contribute to model training
AnthropicClaude-SearchBotIndexes content to improve Claude's search results
AnthropicClaude-UserFetches pages when people ask Claude questions
PerplexityPerplexityBotSurfaces and links websites in Perplexity search results; Perplexity says it is not used to crawl content for AI foundation models
PerplexityPerplexity-UserVisits a page to answer a user's question
GoogleGoogle-ExtendedA robots.txt product token only. Google says it has no separate user agent; it controls whether content Google crawls may be used for Gemini training and grounding, and does not affect Google Search
AppleApplebot-ExtendedAlso a token, not a visiting bot. Apple says it "does not crawl webpages"; disallowing it opts content out of training Apple's foundation models

Countnest also recognises other self-identified AI bots, such as CCBot (Common Crawl), Bytespider, Meta-ExternalAgent, Amazonbot, DuckAssistBot and MistralAI-User.

Why your normal analytics can't see them

Almost all crawlers request the HTML of a page and stop there. They don't run JavaScript, so a script-based tool, whether Google Analytics, Countnest's tracker or any other, never fires for them. That is a good thing for your visitor numbers: bots don't inflate them. But it also means that to see crawler activity you need a record from the server side, where every request arrives, including bots.

Do AI crawlers send visitors back?

Much less often than search engines. Cloudflare publishes a "crawl-to-refer ratio" on Cloudflare Radar: HTML page requests from a platform's bots divided by HTML page visits that arrive with that platform in the referrer. For October 2026 (Radar data read on 10 November 2026), the ratios were roughly:

PlatformCrawls per referred visit, October 2026
Googleabout 3.8
Microsoftabout 40
Anthropicabout 490
OpenAIabout 800
Perplexityabout 4,020

Read these numbers with care. Cloudflare itself notes that traffic from native AI apps often carries no referrer, so the ratios "may overstate" the gap, "but it is unclear by how much". They are network-wide averages across sites behind Cloudflare, not a forecast for your site. They also change fast from month to month.

How to allow or block AI bots with robots.txt

robots.txt is a plain text file at the root of your site. Each company tells you to address its bots by name. A few examples:

# Allow AI search, but say no to training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Things worth knowing before you change it:

  1. Each setting is separate. OpenAI says blocking GPTBot does not remove you from ChatGPT search; that is controlled by OAI-SearchBot. Blocking OAI-SearchBot means your site is not shown in ChatGPT search answers, though it can still appear as a navigational link.
  2. User-triggered fetchers may not follow robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. Anthropic says Claude-User follows robots.txt.
  3. Changes take time. OpenAI and Perplexity both mention up to about 24 hours.
  4. Anthropic asks for robots.txt, not IP blocking. It says blocking its IP addresses may not work reliably, because it stops the bot from reading your robots.txt.
  5. Names can be faked. Google warns that user agent strings can be spoofed. OpenAI, Perplexity and Apple publish IP ranges you can check against if you need certainty.
  6. robots.txt is a request. Well-behaved bots follow it; it doesn't technically stop anyone.

A newer option is a general "no AI training" preference. The IETF's AI Preferences (aipref) working group is standardising a vocabulary for it. Anthropic says that since 30 October 2026 it has begun recognising such a signal, while rules that name ClaudeBot directly still take priority. Google says it intends to support the shared training preference signal in early 2027. Until it is widely supported, naming each bot is the reliable way.

Should you block them?

There is no single right answer. Some sensible trade-offs:

  • Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) help you appear and be cited in AI answers. Blocking them can reduce your visibility there.
  • Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) don't send visitors directly. Blocking them is a content and business choice, not a traffic one.
  • Check your own numbers first. Look at which bots read which pages, and whether the same assistants send you visitors.

How to see AI crawler visits in Countnest

  1. Open the AI traffic tab and find the AI crawlers card.
  2. Add the server-side snippet for your setup: the WordPress plugin, the Next.js middleware, a Cloudflare Worker, or a simple API call from any server. See the install docs. The tracker script alone can't see bots, for the reason above.
  3. Countnest logs only the bot name, the page path, the HTTP status and the time. No visitor data is involved, and crawler fetches never count toward your session quota.
  4. Compare the crawler table with the AI referrals in the same tab. Pages that are crawled a lot but rarely visited may need a clearer answer near the top. Our guide to measuring AI visitors explains the visitor side.

To be straight with you about the limits: Countnest recognises bots by the name in their user agent and does not check IP ranges, so a fake bot using a real name would be counted under that name. Bots that don't identify themselves are not shown at all.

Official sources

Questions and answers

Are AI crawlers counted as visitors in Countnest?

No. Crawler fetches are logged separately, shown in their own table and never added to visitor numbers or your session quota.

Does blocking GPTBot remove my site from ChatGPT search?

No, according to OpenAI. GPTBot is for training; ChatGPT search uses OAI-SearchBot, which you control with its own robots.txt rule.

Why don't I see GPTBot in Google Analytics?

Crawlers don't run JavaScript, so script-based analytics never records them. You need server-side logs or a server-side integration.

Is Google-Extended a separate crawler?

No. Google says Google-Extended has no separate user agent. It is a robots.txt token that controls whether content Google already crawls may be used for Gemini training and grounding. It does not affect Google Search.

Do AI crawlers send traffic back?

Some do, but far less than search engines. Cloudflare Radar's crawl-to-refer ratio for October 2026 was about 3.8 crawls per referral for Google, compared with roughly 490 for Anthropic, 800 for OpenAI and 4,020 for Perplexity. Cloudflare notes these may overstate the gap, because app traffic often carries no referrer.

See your own numbers in minutes

30-day free trial, no card required. Then from $14 a month. Cancel anytime.

Start free for 30 days

Sources

Facts checked on 10 November 2026.