Countnest

Cloudflare: AI crawler settings and robots.txt

By Countnest · Last updated

Check your site with Countnest's free ai readiness checker

No signup, no email. The checker reads only public files and gives a plain-English fix list.

Open the ai readiness checker

Cloudflare's AI controls

ControlWhat it does
AI Crawler and Agent access policiesChoose Allow, Block or Block on pages with ads for Search, Agent and Training behaviour. Training also has Disallow, which on its own only adds a robots.txt request
Bot Preference SyncWrites managed robots.txt directives from those policies. With sync on, Training set to Disallow also blocks training crawlers at the edge, except mixed-purpose crawlers used for both search and training
Block AI botsOlder name for this feature ("Configure AI bot policies"); you may still see it in guides and older dashboards
Content Signals PolicyComment text plus a Content-signal line in robots.txt stating search, ai-input and ai-train preferences

Two different things: asking and blocking

robots.txt and Content Signals ask crawlers to behave. Cloudflare's blocking rules enforce it at the edge, so a blocked crawler never reaches your server, whatever robots.txt says. If you block Search or Agent behaviour, AI search tools and assistants can't fetch your pages, so they can't read new or updated content (an AI tool may still mention what it already knows about you). Cloudflare also says Training actions apply to mixed-purpose crawlers used for both search and training, so a Block on Training can reach further than you expect.

A setup that keeps AI search visibility

  1. Security > Settings > AI Crawler and Agent access policies.
  2. Leave Search and Agent on Allow.
  3. Set Training to Disallow if you want to opt out of training.
  4. Turn on Bot Preference Sync and save.
  5. Reload /robots.txt: you should see Cloudflare's managed lines first, then your own rules.

Check the result

Cloudflare's managed content appears within seconds. Run the AI readiness checker on your domain: it reads the live robots.txt Cloudflare serves. Countnest customers can also see which AI crawlers really reach the site in the AI crawler log.

Questions and answers

Does Cloudflare block AI crawlers by default?

Cloudflare's docs say that for new domains the Search, Agent and Training policies default to Allow. Older zones may have settings someone turned on, so check.

What is Bot Preference Sync?

It makes Cloudflare write robots.txt directives that match your AI Crawler and Agent access policies, and it adds Content Signals.

Is robots.txt enough to stop AI crawlers?

No. robots.txt compliance is voluntary. Cloudflare's blocking rules enforce access at the edge.

What are Content Signals?

A line in robots.txt such as Content-signal: search=yes, ai-train=no, with categories search, ai-input and ai-train.

Will blocking AI on Cloudflare affect Google Search?

Blocking Googlebot would. Cloudflare treats verified search engine crawlers separately, but always check Googlebot and Bingbot are allowed after a change.

How do I see what Cloudflare added to robots.txt?

Open https://yourdomain/robots.txt. Managed directives are prepended above your own rules.

Sources

Facts checked on 14 November 2026.

Related tools and guides

AI readiness by platform

All free Countnest tools · Plain-text summary for AI assistants

See your own numbers in minutes

30-day free trial, no card required. Then from $14 a month. Cancel anytime.

Start free for 30 days