Cloudflare: AI crawler settings and robots.txt
No signup, no email. The checker reads only public files and gives a plain-English fix list.
Open the ai readiness checkerCloudflare's AI controls
| Control | What it does |
|---|---|
| AI Crawler and Agent access policies | Choose Allow, Block or Block on pages with ads for Search, Agent and Training behaviour. Training also has Disallow, which on its own only adds a robots.txt request |
| Bot Preference Sync | Writes managed robots.txt directives from those policies. With sync on, Training set to Disallow also blocks training crawlers at the edge, except mixed-purpose crawlers used for both search and training |
| Block AI bots | Older name for this feature ("Configure AI bot policies"); you may still see it in guides and older dashboards |
| Content Signals Policy | Comment text plus a Content-signal line in robots.txt stating search, ai-input and ai-train preferences |
Two different things: asking and blocking
robots.txt and Content Signals ask crawlers to behave. Cloudflare's blocking rules enforce it at the edge, so a blocked crawler never reaches your server, whatever robots.txt says. If you block Search or Agent behaviour, AI search tools and assistants can't fetch your pages, so they can't read new or updated content (an AI tool may still mention what it already knows about you). Cloudflare also says Training actions apply to mixed-purpose crawlers used for both search and training, so a Block on Training can reach further than you expect.
A setup that keeps AI search visibility
- Security > Settings > AI Crawler and Agent access policies.
- Leave Search and Agent on Allow.
- Set Training to Disallow if you want to opt out of training.
- Turn on Bot Preference Sync and save.
- Reload /robots.txt: you should see Cloudflare's managed lines first, then your own rules.
Check the result
Cloudflare's managed content appears within seconds. Run the AI readiness checker on your domain: it reads the live robots.txt Cloudflare serves. Countnest customers can also see which AI crawlers really reach the site in the AI crawler log.
Questions and answers
Does Cloudflare block AI crawlers by default?
Cloudflare's docs say that for new domains the Search, Agent and Training policies default to Allow. Older zones may have settings someone turned on, so check.
What is Bot Preference Sync?
It makes Cloudflare write robots.txt directives that match your AI Crawler and Agent access policies, and it adds Content Signals.
Is robots.txt enough to stop AI crawlers?
No. robots.txt compliance is voluntary. Cloudflare's blocking rules enforce access at the edge.
What are Content Signals?
A line in robots.txt such as Content-signal: search=yes, ai-train=no, with categories search, ai-input and ai-train.
Will blocking AI on Cloudflare affect Google Search?
Blocking Googlebot would. Cloudflare treats verified search engine crawlers separately, but always check Googlebot and Bingbot are allowed after a change.
How do I see what Cloudflare added to robots.txt?
Open https://yourdomain/robots.txt. Managed directives are prepended above your own rules.
Sources
Related tools and guides
AI readiness by platform
All free Countnest tools · Plain-text summary for AI assistants
30-day free trial, no card required. Then from $14 a month. Cancel anytime.
Start free for 30 days