Webflow: robots.txt for AI crawlers and llms.txt
No signup, no email. The checker reads only public files and gives a plain-English fix list.
Open the ai readiness checkerrobots.txt in Webflow
- Site settings > SEO > Indexing.
- Paste your rules into the robots.txt field.
- Save and publish.
Webflow adds your sitemap to robots.txt by default.
Rules for AI crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
This keeps AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed and asks the main training crawlers to stay out. User-triggered fetchers such as ChatGPT-User act on a person's request, and OpenAI says robots.txt rules may not apply to them. Merge, don't replace: the User-agent: * group above is a minimal public example. Keep any Disallow lines your site already has for admin, cart, checkout, account or search pages in that group. Remove the first group if you're happy to allow training too.
Content-Signal
Webflow can also send a Content-Signal header (same Indexing settings), for example Content-Signal: ai-train=no, search=yes, ai-input=yes. It states how content may be used: search indexing, AI answers (ai-input) and training (ai-train). It's a preference, like robots.txt, not a technical block.
llms.txt in Webflow
- Site settings > SEO > LLMs.txt.
- Upload file (UTF-8, under 100 KB) and save.
- Publish, then open https://yourdomain/llms.txt.
Webflow says llms.txt doesn't guarantee LLMs will read your site. Re-upload it when you add or rename key pages.
Questions and answers
Where do I edit robots.txt in Webflow?
Site settings > SEO > Indexing. Changes go live when you publish.
How do I add llms.txt in Webflow?
Site settings > SEO > LLMs.txt > Upload file. The file must be UTF-8 and under 100 KB.
Why can't I see llms.txt on my webflow.io domain?
Webflow publishes it only on your custom domain, not the staging domain.
What is the Content-Signal header?
A way to state whether content may be used for search, AI answers or AI training. Webflow lets you set it in the Indexing settings.
Does Webflow block AI crawlers by default?
Webflow's help doesn't list an AI block by default. Check your live robots.txt with the AI readiness checker.
Do I need a paid plan?
Webflow ties some site settings and custom code to paid Site plans. Check your plan in Webflow if a field is locked.
Sources
Related tools and guides
AI readiness by platform
All free Countnest tools · Plain-text summary for AI assistants
30-day free trial, no card required. Then from $14 a month. Cancel anytime.
Start free for 30 days