Countnest

Webflow: robots.txt for AI crawlers and llms.txt

By Countnest · Last updated

Check your site with Countnest's free ai readiness checker

No signup, no email. The checker reads only public files and gives a plain-English fix list.

Open the ai readiness checker

robots.txt in Webflow

  1. Site settings > SEO > Indexing.
  2. Paste your rules into the robots.txt field.
  3. Save and publish.

Webflow adds your sitemap to robots.txt by default.

Rules for AI crawlers

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This keeps AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed and asks the main training crawlers to stay out. User-triggered fetchers such as ChatGPT-User act on a person's request, and OpenAI says robots.txt rules may not apply to them. Merge, don't replace: the User-agent: * group above is a minimal public example. Keep any Disallow lines your site already has for admin, cart, checkout, account or search pages in that group. Remove the first group if you're happy to allow training too.

Content-Signal

Webflow can also send a Content-Signal header (same Indexing settings), for example Content-Signal: ai-train=no, search=yes, ai-input=yes. It states how content may be used: search indexing, AI answers (ai-input) and training (ai-train). It's a preference, like robots.txt, not a technical block.

llms.txt in Webflow

  1. Site settings > SEO > LLMs.txt.
  2. Upload file (UTF-8, under 100 KB) and save.
  3. Publish, then open https://yourdomain/llms.txt.

Webflow says llms.txt doesn't guarantee LLMs will read your site. Re-upload it when you add or rename key pages.

Questions and answers

Where do I edit robots.txt in Webflow?

Site settings > SEO > Indexing. Changes go live when you publish.

How do I add llms.txt in Webflow?

Site settings > SEO > LLMs.txt > Upload file. The file must be UTF-8 and under 100 KB.

Why can't I see llms.txt on my webflow.io domain?

Webflow publishes it only on your custom domain, not the staging domain.

What is the Content-Signal header?

A way to state whether content may be used for search, AI answers or AI training. Webflow lets you set it in the Indexing settings.

Does Webflow block AI crawlers by default?

Webflow's help doesn't list an AI block by default. Check your live robots.txt with the AI readiness checker.

Do I need a paid plan?

Webflow ties some site settings and custom code to paid Site plans. Check your plan in Webflow if a field is locked.

Sources

Facts checked on 14 November 2026.

Related tools and guides

AI readiness by platform

All free Countnest tools · Plain-text summary for AI assistants

See your own numbers in minutes

30-day free trial, no card required. Then from $14 a month. Cancel anytime.

Start free for 30 days