Countnest

WordPress: robots.txt for AI crawlers and llms.txt

By Countnest · Last updated

Check your site with Countnest's free ai readiness checker

No signup, no email. The checker reads only public files and gives a plain-English fix list.

Open the ai readiness checker

How robots.txt works on WordPress

If there's no physical robots.txt file, WordPress generates one on the fly. A real robots.txt file in the site root takes over completely. SEO plugins such as Yoast SEO and Rank Math have a robots.txt editor that writes or filters this file for you.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This keeps AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed and asks the main training crawlers to stay out. User-triggered fetchers such as ChatGPT-User act on a person's request, and OpenAI says robots.txt rules may not apply to them. Merge, don't replace: the User-agent: * group above is a minimal public example. Keep any Disallow lines your site already has for admin, cart, checkout, account or search pages in that group. Remove the first group if you're happy to allow training too.

The setting that blocks everything

Settings > Reading > Search engine visibility ("Discourage search engines from indexing this site") is often left on after a site launch. It asks every search engine not to index the site. Turn it off on a live site; the AI readiness checker flags a noindex homepage.

Plugins and bot blockers

  • Yoast SEO Premium has a Bot Blocker that adds training crawlers such as GPTBot, CCBot and Google-Extended to robots.txt.
  • Security plugins and hosts sometimes block bots by user agent at the server. That's stronger than robots.txt and can block AI search crawlers too. Check both.

llms.txt

Yoast SEO can generate llms.txt from your content, or you can upload your own file to the root folder. See the WordPress llms.txt example for a complete file.

Questions and answers

Where is robots.txt in WordPress?

If you haven't created one, it's virtual: WordPress generates it at /robots.txt. A real file in the root folder replaces it.

How do I block GPTBot on WordPress?

Add `User-agent: GPTBot` and `Disallow: /` through your SEO plugin's file editor or a real robots.txt. Blocking GPTBot stops training use, not ChatGPT search.

Will blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended doesn't affect Search, per Google. Blocking Googlebot would. Never disallow Googlebot on a live site.

What does 'Discourage search engines' do?

It asks search engines not to index the site. Leave it off once the site is live.

Do I need a plugin for llms.txt?

No. You can upload a plain llms.txt file to the root folder. Yoast can also generate one.

Why does the checker say crawlers are blocked when robots.txt looks fine?

A security plugin, firewall or host may block bots before robots.txt matters. Check those settings, then re-run the checker.

Sources

Facts checked on 14 November 2026.

Related tools and guides

AI readiness by platform

All free Countnest tools · Plain-text summary for AI assistants

See your own numbers in minutes

30-day free trial, no card required. Then from $14 a month. Cancel anytime.

Start free for 30 days