WordPress: robots.txt for AI crawlers and llms.txt
No signup, no email. The checker reads only public files and gives a plain-English fix list.
Open the ai readiness checkerHow robots.txt works on WordPress
If there's no physical robots.txt file, WordPress generates one on the fly. A real robots.txt file in the site root takes over completely. SEO plugins such as Yoast SEO and Rank Math have a robots.txt editor that writes or filters this file for you.
Recommended rules
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
This keeps AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed and asks the main training crawlers to stay out. User-triggered fetchers such as ChatGPT-User act on a person's request, and OpenAI says robots.txt rules may not apply to them. Merge, don't replace: the User-agent: * group above is a minimal public example. Keep any Disallow lines your site already has for admin, cart, checkout, account or search pages in that group. Remove the first group if you're happy to allow training too.
The setting that blocks everything
Settings > Reading > Search engine visibility ("Discourage search engines from indexing this site") is often left on after a site launch. It asks every search engine not to index the site. Turn it off on a live site; the AI readiness checker flags a noindex homepage.
Plugins and bot blockers
- Yoast SEO Premium has a Bot Blocker that adds training crawlers such as GPTBot, CCBot and Google-Extended to robots.txt.
- Security plugins and hosts sometimes block bots by user agent at the server. That's stronger than robots.txt and can block AI search crawlers too. Check both.
llms.txt
Yoast SEO can generate llms.txt from your content, or you can upload your own file to the root folder. See the WordPress llms.txt example for a complete file.
Questions and answers
Where is robots.txt in WordPress?
If you haven't created one, it's virtual: WordPress generates it at /robots.txt. A real file in the root folder replaces it.
How do I block GPTBot on WordPress?
Add `User-agent: GPTBot` and `Disallow: /` through your SEO plugin's file editor or a real robots.txt. Blocking GPTBot stops training use, not ChatGPT search.
Will blocking AI crawlers hurt my Google rankings?
Blocking Google-Extended doesn't affect Search, per Google. Blocking Googlebot would. Never disallow Googlebot on a live site.
What does 'Discourage search engines' do?
It asks search engines not to index the site. Leave it off once the site is live.
Do I need a plugin for llms.txt?
No. You can upload a plain llms.txt file to the root folder. Yoast can also generate one.
Why does the checker say crawlers are blocked when robots.txt looks fine?
A security plugin, firewall or host may block bots before robots.txt matters. Check those settings, then re-run the checker.
Sources
- WordPress Developer Resources: do_robots() and the virtual robots.txt
- Yoast: Bot Blocker (Yoast SEO Premium)
- Yoast: How to enable llms.txt with Yoast SEO
- OpenAI: Overview of OpenAI crawlers
- Google: Google's common crawlers (incl. Google-Extended)
- IETF: RFC 9309, Robots Exclusion Protocol (September 2022)
Related tools and guides
AI readiness by platform
All free Countnest tools · Plain-text summary for AI assistants
30-day free trial, no card required. Then from $14 a month. Cancel anytime.
Start free for 30 days