Countnest

AI readiness guides by platform

Last updated

WordPress: robots.txt for AI crawlers and llms.txt

Allow AI search crawlers and choose about training bots on WordPress: the virtual robots.txt, editing it with Yoast or a real file, the 'Discourage search engines' trap, and llms.txt.

Shopify: robots.txt for AI crawlers and llms.txt

Control AI crawlers on a Shopify store with the robots.txt.liquid theme template, keep Shopify's default rules, and check the store's built-in /llms.txt and /agents.md.

Wix: robots.txt for AI crawlers and llms.txt

Use the Wix Robots.txt Editor (SEO & GEO > Tools and settings) to choose which AI crawlers may read your site, and review the llms.txt file Wix generates automatically.

Squarespace: the AI crawlers setting and llms.txt

Squarespace doesn't let you edit robots.txt. What its 'Block known artificial intelligence crawlers' checkbox does (and doesn't) block, and how to add llms.txt on version 7.1.

Webflow: robots.txt for AI crawlers and llms.txt

Set AI crawler rules in Webflow (Site settings > SEO > Indexing), upload llms.txt (UTF-8, under 100 KB) and use the Content-Signal header. Publish, then check the live domain.

Cloudflare: AI crawler settings and robots.txt

How Cloudflare's AI Crawler and Agent access policies, Bot Preference Sync, managed robots.txt and the Block AI bots rule affect who can read your site, and how to check the result.

Open the free ai readiness checker

See your own numbers in minutes

30-day free trial, no card required. Then from $14 a month. Cancel anytime.

Start free for 30 days