Countnest

Free AI readiness checker

In short

Countnest's free AI readiness checker reads a site's public robots.txt, llms.txt, sitemap and homepage, tests the homepage rules for 17 named AI and search user agents, and gives a score and a fix list. It tests technical access, not citation likelihood.

AI crawl readiness is the ability of AI search crawlers to reach a website and discover its public pages; it is not proof of indexing, ranking, model training or an AI citation.

What it checks and why it matters

WhatWhy it matters
robots.txt per crawlerSearch, training and user-triggered fetchers have different names and purposes. A broad block may hide your site from search when you only meant to opt out of training.
Homepage status and noindexA failed response or a noindex instruction can stop a search tool from using the page, whatever robots.txt says.
llms.txtLooks for a Markdown title and links. This optional proposed format is a curated site map for AI tools that choose to use it.
SitemapChecks for a sitemap and whether robots.txt lists it. It does not inspect every page or prove Google has indexed the URLs.

How to use it, step by step

  1. Enter a domain. Use the public domain, for example example.com.
  2. Read the fix list. Start with an unreachable or noindex homepage, then crawler-specific rules.
  3. Choose access by purpose. Keep search allowed if you want visibility, and decide about training separately.
  4. Review and publish changes. Merge suggestions with your existing rules; don't overwrite private-path exclusions.
  5. Verify the live files. Open robots.txt yourself, then re-check after the six-hour cache expires. Crawlers can take longer to pick up changes.

Worked examples

Illustrative: a launch left on noindex

A staging rule remained after launch. The homepage returned 200 but contained noindex. Remove that instruction from the live site, keep staging protected, and request a fresh crawl in Search Console.

Illustrative: search allowed, training declined

A publisher disallows GPTBot but allows OAI-SearchBot. These settings are independent: OpenAI documents one for model training and one for ChatGPT search. ChatGPT-User is a user-triggered fetcher, for which robots rules may not apply.

Illustrative: missing llms.txt

A site has a reachable homepage, sitemap and open search crawler rules but no llms.txt. That is not a blocker for Google AI features. Add a curated file if useful to your audience and keep measuring actual referrals.

Common mistakes

  • Treating a score as proof of citations. The score is a Countnest technical checklist, not a score supplied by an AI company.
  • Replacing robots.txt wholesale. Keep existing exclusions and sitemap declarations. Named groups don't inherit the wildcard group's paths.
  • Confusing search with training. Read the documented purpose of each user agent before blocking it.
  • Ignoring CDN blocks. A firewall can deny bots even when robots.txt says Allow. This check does not impersonate every crawler's network or IP.
  • Using robots.txt as a security boundary. It is public, voluntary guidance. Protect private content with authentication.

Key facts, with sources

What we check

  • robots.txt, per crawler: AI search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User), training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider), and Googlebot and Bingbot.
  • Homepage: reachable over https, and not marked noindex.
  • llms.txt: present, with a title and links.
  • Sitemap: found, and listed in robots.txt.

You get a score out of 100, a plain-English fix list and a robots.txt block you can copy. For the reasoning behind each rule, read robots.txt for AI crawlers: allow, block or choose.

What it can't tell you

Robots rules say what a crawler may fetch, not what it actually fetched or whether an assistant cites you. To see which AI crawlers really read your pages and which assistants send visitors, you need server logs or an analytics tool that records them, such as Countnest.

Questions and answers

What does the checker look at?

Your homepage (is it reachable, does it say noindex), your robots.txt rules for 17 AI and search crawlers, your /llms.txt file and your sitemap. It reads only those public files.

Does blocking GPTBot remove me from ChatGPT?

Not by itself. GPTBot collects data for model training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a user asks. The checker shows each one separately so you can choose.

Can you detect IndexNow?

No. An IndexNow key is a public ownership-verification file, not a secret, but an outside scan cannot reliably discover whether submissions are configured or succeeding. Countnest customers can turn IndexNow on in the SEO tab.

Is llms.txt required?

No. It is a proposed standard, not something Google or OpenAI require. It is cheap to add, and some AI tools read it.

Do you store anything about me?

We store the result for the domain for a few hours so repeat checks are fast. We don't store your IP address or who ran the check.

Does a high score mean AI assistants will cite me?

No. This is Countnest's technical-access checklist, not a citation or ranking score. We do not submit prompts to AI models or promise mentions.

Is robots.txt enough to allow a crawler?

Not always. A firewall, CDN, authentication or bot challenge can still block it. Check your hosting and CDN access settings too.

Can I share my result and add a badge?

Yes. The result page has Copy report link and an embeddable score badge. Result pages are noindex, and the badge links to the free checker. A badge is a dated technical check, not certification.

Sources

Facts checked on 14 November 2026.

Guides for this tool

More free tools

  • llms.txt generator: builds a ready-to-edit llms.txt from your homepage and sitemap, with key pages grouped, titled and summarised.
  • Metadata and structured data checker: checks a page's title, meta description, canonical, robots, Open Graph, X card and JSON-LD structured data, with a fix list.
  • UTM link builder: builds campaign links with utm_source, utm_medium and utm_campaign, and flags capital letters, spaces and personal data.
  • Third-party tracker checker: lists the analytics, ad pixels, session recorders, embeds and font services a page loads, and the company behind each.

Last updated by Countnest (IDCPH ApS). Facts are checked against the linked sources on that date.

Want this for your own traffic, every day?

The ai readiness checker is free and stays free. Countnest analytics goes further: visits from ChatGPT, Perplexity and other assistants, AI crawler logs, search data and campaigns, without identifying visitors. 30-day free trial, no card required, then from $14 a month.

Try Countnest free for 30 days What Countnest does
Free AI readiness checker: robots.txt, llms.txt and sitemap | Countnest