AI readiness

Before ChatGPT, Claude, Perplexity, Gemini or Google AI Mode can cite your site, their crawlers have to be allowed in, and your pages have to be readable without JavaScript. The AI readiness audit checks exactly that. In a few seconds it tests your robots.txt rules for each AI crawler, looks for llms.txt, finds your sitemap and analyzes your homepage. It then gives you a score out of 100 with a concrete fix for every problem.

It's for site owners and SEOs who want to rule out technical blockers before working on AI visibility. Open it from Project → AI readiness, or open it directly.

How it works

The audit runs on your project's domain in five stages, and the checklist ticks off live as each check finishes:

1. Connecting to your site – HTTPS.
2. Reading robots.txt – robots.txt, AI search and answer crawlers, AI training crawlers.
3. Looking for llms.txt – llms.txt and llms-full.txt.
4. Finding your XML sitemap – listed in robots.txt, or at /sitemap.xml or /sitemap_index.xml.
5. Analysing homepage content – structured data, server-rendered text, opt-out meta tags, title and description.

Audits run when you click the button, not on a schedule. Every finished run is saved to the history.

The AI crawlers we check

Crawler	Owner	Purpose
OAI-SearchBot	OpenAI	Search / answers
ChatGPT-User	OpenAI	Live fetch
GPTBot	OpenAI	Training
Claude-SearchBot	Anthropic	Search / answers
ClaudeBot	Anthropic	Training
PerplexityBot	Perplexity	Search / answers
Googlebot	Google	Search / answers
Google-Extended	Google	Training
Bingbot	Microsoft	Search / answers
Applebot-Extended	Apple	Training
CCBot	Common Crawl	Training
meta-externalagent	Meta	Training

For each crawler we check whether it may fetch your homepage. A crawler follows the robots.txt group that names it, or the User-agent: * group if none does. If you have no robots.txt, every crawler is allowed.

Step by step

1. Open Project → AI readiness.
2. Click Run audit. It takes a few seconds.
3. Read the verdict and the score ring at the top, then go through the checklist. Each check shows Passed, Needs attention, Problem or Note, its points, and a Fix: line when something is wrong.
4. Look at AI crawler access to see each crawler's status (Allowed or Blocked) and the exact robots.txt rule that matched.
5. Scroll to Generated llms.txt (or Suggested llms.txt if you already have one). Review it, then click Copy or Download and upload it to https://yourdomain/llms.txt.
6. Make your fixes on the site, then click Run again.
7. Check Audit history to see the new score and which checks you fixed or broke since the previous run.

What the numbers mean

The score is the points you earned divided by the maximum (110), shown out of 100:

Check	Points	Full points when
HTTPS	10	The site is served over HTTPS
AI search & answer crawlers allowed	30	None of the 6 search and live-fetch crawlers are blocked (partial points per crawler)
AI training crawlers	5	No training crawler is blocked (2 if some are; shown as a Note)
robots.txt	5	A robots.txt exists (2 without one)
llms.txt	10	A valid /llms.txt is found (5 if it has issues)
llms-full.txt	5	/llms-full.txt exists (optional)
XML sitemap	10	A sitemap is reachable
Structured data (JSON-LD)	10	Homepage has Organization-type schema (6 for other types)
Server-rendered content	10	200+ words of text in the raw HTML (0 if the content only renders with JavaScript)
No AI opt-out meta tags	5	No noai, noimageai or noindex in the robots meta tag or X-Robots-Tag header
Title & meta description	10	Both present (5 each)

Blocking answer crawlers outweighs everything else. If all of them are blocked, the score is capped at 30. If some are, it's capped at 80.

Verdicts: 85+ Excellent, 65–84 Good, 40–64 Needs work, below 40 Blocked or invisible.

llms.txt is valid when it starts with an H1 title (# Site name) and links to at least one page. We also flag a missing > summary blockquote and links that aren't grouped under ## sections. If the URL returns an HTML page (a soft 404), it counts as missing.

The generated llms.txt is built from your homepage title and description and up to 30 URLs from your sitemap, grouped by their first path segment. Without a sitemap it falls back to your homepage.

Audit history keeps your last 50 runs per project. For each run it shows the score with its change since the run before, the counts in Passed · Warnings · Problems, and Changes since the previous run. Click View details to reopen an older run.

Tips

• Training crawlers (GPTBot, ClaudeBot, Google-Extended and others) are a policy choice. Blocking them costs only a few points. Blocking search and live-fetch crawlers stops engines from citing you.
• Allow crawlers by name if your User-agent: * group is strict. A named group overrides the wildcard group.
• If Server-rendered content fails, most AI crawlers see an almost empty page. Server-render (SSR or static generation) your key content.
• Add Organization schema with your name, logo, URL and sameAs profile links so engines can resolve your brand.
• You can run the audit up to 30 times an hour per project. Run it again after each deploy that touches robots.txt or templates.
• Prefer a quick check without a project? Use the free llms.txt checker & generator and the robots.txt AI bot tester.

FAQ

Should I block AI training crawlers?

That's your call. Blocking training crawlers keeps your content out of model training data, but models may then know less about your brand. The audit marks this as a Note and deducts only 3 points. Never block the search and live-fetch crawlers if you want to be cited.

Is llms.txt required?

No standard requires it, but it gives AI tools a clean map of your key pages. It's worth 10 points here, and the audit writes a starting file for you to review and upload.

Does the audit check every page of my site?

No. It checks robots.txt, llms.txt, llms-full.txt and your sitemap, and analyzes your homepage. For a full crawl of your pages, use Site audit.

Does the audit run automatically?

No. It runs when you click Run audit or Run again, and every run is saved to Audit history so you can compare.

Why does a crawler show as blocked when I allowed it?

Look at the Matched rule column. A group that names the crawler takes priority over User-agent: *, and within a group the longest matching path wins. A leftover Disallow: / in a named group is the usual cause.

Related guides

• AI visibility
• Site audit