How the scan measures

The instrument

A scan requests your site the way real AI agents request it: the exact User-Agent strings below (roster roster-v1.0), sent from two independent datacenter networks, beside three controls: a normal browser, a nonsense UA, and Googlebot. The controls separate "blocks AI agents" from "blocks everything unusual" from "site is broken".

Training crawlers: GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider

AI search indexers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot

Live assistant fetchers: ChatGPT-User, Perplexity-User, Claude-User

Policy tokens: Google-Extended, Applebot-Extended (robots.txt directives with nothing probeable; read, never requested). Controls: Chrome-Control, Nonsense-Control, Googlebot-Control.

The verdict

Five possible headlines: Open, Partially blocked, Agent-hostile, Contradictory, Unknown. Three rules decide them:

• Dual confirmation. A finding counts only when both networks saw the same thing. One network's word is never enough.
• Declared blocking is respected. A site whose robots.txt disallows an agent and whose edge enforces exactly that gets full credit. We measure whether policy and enforcement agree, not how open you are.
• Unknown is an answer. What we cannot measure to our own confidence bar is excluded and labeled, never guessed and never scored against you.

The score

The 0-100 readability score is deterministic: score = 100 x passed-weight / decided-weight over the 23 checks below (rubric rubric-v1.1, weights frozen and hash-guarded in CI). Checks that cannot be measured, or do not apply to your site profile, drop out of the denominator. Fewer than 5 decided checks renders "?" instead of a number. No LLM touches the score; the only AI-driven testing we do is the paid Live Agent Test, and its results never feed this score.

Weights are stated, versioned expert judgment (informed by a 2026 study of 44 scanners and the standards they check), not popularity. Every recorded run from the paid live-agent tests is archived, and aggregate run evidence will inform future rubric versions, published here with a new version number. A weight never changes silently.

Readability 43 (9 checks) · Access 41 (6 checks) · llms.txt 19 (4 checks) · Discovery 7 (2 checks) · API 5 (1 checks) · Structured data 4 (1 checks) · total weight 119

Access

• robots.txt is readable and sane (AC-ROB-02 · weight 12)
  robots.txt is the first file every agent reads; if it is broken or self-defeating, everything downstream suffers.
• Agents can get in (AC-UAD-02 · weight 12)
  This is the bottom line: when an AI agent knocks, does it get the page?
• Agents get the same content as a browser (AC-UAD-03 · weight 6)
  Some sites let agents in but serve them a stripped-down page; the agent then answers from thin content.
• No page-level AI bans (noindex / noai) (AC-ROB-04 · weight 5)
  A single meta tag can quietly remove your pages from AI answers.
• Live assistants are not silently denied (AC-ROB-03 · weight 3)
  When a person asks their AI about your site mid-conversation, this is the request that fails.
• No CAPTCHA on the front door (AC-ROB-06 · weight 3)
  Agents cannot solve CAPTCHAs; one on the front door turns every agent away.

Discovery

• Sitemap exists and is valid XML (AC-DIS-01 · weight 5)
  The sitemap is how agents and crawlers enumerate what you publish.
• Agent files served with correct content types (AC-DIS-04 · weight 2)
  A file served with the wrong type gets misread or discarded by strict clients.

llms.txt

• llms.txt exists (AC-LLM-01 · weight 8)
  llms.txt is the welcome note AI systems look for first; sites that have one get summarized on their own terms.
• llms.txt starts with a title (AC-LLM-02 · weight 5)
  Without the title line, many parsers treat the file as malformed.
• llms.txt fits an AI's reading budget (AC-LLM-03 · weight 3)
  Agents read with a fixed budget; an oversized file gets truncated mid-sentence.
• llms.txt links actually resolve (AC-LLM-04 · weight 3)
  Agents follow these links literally; each dead one is a dead end in a conversation.

Readability

• Content is readable without JavaScript (AC-REN-01 · weight 10)
  Most agents read raw HTML and never run your JavaScript; an empty shell means you are invisible.
• Markdown when asked (content negotiation) (AC-REN-02 · weight 8)
  Markdown is roughly 10x cheaper for an AI to read than rendered HTML.
• Missing pages return an honest 404 (AC-REN-09 · weight 5)
  A page that returns 200 OK while actually showing a not-found message sends agents in circles.
• No login, consent wall, or paywall in front of content (AC-REN-10 · weight 5)
  Whatever sits in front of your content is all an agent will ever see of it.
• Clean status and redirects (AC-REN-11 · weight 5)
  Redirect chains and cross-host hops make agents abort the fetch.
• A markdown mirror is fetchable (AC-REN-03 · weight 3 · dev-platform/docs only)
  A markdown twin gives agents a clean, cheap version of each page.
• Page weight fits an AI's budget (AC-REN-07 · weight 3)
  Oversized pages get cut off mid-read; whatever falls past the cut is never seen.
• Markdown alternate is declared (AC-REN-04 · weight 2 · dev-platform/docs only)
  The declaration is how an agent discovers your markdown version exists.
• Forms work without JavaScript (AC-REN-12 · weight 2)
  A form without action and method attributes only works with JavaScript, which most agents never run.

Structured data

• Structured data (JSON-LD) present (AC-SD-01 · weight 4)
  Structured data lets agents quote your facts precisely instead of guessing from prose.

API

• OpenAPI spec published (developer platforms) (AC-API-01 · weight 5 · dev-platform/docs only)
  An agent integrates with a developer platform through its API; the published spec is how it learns the endpoints.

What we never claim

Our requests come from datacenter networks, where AI agents actually run, not from the AI vendors' own verified addresses. The strongest claim a report makes is: your edge treats agent requests from cloud servers as hostile. Never that a specific vendor's bot is blocked. Findings quote raw HTTP status per network, and every sentence in a report is capped at what the evidence supports.

Versioning

Check ids are stable and never change meaning. Weight changes or new checks arrive only as a new frozen rubric version (the current files are immutable; CI verifies their hashes). The machine-readable catalog at /api/checks carries the same ids, weights, and fix guidance, and is safe to gate CI on.