--- title: "Pass rates from the shared crawl | Agent Ready Checklist" description: "What share of the top 10,000 sites passes each check the Verify Kit can run: real 404, llms.txt, markdown, JSON-LD, OpenAPI and more, with CSV and JSON." canonical: https://agentreadychecklist.com/data site: "Agent Ready Checklist" last_updated: 2026-11-05 date: 2026-11-04 section: "Data" --- # Pass rates from the shared crawl | Agent Ready Checklist > What share of the top 10,000 sites passes each check the Verify Kit can run: real 404, llms.txt, markdown, JSON-LD, OpenAPI and more, with CSV and JSON. Source: https://agentreadychecklist.com/data Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # Pass rates from the shared crawl Of 7,840 reachable sites in the Tranco top 10,000, 66% return a real 404 for a missing path and 14% serve an llms.txt. Only 1.3% link it from the home page. The table gives 10 pass rates, one per check the Verify Kit can run, from the crawl of 2026-11-04. ## The rates | Check | Passing | Of | Share | Kit check | Requirements | | --- | --- | --- | --- | --- | --- | | robots.txt allows all 17 agent tokens at / | 5,580 | 7,840 | 71.2% | robots-verdicts | AR-FIND-01 | | robots.txt blocks no search or user-fetch token | 5,738 | 7,840 | 73.2% | robots-verdicts | AR-FIND-01, VERCEL-ROBOTS-EXCLUSIONS | | Home page has an h1 and 50 words in the initial HTML | 3,767 | 5,890 | 64.0% | initial-html | AR-READ-01, AR-READ-03 | | A missing path returns 404 or 410 | 4,985 | 7,500 | 66.5% | status-codes | AR-READ-02 | | /llms.txt exists | 1,071 | 7,514 | 14.3% | llms-linked | AR-READ-07 | | Home page links to llms.txt | 78 | 5,890 | 1.3% | llms-linked | AR-READ-06, AR-READ-07 | | Accept: text/markdown returns text/markdown | 253 | 7,534 | 3.4% | markdown-negotiation | CF-MARKDOWN-NEGOTIATION, AR-READ-09 | | Home page carries JSON-LD that parses | 2,793 | 5,890 | 47.4% | json-ld | AR-READ-08 | | OpenAPI linked from the home page or served at /openapi.json | 37 | 7,559 | 0.5% | openapi | AR-ACT-02, AR-READ-06 | | /.well-known/mcp.json serves JSON | 27 | 7,461 | 0.4% | mcp-or-cli | AR-ACT-03, AR-ACT-10 | [The edition page](https://agentreadychecklist.com/data/2026-11-04/) has the chart and the findings written for citation. Definitions are in the CSV and on the [methodology page](https://agentreadychecklist.com/method/#crawl). ## More cuts of the crawl - [Pass rates by rank band](https://agentreadychecklist.com/data/by-rank/)The ten rates in ten bands of 1,000 ranks, with a CSV and the gap between the top and bottom bands. - [The five most failed rows](https://agentreadychecklist.com/data/most-failed-requirements/)The rows that apply to every site and fail most, with the fix on three platforms, the Ora check and the kit command. - [Ora check for each row](https://agentreadychecklist.com/checklist/ora-checks/)The crosswalk between the rows and Ora's catalogue of 132 checks, with a CSV and JSON. ## Download - [pass-rates-2026-11-04.csv](https://agentreadychecklist.com/data/pass-rates-2026-11-04.csv)One row per check: counts, share and the definition. Comma-separated, UTF-8. - [pass-rates-2026-11-04.json](https://agentreadychecklist.com/data/pass-rates-2026-11-04.json)The same rates with the crawl metadata: Tranco list ID, row counts, user agent. - [pass-rates.csv](https://agentreadychecklist.com/data/pass-rates.csv)Always the latest edition. Licence: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Cite it as: ```text Agent Ready Checklist, "Pass rates for the checks the Verify Kit can run", edition 2026-11-04, https://agentreadychecklist.com/data/2026-11-04/ ``` ## The crawl behind it Crawl date 2026-11-04 Universe Tranco top 10,000 domains, crawled on 2026-11-04 with a plain-fetch client (no JavaScript), 10-second timeout, one pass. Tranco list 746WX Rows in the crawl 13,071 domains, of which 9,990 are in the Tranco top 10,000 Reachable 7,840 sites answered a robots.txt request User agent Mozilla/5.0 (compatible; agent-readiness-crawl/1.0) ## Sources - [tranco-list.eu](https://tranco-list.eu/)read 2026-11-04 - [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04 Edition Data 2026-11-04 Reviewed 2026-11-05 Spec AgentReady v2.0, 273d2d7 Crawl 13,071 rows, 2026-11-04 Ora 132 checks, contract 1.38.0, 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Method - [How rows, commands and rates are made](https://agentreadychecklist.com/method/) - [How an Ora check is matched to a row](https://agentreadychecklist.com/checklist/ora-checks/) - [Pass rates by rank band](https://agentreadychecklist.com/data/by-rank/) Sources and read dates sit at the foot of the page. Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.