--- title: "Make a website readable by AI agents | Agent Ready Checklist" description: "Make your website readable by AI agents, in order. Eight rows in the order to do them, with the crawl failure rate, the kit command and the Ora check for each." canonical: https://agentreadychecklist.com/guides/make-your-website-readable-by-ai-agents site: "Agent Ready Checklist" last_updated: 2026-11-05 date: 2026-11-05 section: "Guides" --- # Make a website readable by AI agents | Agent Ready Checklist > Make your website readable by AI agents, in order. Eight rows in the order to do them, with the crawl failure rate, the kit command and the Ora check for each. Source: https://agentreadychecklist.com/guides/make-your-website-readable-by-ai-agents Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # Make your website readable by AI agents, in order Three MUST rows decide whether an agent can read your site: let agents in, put the answer in the initial HTML and return real status codes. In the crawl of 2026-11-04, 29% to 36% of sites fail each one. This guide orders eight rows, with a check, the Ora read and the platform fix for each. ## Before you start You need shell access to run the checks, and the right to change robots.txt, the response codes and the page templates. Readable here means an agent that fetches your pages without running JavaScript gets the answer. The order is by what breaks an agent first, then by how many sites fail. The [one-day sprint](https://agentreadychecklist.com/guides/one-day-agent-readiness-sprint/) covers the same rows as a timetable. ## Why the order starts here Ora's lab served one docs site in 19 configurations to a plain-fetch client and a JavaScript client, 190 probes in all. Two changes broke an agent: hiding the answer in JavaScript and returning 403 to agent user agents. The other 16 configurations passed 5 of 5. The first two steps below close those two failures, and the third is the status-code row that stops a dead page from looking alive. Source: [Vercel, Make your site readable by AI agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04, a vendor claim by the lab that ran it. ## The steps 1. **[AR-FIND-01: Do not block agents.](https://agentreadychecklist.com/checklist/ar-find-01/)** MUST. 28.8% of the 7,840 sites tested fail the crawl check for it: robots.txt allows all 17 agent tokens at /. Direct recipe on 19 of 19 platforms. Ora reads it with robots-ai-policy-quality and robots-agent-user-policy and 2 more. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=reach-as-agent,robots-verdicts ``` Watch for: A named group (User-agent: GPTBot, Disallow: /) left over from 2023 that overrides a permissive wildcard. 2. **[AR-READ-01: Answer in the initial HTML.](https://agentreadychecklist.com/checklist/ar-read-01/)** MUST. 36.0% of the 5,890 sites tested fail the crawl check for it: Home page has an h1 and 50 words in the initial HTML. Direct recipe on 19 of 19 platforms. Ora reads it with content-no-js. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=initial-html ``` Watch for: Prerendering the shell but fetching the answer client-side. 3. **[AR-READ-02: Real HTTP status codes.](https://agentreadychecklist.com/checklist/ar-read-02/)** MUST. 33.5% of the 7,500 sites tested fail the crawl check for it: A missing path returns 404 or 410. Direct recipe on 17 of 19 platforms. Ora reads it with redirect-hygiene. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=status-codes ``` Watch for: SPA hosts rewriting every path to index.html with 200. 4. **[AR-READ-08: JSON-LD on key pages.](https://agentreadychecklist.com/checklist/ar-read-08/)** SHOULD. 52.6% of the 5,890 sites tested fail the crawl check for it: Home page carries JSON-LD that parses. Direct recipe on 15 of 19 platforms. Ora reads it with json-ld. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=json-ld ``` Watch for: Invalid JSON from a template (trailing comma, unescaped quote). 5. **[AR-FIND-03: Sitemap listed in robots.txt.](https://agentreadychecklist.com/checklist/ar-find-03/)** SHOULD. The crawl has no rate for it. Direct recipe on 19 of 19 platforms. Ora reads it with sitemap. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=sitemap-listed ``` Watch for: Sitemap lists URLs that redirect or 404. 6. **[AR-READ-07: Ship llms.txt.](https://agentreadychecklist.com/checklist/ar-read-07/)** SHOULD. 85.8% of the 7,514 sites tested fail the crawl check for it: /llms.txt exists. Direct recipe on 14 of 19 platforms. Ora reads it with llms-txt-exists and llms-txt-formatting and 1 more. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=llms-linked ``` Watch for: One multi-megabyte file that overflows the context window. 7. **[AR-READ-06: Link the discovery files.](https://agentreadychecklist.com/checklist/ar-read-06/)** SHOULD. 98.7% of the 5,890 sites tested fail the crawl check for it: Home page links to llms.txt. Direct recipe on 17 of 19 platforms. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=llms-linked,openapi ``` Watch for: llms.txt published at the root with nothing pointing to it. 8. **[AR-READ-09: Advertise a markdown mirror.](https://agentreadychecklist.com/checklist/ar-read-09/)** MAY. 96.6% of the 7,534 sites tested fail the crawl check for it: Accept: text/markdown returns text/markdown. Direct recipe on 10 of 19 platforms. Ora reads it with markdown-link-alternate. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=markdown-negotiation ``` Watch for: Mirror drifts from the HTML because it is built from a different source. ## How you know it is done Run the kit on the home page and on one docs page, and every MUST that applies to the site prints PASS. Then run `npx ax@0.10 audit https://example.com` against the live site once and keep the score as your baseline. [Gate deploys on it](https://agentreadychecklist.com/guides/gate-deploys-on-agent-readiness-in-ci/) so the next release cannot undo the work. The full table of rows is [the checklist](https://agentreadychecklist.com/). ## Sources - [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents)read 2026-11-04 - [github.com/agentready-org/standard, spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04 - [ora.ai/api/checks, contract 1.38.0](https://ora.ai/api/checks?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=guide)read 2026-11-05 - [tranco-list.eu](https://tranco-list.eu/)read 2026-11-04 Edition Data 2026-11-04 Reviewed 2026-11-05 Spec AgentReady v2.0, 273d2d7 Crawl 13,071 rows, 2026-11-04 Ora 132 checks, contract 1.38.0, 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Method - [How rows, commands and rates are made](https://agentreadychecklist.com/method/) - [How an Ora check is matched to a row](https://agentreadychecklist.com/checklist/ora-checks/) - [Pass rates by rank band](https://agentreadychecklist.com/data/by-rank/) Sources and read dates sit at the foot of the page. Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.