--- title: "FAQ | Agent Ready Checklist" description: "32 questions and answers on agent readiness for developers: the minimum to pass, robots.txt, llms.txt, status codes, markdown, MCP, OAuth and WebMCP." canonical: https://agentreadychecklist.com/faq site: "Agent Ready Checklist" last_updated: 2026-11-05 section: "FAQ" --- # FAQ | Agent Ready Checklist > 32 questions and answers on agent readiness for developers: the minimum to pass, robots.txt, llms.txt, status codes, markdown, MCP, OAuth and WebMCP. Source: https://agentreadychecklist.com/faq Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # FAQ 32 questions from developers who work through the checklist: what the minimum is, which changes stopped agents in the research, what robots.txt and llms.txt do not do, and when an MCP server is needed. Each answer is short and cites its source. ## All 32 questions ### What is the minimum a site needs to pass AgentReady? Meet every MUST that applies. For an informational site that is three: do not block agents (AR-FIND-01), put the answer in the initial HTML (AR-READ-01) and return real status codes (AR-READ-02). A product agents act on adds AR-ACT-10, an MCP server or a CLI. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### How many requirements are in the current AgentReady spec? Version 2.0 (October 2026) has 24 live identifiers: 4 MUST, 14 SHOULD and 6 MAY, split across AR-FIND (4), AR-READ (10) and AR-ACT (10). AR-READ-10 is retired and its number is not reused. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Which changes actually stopped agents in the research? Two. Ora's lab served one docs site in 19 configurations to a plain-fetch client and a JavaScript client: hiding the answer in JavaScript failed the plain fetch 0 of 5, and a 403 to agent user agents failed both clients 0 of 5. The other 16 configurations passed 5 of 5. Source: [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04 ### Does llms.txt control which crawlers can read my site? No. llms.txt is an index for agents to read; it grants or denies nothing. Access is decided by robots.txt for compliant crawlers and by your WAF or bot manager for everything else. Source: [llmstxt.org](https://llmstxt.org/), read 2026-11-04 ### Does robots.txt stop an AI agent browsing with a user session? No. robots.txt is advisory and applies to crawlers. The spec notes that user-initiated fetchers may not consult it at all, and an agent driving a browser in a user session behaves like the user. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Should I block GPTBot? That is a training-data decision, separate from agent access. GPTBot and ClaudeBot are training crawlers; OAI-SearchBot and Claude-SearchBot build search indexes; ChatGPT-User and Claude-User fetch for a user. Blocking a training crawler does not drop you from agent answers, if the other groups stay open. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Why does my wildcard Allow not apply to GPTBot? Because RFC 9309 crawlers use only the most specific group that names them. If robots.txt has a User-agent: GPTBot group, GPTBot ignores the \* group entirely. Audit named groups before relying on the wildcard. Source: [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309), read 2026-11-04 ### Is a curl with an agent user agent enough to prove agents can reach my site? No. Bot managers weigh request origin more than the UA string, so a laptop curl can pass where the real agent is blocked, or the reverse. Check server logs for requests from the published OpenAI and Anthropic IP ranges and confirm 200. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Do I need a robots.txt at all? The Vercel guide says a public site does not need one to permit crawling and recommends rules only for deliberate exclusions. The AgentReady spec recommends one wildcard group plus a Sitemap line. Cloudflare's scanner scores its presence. Source: [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04 ### What status code should a missing page return? A 404, or a redirect to a real replacement. Never 200 with an error body: agents cannot tell a soft 404 from a live page. Check with curl -s -o /dev/null -w "%{http\_code}" on a random path. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### What should my rate limiter return to agents? A 429 Too Many Requests with a Retry-After header, not a CAPTCHA or challenge page. Agents and crawlers treat 429 as a back-off signal and come back; a challenge page looks like content or a block. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Does JSON-LD help agents? The spec says agents can parse facts from it, but the studies did not measure an accuracy gain, and markdown converters may drop script blocks. Add it where it is accurate and repeat critical facts in visible text. Source: [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04 ### Is markdown content negotiation required? Not by AgentReady. AR-READ-09 (MAY) is about advertising a mirror, and the spec says negotiation did not change success in the studies. Cloudflare scores negotiation as a check. If you do it, send Content-Type text/markdown and Vary: Accept. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### How often do agents ask for markdown? Ora's lab found (published 2026-08-28) that 2,259 of 3,470 web fetches (65.1%) requested markdown, and 95.7% of fetches that named a format chose it. Format did not decide task success. Source: [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04 ### Where should I link llms.txt from? From pages agents already read: the footer, a link rel=alternate in the head, or a resources section. 86% of llms.txt fetches in the traced runs arrived through a link, not a guessed path. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### What goes in an llms.txt? An H1 with the name, a one-line blockquote summary, then H2 sections listing key pages as markdown links with a short description each. Keep it lean; large sites split it per section. Source: [llmstxt.org](https://llmstxt.org/), read 2026-11-04 ### How do I make an OpenAPI file agents find? Serve the raw file, link it from docs and the home page, and list it in /.well-known/api-catalog with a service-desc link. On sites that shipped one, 97% of openapi.json fetches came through a link. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Does every site need an MCP server? No. AR-ACT applies only to products agents are expected to act on. For those, AR-ACT-10 requires an MCP server or a CLI, at least one. A purely informational site conforms without either. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Is a CLI enough instead of an MCP server? Yes for AR-ACT-10: either satisfies it. A CLI serves agents working in a shell, an MCP server serves chat and IDE hosts. Do not ship an empty surface to tick the box. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Where should the MCP server live? The spec says Streamable HTTP at /mcp, docs MCP public, product MCP behind OAuth with protected-resource metadata and PKCE. The Vercel guide calls /mcp a stable choice but not a protocol requirement. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### What is WebMCP and do I need it? A browser API where the page registers tools on document.modelContext for an in-browser agent. AgentReady v2.0 made it SHOULD for products with a web UI agents operate. Feature-detect it; Chrome offers it through an early preview program. Source: [github.com/agentready-org/standard/blob/main/CHANGELOG.md](https://github.com/agentready-org/standard/blob/main/CHANGELOG.md), read 2026-11-04 ### What OAuth files should an agent be able to find? Authorization server metadata at /.well-known/oauth-authorization-server (RFC 8414) or OIDC discovery, and protected resource metadata at /.well-known/oauth-protected-resource (RFC 9728). Cloudflare checks both separately. Source: [isitagentready.com/.well-known/agent-skills/oauth-discovery/SKILL.md](https://isitagentready.com/.well-known/agent-skills/oauth-discovery/SKILL.md), read 2026-11-04 ### What is the difference between the AI Catalog and the API catalog? The AI Catalog (AR-FIND-04) at /.well-known/ai-catalog.json lists every agent resource, MCP cards and skills included. The API catalog (RFC 9727) at /.well-known/api-catalog lists APIs with links to OpenAPI and docs. Source: [isitagentready.com/.well-known/agent-skills/ard/SKILL.md](https://isitagentready.com/.well-known/agent-skills/ard/SKILL.md), read 2026-11-04 ### Which Cloudflare checks have no AgentReady requirement? Content Signals, markdown negotiation, OAuth discovery as a standalone file, Agent Skills discovery, and items such as DNS-AID, Auth.md and Personal Agent Protocol. Pay per crawl is a Cloudflare feature but not listed as a scanner check on 2026-11-04. Source: [isitagentready.com](https://isitagentready.com/), read 2026-11-04 ### What does Content-Signal do? It states in robots.txt whether content may be used for AI training (ai-train), search (search) and AI input such as grounding (ai-input). It is a preference, not a block. Radar found 4% of scanned sites declare it. Source: [blog.cloudflare.com/agent-readiness/](https://blog.cloudflare.com/agent-readiness/), read 2026-11-04 ### Can the Verify Kit replace a scanner? No. Each kit check proves one thing a one-line request can show and says what it cannot. Full scanners run many more checks: Ora, Cloudflare isitagentready.com and is-agentic.com. Source: [blog.cloudflare.com/agent-readiness/](https://blog.cloudflare.com/agent-readiness/), read 2026-11-04 ### Why do some recipes say workaround? Hosted builders often lock robots.txt, headers or status codes. The workaround is usually a proxy in front, such as a Cloudflare Worker or Transform Rule, an edge function, or a different page type that the platform renders server-side. Source: [developers.cloudflare.com/fundamentals/reference/markdown-for-agents/](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), read 2026-11-04 ### Which spec version does this checklist follow? AgentReady v2.0, commit 273d2d7 merged on 2026-10-21, read on 2026-11-04. The spec follows semantic versioning; a major bump means a normative change, and the checklist diffs its identifiers against the spec on every refresh. Source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 ### Where does the research behind the spec come from? From the Ora research lab, which co-wrote the spec with Vercel. The published subset is 1,033 traced runs on 25 sites and 190 fetch probes under CC BY 4.0, so every number can be recomputed. Treat the lab figures as Ora's findings with their dates. Source: [github.com/agentready-org/standard/blob/main/data/README.md](https://github.com/agentready-org/standard/blob/main/data/README.md), read 2026-11-04 ### Does passing AgentReady guarantee agents will recommend my site? No. The spec defines requirements, not outcomes. Ora's lab found (published 2026-10-08) that agent-ready businesses were clearly recommended 20% of the time against 11%, an association in its sample, not a guarantee. Source: [ora.ai/blog/ax-is-the-new-aeo](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=faq), read 2026-11-04 ### Which scanner should I use to verify the whole checklist? Ora. Its catalogue names a matching check for 26 of the 31 rows, its audit needs no account and no key, and one flag turns it into a CI gate. It reads public URLs only and gives no repeat-scan figures, so keep the Verify Kit for pages it cannot reach and for the five rows it has no check for. Source: [ora.ai/api/checks](https://ora.ai/api/checks?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=faq), read 2026-11-05 ### How do I fail a build when agent readiness drops? Run npx ax@0.10 audit against the preview URL with --min-score set a few points under your current score. It exits with 1 below the floor and with 3 when Ora cannot be reached. Agent Ready's GitHub Action does the same for the Vercel spec score, on its Pro plan. Source: [registry.npmjs.org/ax](https://registry.npmjs.org/ax), read 2026-11-05 ## Sources - [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04 - [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents)read 2026-11-04 - [llmstxt.org](https://llmstxt.org/)read 2026-11-04 - [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04 - [github.com/agentready-org/standard/blob/main/CHANGELOG.md](https://github.com/agentready-org/standard/blob/main/CHANGELOG.md)read 2026-11-04 - [isitagentready.com/.well-known/agent-skills/oauth-discovery/SKILL.md](https://isitagentready.com/.well-known/agent-skills/oauth-discovery/SKILL.md)read 2026-11-04 - [isitagentready.com/.well-known/agent-skills/ard/SKILL.md](https://isitagentready.com/.well-known/agent-skills/ard/SKILL.md)read 2026-11-04 - [isitagentready.com](https://isitagentready.com/)read 2026-11-04 - [blog.cloudflare.com/agent-readiness/](https://blog.cloudflare.com/agent-readiness/)read 2026-11-04 - [developers.cloudflare.com/fundamentals/reference/markdown-for-agents/](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)read 2026-11-04 - [github.com/agentready-org/standard/blob/main/data/README.md](https://github.com/agentready-org/standard/blob/main/data/README.md)read 2026-11-04 - [ora.ai/blog/ax-is-the-new-aeo](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=faq)read 2026-11-04 - [ora.ai/api/checks](https://ora.ai/api/checks?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=faq)read 2026-11-05 - [registry.npmjs.org/ax](https://registry.npmjs.org/ax)read 2026-11-05 Edition Data 2026-11-04 Reviewed 2026-11-05 Spec AgentReady v2.0, 273d2d7 Crawl 13,071 rows, 2026-11-04 Ora 132 checks, contract 1.38.0, 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Method - [How rows, commands and rates are made](https://agentreadychecklist.com/method/) - [How an Ora check is matched to a row](https://agentreadychecklist.com/checklist/ora-checks/) - [Pass rates by rank band](https://agentreadychecklist.com/data/by-rank/) Sources and read dates sit at the foot of the page. Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.