--- title: "Read server logs for agent user agents | Agent Ready Checklist" description: "How to read your server logs for agent user agents. Find real agent requests by IP range and user agent, and confirm they get 200 with the answer." canonical: https://agentreadychecklist.com/guides/read-server-logs-for-agent-user-agents site: "Agent Ready Checklist" last_updated: 2026-11-05 date: 2026-11-04 section: "Guides" --- # Read server logs for agent user agents | Agent Ready Checklist > How to read your server logs for agent user agents. Find real agent requests by IP range and user agent, and confirm they get 200 with the answer. Source: https://agentreadychecklist.com/guides/read-server-logs-for-agent-user-agents Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # How to read your server logs for agent user agents Find real agent requests by IP range and user agent, and confirm they get 200 with the answer. It covers 2 requirements: AR-FIND-01, AR-READ-02. The spec calls server logs the dependable test and says OpenAI publishes separate per-bot IP lists while Anthropic publishes one combined list. ## Before you start You need access to the web server or CDN logs for the last week and the published IP ranges of the operators you care about. ## Steps 1. **[AR-FIND-01: Do not block agents.](https://agentreadychecklist.com/checklist/ar-find-01/)** Do not block agent and crawler access to public content, in robots.txt or at the network layer (WAF, bot manager). ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=reach-as-agent,robots-verdicts ``` Watch for: A named group (User-agent: GPTBot, Disallow: /) left over from 2023 that overrides a permissive wildcard. 2. **[AR-READ-02: Real HTTP status codes.](https://agentreadychecklist.com/checklist/ar-read-02/)** Return accurate HTTP status codes: 404 or a real redirect for missing paths, 429 with Retry-After when throttling; never 200 for a missing page and never a silent challenge page in place of 429. ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=status-codes ``` Watch for: SPA hosts rewriting every path to index.html with 200. 3. **Count agent requests by status.** In a combined log format, count requests whose user agent names an agent token and group them by status code. A 403 or a 200 with a challenge page is the failure to look for. ```bash grep -iE "GPTBot|ClaudeBot|Claude-User|ChatGPT-User|OAI-SearchBot|PerplexityBot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn ``` ## How you know it is done You are done when you can list, for each agent user agent, how many requests arrived, from which ranges, and which status they received. ## The facts behind it - The spec calls server logs the dependable test and says OpenAI publishes separate per-bot IP lists while Anthropic publishes one combined list.primary source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 - Model providers run separate crawlers per job: training (GPTBot, ClaudeBot), search index (OAI-SearchBot, Claude-SearchBot) and user-initiated (ChatGPT-User, Claude-User).primary source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 - User-initiated fetchers may not consult robots.txt at all; the spec lists this as an open question.primary source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 - Cloudflare Web Bot Auth lets a bot sign requests; sites verify with keys published at /.well-known/http-message-signatures-directory.primary source, 2026-04-17: [blog.cloudflare.com/agent-readiness/](https://blog.cloudflare.com/agent-readiness/), read 2026-11-04 - Signed requests carry Signature-Agent and Signature-Input headers.primary source: [isitagentready.com/.well-known/agent-skills/web-bot-auth/SKILL.md](https://isitagentready.com/.well-known/agent-skills/web-bot-auth/SKILL.md), read 2026-11-04 ## Sources - [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04 - [blog.cloudflare.com/agent-readiness/](https://blog.cloudflare.com/agent-readiness/)read 2026-11-04 - [isitagentready.com/.well-known/agent-skills/web-bot-auth/SKILL.md](https://isitagentready.com/.well-known/agent-skills/web-bot-auth/SKILL.md)read 2026-11-04 Edition Data 2026-11-04 Reviewed 2026-11-05 Spec AgentReady v2.0, 273d2d7 Crawl 13,071 rows, 2026-11-04 Ora 132 checks, contract 1.38.0, 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Method - [How rows, commands and rates are made](https://agentreadychecklist.com/method/) - [How an Ora check is matched to a row](https://agentreadychecklist.com/checklist/ora-checks/) - [Pass rates by rank band](https://agentreadychecklist.com/data/by-rank/) Sources and read dates sit at the foot of the page. Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.