---
title: "Kit check robots-verdicts | Agent Ready Checklist"
description: "Kit check robots-verdicts: robots.txt verdict per agent token. The command that runs it, the evidence it prints and what it cannot prove."
canonical: https://agentreadychecklist.com/kit/robots-verdicts
site: "Agent Ready Checklist"
last_updated: 2026-11-05
section: "Kit"
---

# Kit check robots-verdicts | Agent Ready Checklist

> Kit check robots-verdicts: robots.txt verdict per agent token. The command that runs it, the evidence it prints and what it cannot prove.

Source: https://agentreadychecklist.com/kit/robots-verdicts
Site: Agent Ready Checklist
Last reviewed: 2026-11-05
Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search)
Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free).

# Kit check: robots.txt verdict per agent token

The robots-verdicts check tests this: robots.txt verdict per agent token. It maps to AR-FIND-01, CF-CONTENT-SIGNALS, VERCEL-ROBOTS-EXCLUSIONS. Run it alone with --only=robots-verdicts. It makes read requests only, and each evidence line says what the check saw. On the pass fixture it reports: 17 of 17 tokens allowed; Content-Signal yes, Sitemap lines 1.

## Run it

```bash
curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=robots-verdicts
```

```bash
node verify.mjs https://example.com --only=robots-verdicts
```

## What it tests

Parses /robots.txt per RFC 9309 group matching and prints allow or disallow for / for each of 17 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Applebot, Applebot-Extended, Bingbot, Amazonbot, Meta-ExternalAgent, DuckAssistBot, MistralAI-User), plus any Content-Signal and Sitemap lines. FAIL when a search or user-initiated token is disallowed at /.

## What it cannot prove

User-initiated fetchers may ignore robots.txt; a disallow for a training crawler can be a deliberate, valid choice; the parser checks / only unless given a path.

## Requirements it maps to

-   [AR-FIND-01](https://agentreadychecklist.com/checklist/ar-find-01/)Do not block agents
-   [CF-CONTENT-SIGNALS](https://agentreadychecklist.com/checklist/cf-content-signals/)Content-Signal line
-   [VERCEL-ROBOTS-EXCLUSIONS](https://agentreadychecklist.com/checklist/vercel-robots-exclusions/)Robots exclusions

## Evidence on the fixtures

The pass fixture

PASS 17 of 17 tokens allowed; Content-Signal yes, Sitemap lines 1

The js-only fixture

PASS 17 of 17 tokens allowed; Content-Signal no, Sitemap lines 0

The 403 fixture

FAIL robots.txt returned 403 (access denied or unexpected)

## The one-line sketch

The shipped script is more careful than this sketch, which shows the idea in one line of shell.

```bash
curl -s -A "$UA" "$ORIGIN/robots.txt" > robots.txt; node -e "/* group by User-agent, pick most specific match per token, longest-match Allow/Disallow for / */" robots.txt; grep -i "^content-signal:" robots.txt
```

## Sources

-   [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04

Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.