---
title: "robots.txt defined | Agent Ready Checklist"
description: "robots.txt: Plain-text file at the site root that tells compliant crawlers which paths to skip, standardised as RFC 9309."
canonical: https://agentreadychecklist.com/glossary/robots-txt
site: "Agent Ready Checklist"
last_updated: 2026-11-05
section: "Glossary"
---

# robots.txt defined | Agent Ready Checklist

> robots.txt: Plain-text file at the site root that tells compliant crawlers which paths to skip, standardised as RFC 9309.

Source: https://agentreadychecklist.com/glossary/robots-txt
Site: Agent Ready Checklist
Last reviewed: 2026-11-05
Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search)
Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free).

# robots.txt

Plain-text file at the site root that tells compliant crawlers which paths to skip, standardised as RFC 9309. Crawlers obey the most specific matching user-agent group. Example: User-agent: \* Disallow: /search/ Sitemap: https://example.com/sitemap.xml. Related rows: AR-FIND-01, AR-FIND-03, CF-CONTENT-SIGNALS, VERCEL-ROBOTS-EXCLUSIONS. RFC 9309 lets robots.txt carry Sitemap lines alongside the groups. The Verify Kit check robots-verdicts tests robots.txt.

## Definition

Plain-text file at the site root that tells compliant crawlers which paths to skip, standardised as RFC 9309. Crawlers obey the most specific matching user-agent group.

## Example

```text
User-agent: *
Disallow: /search/

Sitemap: https://example.com/sitemap.xml
```

## Where it appears on the checklist

-   [AR-FIND-01](https://agentreadychecklist.com/checklist/ar-find-01/)Do not block agents: Do not block agent and crawler access to public content, in robots.txt or at the network layer (WAF, bot manager).
-   [AR-FIND-03](https://agentreadychecklist.com/checklist/ar-find-03/)Sitemap listed in robots.txt: List the site pages in a sitemap.xml and reference it from robots.txt with a Sitemap: line.
-   [CF-CONTENT-SIGNALS](https://agentreadychecklist.com/checklist/cf-content-signals/)Content-Signal line: Declare AI usage preferences with a Content-Signal line in robots.txt (ai-train, search, ai-input).
-   [VERCEL-ROBOTS-EXCLUSIONS](https://agentreadychecklist.com/checklist/vercel-robots-exclusions/)Robots exclusions: Use robots.txt only for deliberate exclusions: a public site does not need a robots.txt to permit crawling, so audit existing rules rather than adding agent-specific ones by default.
-   [Guide](https://agentreadychecklist.com/guides/one-day-agent-readiness-sprint/)The one-day agent-readiness sprint
-   [Guide](https://agentreadychecklist.com/guides/test-with-a-plain-fetch-client/)How to test with a plain-fetch client
-   [Guide](https://agentreadychecklist.com/guides/read-server-logs-for-agent-user-agents/)How to read your server logs for agent user agents

## Asked about it

### Does llms.txt control which crawlers can read my site?

No. llms.txt is an index for agents to read; it grants or denies nothing. Access is decided by robots.txt for compliant crawlers and by your WAF or bot manager for everything else.

### Does robots.txt stop an AI agent browsing with a user session?

No. robots.txt is advisory and applies to crawlers. The spec notes that user-initiated fetchers may not consult it at all, and an agent driving a browser in a user session behaves like the user.

## Sources

-   [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04
-   [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04
-   [llmstxt.org](https://llmstxt.org/)read 2026-11-04

Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.