---
title: "User-agent group defined | Agent Ready Checklist"
description: "User-agent group: A block in robots.txt starting with one or more User-agent lines. Related rows: CF-CONTENT-SIGNALS. Sourced example from rfc-editor.org."
canonical: https://agentreadychecklist.com/glossary/user-agent-group
site: "Agent Ready Checklist"
last_updated: 2026-11-05
section: "Glossary"
---

# User-agent group defined | Agent Ready Checklist

> User-agent group: A block in robots.txt starting with one or more User-agent lines. Related rows: CF-CONTENT-SIGNALS. Sourced example from rfc-editor.org.

Source: https://agentreadychecklist.com/glossary/user-agent-group
Site: Agent Ready Checklist
Last reviewed: 2026-11-05
Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search)
Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free).

# User-agent group

A block in robots.txt starting with one or more User-agent lines. A crawler uses only the most specific group that names it, ignoring the \* group. Example: User-agent: GPTBot Disallow: / # GPTBot now ignores the \* group. Related row: CF-CONTENT-SIGNALS (Content-Signal line). RFC 9309 merges two groups that match the same crawler into one group.

## Definition

A block in robots.txt starting with one or more User-agent lines. A crawler uses only the most specific group that names it, ignoring the \* group.

## Example

```text
User-agent: GPTBot
Disallow: /
# GPTBot now ignores the * group
```

## Where it appears on the checklist

-   [CF-CONTENT-SIGNALS](https://agentreadychecklist.com/checklist/cf-content-signals/)Content-Signal line: Declare AI usage preferences with a Content-Signal line in robots.txt (ai-train, search, ai-input).

## Sources

-   [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04

Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.