---
title: "Training crawler defined | Agent Ready Checklist"
description: "Training crawler: A crawler that collects pages to train models, such as GPTBot or ClaudeBot. Blocking it does not remove a site from AI search answers."
canonical: https://agentreadychecklist.com/glossary/training-crawler
site: "Agent Ready Checklist"
last_updated: 2026-11-05
section: "Glossary"
---

# Training crawler defined | Agent Ready Checklist

> Training crawler: A crawler that collects pages to train models, such as GPTBot or ClaudeBot. Blocking it does not remove a site from AI search answers.

Source: https://agentreadychecklist.com/glossary/training-crawler
Site: Agent Ready Checklist
Last reviewed: 2026-11-05
Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search)
Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free).

# Training crawler

A crawler that collects pages to train models, such as GPTBot or ClaudeBot. Blocking it does not remove a site from AI search answers. Example: User-agent: GPTBot Disallow: /. Related row: CF-PAY-PER-CRAWL (Pay per crawl, HTTP 402). A crawler that pays receives a 200, and one that does not receives 402 Payment Required with pricing.

## Definition

A crawler that collects pages to train models, such as GPTBot or ClaudeBot. Blocking it does not remove a site from AI search answers.

## Example

```text
User-agent: GPTBot
Disallow: /
```

## Where it appears on the checklist

-   [CF-PAY-PER-CRAWL](https://agentreadychecklist.com/checklist/cf-pay-per-crawl/)Pay per crawl, HTTP 402: If you charge crawlers, answer with HTTP 402 Payment Required and pricing headers, or 200 when the crawler presents payment intent.

## Asked about it

### Should I block GPTBot?

That is a training-data decision, separate from agent access. GPTBot and ClaudeBot are training crawlers; OAI-SearchBot and Claude-SearchBot build search indexes; ChatGPT-User and Claude-User fetch for a user. Blocking a training crawler does not drop you from agent answers, if the other groups stay open.

## Sources

-   [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04
-   [developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/)read 2026-11-04

Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.