--- title: "Cloudflare Pages and Workers: vendor profile | Agent Ready Checklist" description: "Cloudflare Pages and Workers, a host or CDN: 0 of 8 checklist rows read are met by default, with 9 sourced facts, dated plans and what it does not do." canonical: https://agentreadychecklist.com/vendors/cloudflare site: "Agent Ready Checklist" last_updated: 2026-11-05 section: "Vendors" --- # Cloudflare Pages and Workers: vendor profile | Agent Ready Checklist > Cloudflare Pages and Workers, a host or CDN: 0 of 8 checklist rows read are met by default, with 9 sourced facts, dated plans and what it does not do. Source: https://agentreadychecklist.com/vendors/cloudflare Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # Cloudflare Pages and Workers Cloudflare hosts static sites on Pages and Workers and fronts other origins as a CDN. Since 2025-07-01 its default has blocked AI crawlers unless they pay creators, and its zone settings control a managed robots.txt with Content Signals, Markdown for Agents on Pro and higher plans, and pay per crawl, which is in private beta. ## At a glance Type [Host or CDN](https://agentreadychecklist.com/vendors/by-type/host/) Operator Cloudflare Homepage [cloudflare.com](https://www.cloudflare.com), answered HTTP 200 on 2026-11-05 Checklist rows 8 of 31 read: **0** supported, 6 partial, 1 announced, 1 no statement In the crawl 2,724 of 7,545 home pages counted on 2026-11-04. Counted from the crawl's headers and generator tags, an undercount for hosted builders. Last read 2026-11-05 ## What it does for this layer - Cloudflare announced on 2025-07-01, which it called "Content Independence Day", that it was "changing the default to block AI crawlers unless they pay creators" for content. primary source, 2025-07-01: [blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), read 2026-11-05 - Cloudflare's Markdown for Agents converts HTML to Markdown at the edge when a request asks for text/markdown in the Accept header, on zones where the feature is enabled. primary source: [developers.cloudflare.com/fundamentals/reference/markdown-for-agents/](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), read 2026-11-05 - Cloudflare states "Markdown for Agents is available to Pro, Business and Enterprise plans, and SSL for SaaS customers at no cost". primary source: [developers.cloudflare.com/fundamentals/reference/markdown-for-agents/](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), read 2026-11-05 - Markdown for Agents responses carry vary: accept and a content-signal header, so caches keep separate HTML and markdown variants. primary source: [developers.cloudflare.com/fundamentals/reference/markdown-for-agents/](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), read 2026-11-05 - Cloudflare's developer docs navigation labels Markdown for Agents as Beta. observed: [developers.cloudflare.com/fundamentals/reference/markdown-for-agents/](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), read 2026-11-05 - Cloudflare prepends managed directives and its Content Signals Policy to a site's robots.txt only when Bot Preference Sync is on and at least one Search, Agent or Training policy is set to block. primary source: [developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), read 2026-11-05 - With every AI crawler policy set to allow, Cloudflare serves the site's robots.txt unchanged, according to its managed robots.txt docs. primary source: [developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), read 2026-11-05 - Cloudflare AI Crawl Control lets a site owner set allow or block rules per crawler and set pay per crawl pricing, which is in private beta. primary source: [developers.cloudflare.com/ai-crawl-control/](https://developers.cloudflare.com/ai-crawl-control/), read 2026-11-05 - Cloudflare Pages sites can contain up to 20,000 files on the Free plan, and Pages builds time out after 20 minutes. primary source: [developers.cloudflare.com/pages/platform/limits/](https://developers.cloudflare.com/pages/platform/limits/), read 2026-11-05 ## Products and plans | Product | What it is | Price | As of | | --- | --- | --- | --- | | Cloudflare Pages | Static and full-stack site hosting with Git builds. | Free plan limits; paid plans raise them; developers.cloudflare.com/pages/platform/limits/ | 2026-11-05 | | Cloudflare Workers | Serverless runtime that also serves static assets. | No price read | not read | | AI Crawl Control | Per-crawler allow, block and pay per crawl controls for a zone. | No price read | not read | ## Standards | Row | What it asks | Status | What the vendor's pages say | Source | | --- | --- | --- | --- | --- | | AR-FIND-01 | Do not block agents | partial | Cloudflare changed its default on 2025-07-01 to block AI crawlers unless they pay creators. | blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ | | AR-FIND-04 | AI Catalog at .well-known | partial | Pages and Workers serve any uploaded file, including under /.well-known/. | developers.cloudflare.com/pages/platform/limits/ | | AR-READ-01 | Answer in the initial HTML | no statement | Initial HTML depends on the framework deployed to Pages or Workers. | none | | AR-READ-02 | Real HTTP status codes | partial | Workers static assets can be set to SPA mode, which answers unknown paths with index.html; 404 handling depends on that setting. | developers.cloudflare.com/workers/static-assets/routing/single-page-application/ | | CF-CONTENT-SIGNALS | Content-Signal line | partial | Managed robots.txt adds the Content Signals Policy when Bot Preference Sync blocks a category. | developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ | | CF-MARKDOWN-NEGOTIATION | Markdown by Accept | partial | Markdown for Agents converts HTML to markdown on Pro and higher plans once enabled. | developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ | | CF-PAY-PER-CRAWL | Pay per crawl, HTTP 402 | announced | Pay per crawl pricing is in private beta. | developers.cloudflare.com/ai-crawl-control/ | | VERCEL-NEGOTIATION-HEADERS | Markdown headers | partial | Markdown for Agents responses carry vary: accept. | developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ | ## Checks on its own site | Reading | Result | Date | Source | | --- | --- | --- | --- | | Cloudflare scan of cloudflare.com | 60 of 100, band L4 Agent-Integrated | 2026-11-05 | isitagentready.com | | Crawl row for cloudflare.com | 10 of 10 checks pass; Tranco rank 2 | 2026-11-04 | agentreadychecklist.com/data/ | | Home pages counted in the crawl | 2,724 of 7,545; Counted from the shared crawl of 2026-11-04 by cf-ray response header (served through Cloudflare's network, not necessarily hosted on Pages or Workers); the generator rule only sees tags whose name attribute comes first, so it undercounts | 2026-11-04 | agentreadychecklist.com/data/ | | Check | Result | Verify Kit check | | --- | --- | --- | | robots.txt allows all 17 agent tokens at / | pass | robots-verdicts | | robots.txt blocks no search or user-fetch token | pass | robots-verdicts | | Home page has an h1 and 50 words in the initial HTML | pass | initial-html | | A missing path returns 404 or 410 | pass | status-codes | | /llms.txt exists | pass | llms-linked | | Home page links to llms.txt | pass | llms-linked | | Accept: text/markdown returns text/markdown | pass | markdown-negotiation | | Home page carries JSON-LD that parses | pass | json-ld | | OpenAPI linked from the home page or served at /openapi.json | pass | openapi | | /.well-known/mcp.json serves JSON | pass | mcp-or-cli | ## What it does not do - Cloudflare does not convert pages to markdown on the Free plan or without the zone setting turned on. - Cloudflare does not add Content Signals to robots.txt while every AI crawler policy is set to allow. ## Reading notes Cloudflare is also profiled on other sites of the network; this record covers what a builder meets when hosting on or proxying through it. ## Related pages on this site Recipes [Static hosting](https://agentreadychecklist.com/platforms/static-hosting/) ([checklist view](https://agentreadychecklist.com/platform/static-hosting/checklist/)) Same recipe page [AWS Amplify Hosting](https://agentreadychecklist.com/vendors/aws-amplify/), [Eleventy (Build Awesome)](https://agentreadychecklist.com/vendors/eleventy/), [Firebase Hosting](https://agentreadychecklist.com/vendors/firebase-hosting/), [Gatsby](https://agentreadychecklist.com/vendors/gatsby/), [GitHub Pages](https://agentreadychecklist.com/vendors/github-pages/), [Hugo](https://agentreadychecklist.com/vendors/hugo/), [Jekyll](https://agentreadychecklist.com/vendors/jekyll/), [Netlify](https://agentreadychecklist.com/vendors/netlify/), [Render](https://agentreadychecklist.com/vendors/render/), [Vercel](https://agentreadychecklist.com/vendors/vercel/) Same type [Hosts and CDNs](https://agentreadychecklist.com/vendors/by-type/host/), [all vendors](https://agentreadychecklist.com/vendors/) ## Sources - [cloudflare.com](https://www.cloudflare.com)read 2026-11-05 - [blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/)read 2026-11-05 - [developers.cloudflare.com/fundamentals/reference/markdown-for-agents/](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)read 2026-11-05 - [developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)read 2026-11-05 - [developers.cloudflare.com/ai-crawl-control/](https://developers.cloudflare.com/ai-crawl-control/)read 2026-11-05 - [developers.cloudflare.com/pages/platform/limits/](https://developers.cloudflare.com/pages/platform/limits/)read 2026-11-05 - [developers.cloudflare.com/workers/static-assets/routing/single-page-application/](https://developers.cloudflare.com/workers/static-assets/routing/single-page-application/)read 2026-11-05 - [isitagentready.com](https://isitagentready.com/)read 2026-11-05 - [agentreadychecklist.com/data/](https://agentreadychecklist.com/data/)read 2026-11-04 Edition Data 2026-11-04 Reviewed 2026-11-05 Spec AgentReady v2.0, 273d2d7 Crawl 13,071 rows, 2026-11-04 Ora 132 checks, contract 1.38.0, 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Method - [How rows, commands and rates are made](https://agentreadychecklist.com/method/) - [How an Ora check is matched to a row](https://agentreadychecklist.com/checklist/ora-checks/) - [Pass rates by rank band](https://agentreadychecklist.com/data/by-rank/) Sources and read dates sit at the foot of the page. Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.