--- title: "AR-FIND-01: Do not block agents | Agent Ready Checklist" description: "AR-FIND-01 (MUST, Find layer): Do not block agents. What it asks, the evidence, the command that verifies it and the fix on 19 platforms." canonical: https://agentreadychecklist.com/checklist/ar-find-01 site: "Agent Ready Checklist" last_updated: 2026-11-05 section: "Checklist" --- # AR-FIND-01: Do not block agents | Agent Ready Checklist > AR-FIND-01 (MUST, Find layer): Do not block agents. What it asks, the evidence, the command that verifies it and the fix on 19 platforms. Source: https://agentreadychecklist.com/checklist/ar-find-01 Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # AR-FIND-01: Do not block agents AR-FIND-01 is a MUST on the find layer: Do not block agent and crawler access to public content, in robots.txt or at the network layer (WAF, bot manager). It applies to every public site. 19 of 19 platforms have a recipe below. The kit check reach-as-agent and robots-verdicts tests it with one command. ## The requirement Identifier AR-FIND-01 Level MUST Layer Find Applies to every public site Checked by Ora, matched on this site, Cloudflare robotsTxt, Cloudflare robotsTxtAiRules, is-agentic.com, Verify Kit reach-as-agent, Verify Kit robots-verdicts Ora check [robots-ai-policy-quality, robots-agent-user-policy, agent-crawler-reachability, bot-detection](#ora) The statement Do not block agent and crawler access to public content, in robots.txt or at the network layer (WAF, bot manager). Throttle with 429, not a challenge page. ## Why - Ora's lab found (published 2026-10-21): blocking was one of only two changes that broke an agent; in the controlled experiment a 403 to agent user agents left both the plain-fetch and the JavaScript client at 0 of 5.vendor claim, 2026-10-21: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 - The 19-configuration fetch experiment had two bot-block configurations (bot-block and hostile) and both returned 0 of 5 for both clients.primary source, 2026-08-28: [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04 - v2.0 extended AR-FIND-01 from robots.txt to bot protection: let verified agents through the WAF or bot manager and throttle with 429 instead of a challenge page.primary source, 2026-10-20: [github.com/agentready-org/standard/blob/main/CHANGELOG.md](https://github.com/agentready-org/standard/blob/main/CHANGELOG.md), read 2026-11-04 - A more specific group such as User-agent: GPTBot overrides the \* group, so a permissive wildcard does not help a bot that has its own Disallow group.primary source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 - RFC 9309 defines the robots.txt format; crawlers match the most specific user-agent group and apply only that group.primary source, 2022-09-01: [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309), read 2026-11-04 ## How to verify ### [Reachable with an agent user agent](https://agentreadychecklist.com/kit/reach-as-agent/) ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=reach-as-agent ``` Fetches the URL with Claude-User and GPTBot user agents and with the kit UA; PASS when every final status is 200 after at most 5 redirects and the body is not a challenge page (no cf-mitigated header, no "Just a moment"). **It cannot prove:** Bot managers decide on source IP and TLS fingerprint more than UA; a PASS from a laptop does not prove the real crawler from its published IP ranges gets through. Check server logs for that. ### [robots.txt verdict per agent token](https://agentreadychecklist.com/kit/robots-verdicts/) ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=robots-verdicts ``` Parses /robots.txt per RFC 9309 group matching and prints allow or disallow for / for each of 17 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Applebot, Applebot-Extended, Bingbot, Amazonbot, Meta-ExternalAgent, DuckAssistBot, MistralAI-User), plus any Content-Signal and Sitemap lines. FAIL when a search or user-initiated token is disallowed at /. **It cannot prove:** User-initiated fetchers may ignore robots.txt; a disallow for a training crawler can be a deliberate, valid choice; the parser checks / only unless given a path. ### The one-line version curl -s -o /dev/null -w "%{http\_code}" -A "Claude-User/1.0" must print 200, then confirm in server logs that requests from the published OpenAI and Anthropic IP ranges get 200. ## Verify with Ora 4 checks in Ora's catalogue (contract 1.38.0, read 2026-11-05) read this row. Each entry says what the check looks at and why this page lists it. - `robots-ai-policy-quality` robots.txt AI crawler policy Reads whether robots.txt lets the AI search crawlers OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages. Blocking training crawlers such as GPTBot or ClaudeBot never lowers it. Discovery, 2 points, required; catalogue applicability domain-only. Listed because: same name, `robots.txt`. - `robots-agent-user-policy` robots.txt agent-user policy Reads the permission robots.txt declares for the homepage to ChatGPT-User, Claude-User and Perplexity-User. A policy that cannot be verified reads as not applicable. Access, 2 points, required; catalogue applicability domain-only. Listed because: same name, `robots.txt`. - `agent-crawler-reachability` Agent crawler reachability Tests whether the homepage is reachable to ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot. Access, 2 points, recommended; catalogue applicability domain-only. Listed because: same name, `crawler`. - `bot-detection` Not blocked by bot detection Tests whether AI agents can reach the site without bot protection turning them away. Access, 2 points, required; catalogue applicability domain-only. Listed because: same name, `bot`. ### Run just these checks ```bash curl -s -X POST https://ora.ai/api/scan/checks -H 'content-type: application/json' -d '{"url":"https://example.com","checkIds":["robots-ai-policy-quality","robots-agent-user-policy","agent-crawler-reachability","bot-detection"]}' ``` This calls the endpoint Ora documents for re-verifying a fix. It sends no cache, returns a result for each id and no total score, and spends one scan from the daily budget. Replace the URL with yours. [How to re-verify one fix](https://agentreadychecklist.com/guides/re-verify-one-fix-with-ora/). ### Or audit the whole site ```bash npx ax@0.10 audit https://example.com --min-score 70 ``` Exit code 1 means the score is below the number you passed, so the same line gates a deploy: [the CI recipe](https://agentreadychecklist.com/guides/gate-deploys-on-agent-readiness-in-ci/). [Why this site recommends Ora, and where it falls short](https://agentreadychecklist.com/verify-with-ora/). ## In the crawl This row has 2 crawl rates. Each panel splits the 2026-11-04 crawl into ten bands of 1,000 ranks, on a 0 to 100 percent scale. robots.txt allows all 17 agent tokens at / 71.2% overall robots.txt blocks no search or user-fetch token 73.2% overall Reachable sites in the Tranco top 10,000 on which the check ran, 2026-11-04. Left is ranks 1 to 1,000. Numbers behind the chart | Rate | 1 to 1,000 | 1,001 to 2,000 | 2,001 to 3,000 | 3,001 to 4,000 | 4,001 to 5,000 | 5,001 to 6,000 | 6,001 to 7,000 | 7,001 to 8,000 | 8,001 to 9,000 | 9,001 to 10,000 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | robots.txt allows all 17 agent tokens at / | 67% | 66% | 74% | 74% | 73% | 73% | 69% | 75% | 71% | 70% | | robots.txt blocks no search or user-fetch token | 70% | 69% | 76% | 76% | 74% | 74% | 71% | 77% | 73% | 71% | ## Fix it on your platform ### Next.js (App Router)recipe In app/robots.ts return rules: \[{ userAgent: "\*", allow: "/" }\] and add a sitemap URL. Do not add per-bot disallow groups for GPTBot or ClaudeBot unless you mean to opt out of training. Check the WAF on your host separately. Next.js 16.x (npm latest 16.4.5, read 2026-11-04); docs at nextjs.org/docs/app. Read 2026-11-04. [nextjs.org/docs/app/api-reference/file-conventions/metadata/robots](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots). [All Next.js (App Router) recipes](https://agentreadychecklist.com/platforms/nextjs-app-router/#ar-find-01) ### Astrorecipe Write public/robots.txt with "User-agent: \*" and "Allow: /". Avoid named disallow groups for agent tokens. Check the host WAF separately. Astro 7.x (npm latest astro 7.4.5, @astrojs/sitemap 3.7.4, read 2026-11-04). Read 2026-11-04. [docs.astro.build/en/basics/project-structure/](https://docs.astro.build/en/basics/project-structure/). [All Astro recipes](https://agentreadychecklist.com/platforms/astro/#ar-find-01) ### Nuxtrecipe Write public/robots.txt with "User-agent: \*" and "Allow: /", or configure the Robots module without per-bot disallows. Check the host WAF separately. Nuxt 4.x (npm latest nuxt 4.6.4, @nuxtjs/robots 7.0.2, @nuxtjs/sitemap 9.1.0, read 2026-11-04). Read 2026-11-04. [nuxt.com/modules/robots](https://nuxt.com/modules/robots). [All Nuxt recipes](https://agentreadychecklist.com/platforms/nuxt/#ar-find-01) ### SvelteKitrecipe Write static/robots.txt with "User-agent: \*" and "Allow: /". Check the host WAF separately. SvelteKit (npm latest @sveltejs/kit 3.1.1, read 2026-11-04); docs at svelte.dev/docs/kit. Read 2026-11-04. [svelte.dev/docs/kit/project-structure](https://svelte.dev/docs/kit/project-structure). [All SvelteKit recipes](https://agentreadychecklist.com/platforms/sveltekit/#ar-find-01) ### Remix and React Router (framework mode)recipe Write public/robots.txt with "User-agent: \*" and "Allow: /", or return it from a robots\[.\]txt resource route. Check the host WAF separately. React Router framework mode (npm latest react-router 8.5.1; Remix v2 @remix-run/node 2.17.5, read 2026-11-04); docs at reactrouter.com. Read 2026-11-04. [reactrouter.com/how-to/resource-routes](https://reactrouter.com/how-to/resource-routes). [All Remix and React Router (framework mode) recipes](https://agentreadychecklist.com/platforms/remix-react-router/#ar-find-01) ### Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront)recipe Upload robots.txt with "User-agent: \*" and "Allow: /". On Cloudflare Pages also check the zone Bot Fight Mode and AI Crawl Control settings, which apply before your file. Host documentation as published on 2026-11-04 (no version numbers; config files \_headers, \_redirects, vercel.json, 404.html). Read 2026-11-04. [developers.cloudflare.com/pages/configuration/serving-pages/](https://developers.cloudflare.com/pages/configuration/serving-pages/). [All Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront) recipes](https://agentreadychecklist.com/platforms/static-hosting/#ar-find-01) ### WordPress and Elementorrecipe Leave Settings > Reading > Search engine visibility unchecked. Keep the default `User-agent: *` group. Audit security and caching plugins and the host WAF for rules that return 403 or a challenge to ClaudeBot, GPTBot, OAI-SearchBot or Claude-User; fetch a page with `curl -A 'Claude-User/1.0'` and expect 200. WordPress 7.1.4 (core release from the version-check API), Elementor 4.3.6 (plugin directory, updated 2026-11-02), Yoast SEO 28.8 noted for llms.txt; docs read 2026-11-04. Read 2026-11-04. [developer.wordpress.org/reference/functions/do\_robots/](https://developer.wordpress.org/reference/functions/do_robots/). [All WordPress and Elementor recipes](https://agentreadychecklist.com/platforms/wordpress-elementor/#ar-find-01) ### Shopify (Online Store 2.0 themes)recipe Keep Shopify's default `User-agent: *` rules. If you add robots.txt.liquid, do not add Disallow groups for AI crawlers. If a third-party CDN or bot rule sits in front, confirm `curl -A 'Claude-User/1.0' https://shop.example/` and `/.well-known/ucp` both return 200. Hosted SaaS, Online Store 2.0 Liquid themes, docs read 2026-11-04; storefront behaviour probed live on one Shopify store the same day. Read 2026-11-04. [shopify.dev/docs/storefronts/themes/seo/robots-txt](https://shopify.dev/docs/storefronts/themes/seo/robots-txt). [All Shopify (Online Store 2.0 themes) recipes](https://agentreadychecklist.com/platforms/shopify/#ar-find-01) ### Webflowrecipe In Site settings > SEO > Indexing keep rules to deliberate exclusions; do not paste generic 'block AI bots' lists. Republish and check `curl -A 'Claude-User/1.0' -o /dev/null -w '%{http_code}' https://example.com/` prints 200. Hosted SaaS, help center articles read 2026-11-04 (updated dates 2024-10-14 to 2026-10-29). Read 2026-11-04. [help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules](https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules). [All Webflow recipes](https://agentreadychecklist.com/platforms/webflow/#ar-find-01) ### Framerrecipe Keep the generated robots.txt unless you need exclusions; if you upload a custom one, keep `User-agent: *` with only deliberate Disallow lines. Framer's doc lists GPTBot, PerplexityBot and Google-Extended among crawlers it tests with. Hosted SaaS, help articles read 2026-11-04 (marked updated 2026-09-15). Read 2026-11-04. [framer.com/help/articles/make-site-readable-by-ai-agents/](https://www.framer.com/help/articles/make-site-readable-by-ai-agents/). [All Framer recipes](https://agentreadychecklist.com/platforms/framer/#ar-find-01) ### Wix (Wix Editor and Wix Studio)recipe Leave robots.txt at the Wix default; do not paste the AI-blocking sample from Wix's 'Blocking AI Crawlers' article (it disallows GPTBot and ChatGPT-User). Check with `curl -A 'Claude-User/1.0' -o /dev/null -w '%{http_code}' https://example.com/`. Hosted SaaS, help center read 2026-11-04. Read 2026-11-04. [support.wix.com/en/article/blocking-ai-crawlers-from-your-site](https://support.wix.com/en/article/blocking-ai-crawlers-from-your-site). [All Wix (Wix Editor and Wix Studio) recipes](https://agentreadychecklist.com/platforms/wix/#ar-find-01) ### Squarespace (version 7.1)recipe Settings > Crawlers: leave 'Block known artificial intelligence crawlers' unchecked. Checking it writes Disallow groups for GPTBot, ClaudeBot, Applebot-Extended and others into robots.txt. Hosted SaaS, version 7.1 sites, help center read 2026-11-04 (articles updated 2026-07-17 to 2026-11-03). Read 2026-11-04. [support.squarespace.com/hc/en-us/articles/38299115788813-Optimize-your-site-for-AI-powered-search-engines](https://support.squarespace.com/hc/en-us/articles/38299115788813-Optimize-your-site-for-AI-powered-search-engines). [All Squarespace (version 7.1) recipes](https://agentreadychecklist.com/platforms/squarespace/#ar-find-01) ### HubSpot CMS (Content Hub)recipe Settings > Content > Pages > Default settings for all domains > SEO & Crawlers > Robots.txt: keep `User-agent: *` with only deliberate Disallow lines. Pages on hs-sites system domains are always no-index, so test the connected domain. Hosted SaaS, Content Hub, knowledge base and developer docs read 2026-11-04. Read 2026-11-04. [knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results](https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results). [All HubSpot CMS (Content Hub) recipes](https://agentreadychecklist.com/platforms/hubspot-cms/#ar-find-01) ### Docusaurusrecipe Create static/robots.txt: User-agent: \* Allow: / Sitemap: https://example.com/sitemap.xml Check the host or CDN bot rules separately; Docusaurus does not control the WAF. 3.10.2 (npm @docusaurus/core latest, read 2026-11-04). Read 2026-11-04. [docusaurus.io/docs/static-assets](https://docusaurus.io/docs/static-assets). [All Docusaurus recipes](https://agentreadychecklist.com/platforms/docusaurus/#ar-find-01) ### Mintlifyrecipe Keep the generated robots.txt. If you add a custom robots.txt at the project root, do not block GPTBot, ClaudeBot or PerplexityBot. Fully authenticated sites redirect robots.txt to login, so keep public docs public. Hosted SaaS, docs read 2026-11-04. Read 2026-11-04. [mintlify.com/docs/optimize/seo](https://mintlify.com/docs/optimize/seo). [All Mintlify recipes](https://agentreadychecklist.com/platforms/mintlify/#ar-find-01) ### GitBookrecipe Default robots.txt allows all agents. Check your own CDN or WAF if the docs sit behind a custom domain proxy. Hosted SaaS, docs read 2026-11-04. Read 2026-11-04. [gitbook.com/docs/publish/seo](https://gitbook.com/docs/publish/seo). [All GitBook recipes](https://agentreadychecklist.com/platforms/gitbook/#ar-find-01) ### ReadMerecipe Leave the generated robots.txt open. In AI > AI Discoverability set Content Signals instead of blocking crawlers. If you add a reverse proxy, do not add bot challenges in front of it. Hosted SaaS, docs read 2026-11-04. Read 2026-11-04. [docs.readme.com/main/docs/ai-discoverability](https://docs.readme.com/main/docs/ai-discoverability). [All ReadMe recipes](https://agentreadychecklist.com/platforms/readme/#ar-find-01) ### Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront)recipe Business Manager: Merchant Tools > Site > SEO & Discoverability > Robots > Define an instance type-specific robots.txt > All spiders are allowed (Production) or Custom with AI crawler entries uncommented to Allow. Click Apply, then invalidate the static content cache. Hosted SaaS (B2C Commerce help release 264.0.0); PWA Kit retail-react-app 10.3.0 and pwa-kit-runtime 3.21.0 on npm, read 2026-11-04. Read 2026-11-04. [help.salesforce.com/s/articleView](https://help.salesforce.com/s/articleView?id=cc.b2c_generating_a_robots_txt_file.htm&type=5). [All Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront) recipes](https://agentreadychecklist.com/platforms/salesforce-commerce-cloud/#ar-find-01) ### Adobe Commerce and Magento Open Sourcerecipe Admin: Content > Design > Configuration > Global > Edit > Search Engine Robots. Set Default Robots to INDEX, FOLLOW and keep custom instructions free of Disallow: / for AI crawlers. Save, then flush cache. On cloud, also check Fastly and WAF rules. 2.4.9 (GitHub tag published 2026-05-12; 2.4.10-alpha1 tagged 2026-10-13); release notes read 2026-11-04. Read 2026-11-04. [experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview](https://experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview). [All Adobe Commerce and Magento Open Source recipes](https://agentreadychecklist.com/platforms/adobe-commerce/#ar-find-01) ## What it does not do - robots.txt does not stop a browsing agent that runs in a user session; user-initiated fetchers may not consult robots.txt at all. - A curl with an agent user agent does not prove a real agent gets through: bot managers weigh source IP more than the UA string. - llms.txt has no say over crawling; only robots.txt and the network layer do. ## Common mistakes - A named group (User-agent: GPTBot, Disallow: /) left over from 2023 that overrides a permissive wildcard. - A WAF challenge page served with status 200 to unknown bots. - Allowlisting named crawlers and forgetting agent search APIs such as Tavily, Exa, Brave and Parallel, which the wildcard would cover. ## Related - Guide: [The one-day agent-readiness sprint](https://agentreadychecklist.com/guides/one-day-agent-readiness-sprint/) - Guide: [How to test with a plain-fetch client](https://agentreadychecklist.com/guides/test-with-a-plain-fetch-client/) - Guide: [How to read your server logs for agent user agents](https://agentreadychecklist.com/guides/read-server-logs-for-agent-user-agents/) - Guide: [How to keep a site agent ready in CI](https://agentreadychecklist.com/guides/keep-a-site-agent-ready-in-ci/) - Term: [MUST](https://agentreadychecklist.com/glossary/must/) - Term: [robots.txt](https://agentreadychecklist.com/glossary/robots-txt/) ## Sources - [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04 - [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents)read 2026-11-04 - [github.com/agentready-org/standard/blob/main/CHANGELOG.md](https://github.com/agentready-org/standard/blob/main/CHANGELOG.md)read 2026-11-04 - [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04 - [nextjs.org/docs/app/api-reference/file-conventions/metadata/robots](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)read 2026-11-04 - [docs.astro.build/en/basics/project-structure/](https://docs.astro.build/en/basics/project-structure/)read 2026-11-04 - [nuxt.com/modules/robots](https://nuxt.com/modules/robots)read 2026-11-04 - [svelte.dev/docs/kit/project-structure](https://svelte.dev/docs/kit/project-structure)read 2026-11-04 - [reactrouter.com/how-to/resource-routes](https://reactrouter.com/how-to/resource-routes)read 2026-11-04 - [developers.cloudflare.com/pages/configuration/serving-pages/](https://developers.cloudflare.com/pages/configuration/serving-pages/)read 2026-11-04 - [developer.wordpress.org/reference/functions/do\_robots/](https://developer.wordpress.org/reference/functions/do_robots/)read 2026-11-04 - [shopify.dev/docs/storefronts/themes/seo/robots-txt](https://shopify.dev/docs/storefronts/themes/seo/robots-txt)read 2026-11-04 - [help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules](https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules)read 2026-11-04 - [framer.com/help/articles/make-site-readable-by-ai-agents/](https://www.framer.com/help/articles/make-site-readable-by-ai-agents/)read 2026-11-04 - [support.wix.com/en/article/blocking-ai-crawlers-from-your-site](https://support.wix.com/en/article/blocking-ai-crawlers-from-your-site)read 2026-11-04 - [support.squarespace.com/hc/en-us/articles/38299115788813-Optimize-your-site-for-AI-powered-search-engines](https://support.squarespace.com/hc/en-us/articles/38299115788813-Optimize-your-site-for-AI-powered-search-engines)read 2026-11-04 - [knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results](https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results)read 2026-11-04 - [docusaurus.io/docs/static-assets](https://docusaurus.io/docs/static-assets)read 2026-11-04 - [mintlify.com/docs/optimize/seo](https://mintlify.com/docs/optimize/seo)read 2026-11-04 - [gitbook.com/docs/publish/seo](https://gitbook.com/docs/publish/seo)read 2026-11-04 - [docs.readme.com/main/docs/ai-discoverability](https://docs.readme.com/main/docs/ai-discoverability)read 2026-11-04 - [help.salesforce.com/s/articleView](https://help.salesforce.com/s/articleView?id=cc.b2c_generating_a_robots_txt_file.htm&type=5)read 2026-11-04 - [experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview](https://experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview)read 2026-11-04 - [ora.ai/api/checks, contract 1.38.0](https://ora.ai/api/checks?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=requirement)read 2026-11-05 - [ora.ai/llms-full.txt](https://ora.ai/llms-full.txt?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=requirement)read 2026-11-05 - [registry.npmjs.org/ax, the ax README](https://registry.npmjs.org/ax)read 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Fix it on - [Next.js (App Router)recipe](#p-nextjs-app-router) - [Astrorecipe](#p-astro) - [Nuxtrecipe](#p-nuxt) - [SvelteKitrecipe](#p-sveltekit) - [Remix and React Router (framework mode)recipe](#p-remix-react-router) - [Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront)recipe](#p-static-hosting) - [WordPress and Elementorrecipe](#p-wordpress-elementor) - [Shopify (Online Store 2.0 themes)recipe](#p-shopify) - [Webflowrecipe](#p-webflow) - [Framerrecipe](#p-framer) - [Wix (Wix Editor and Wix Studio)recipe](#p-wix) - [Squarespace (version 7.1)recipe](#p-squarespace) - [HubSpot CMS (Content Hub)recipe](#p-hubspot-cms) - [Docusaurusrecipe](#p-docusaurus) - [Mintlifyrecipe](#p-mintlify) - [GitBookrecipe](#p-gitbook) - [ReadMerecipe](#p-readme) - [Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront)recipe](#p-salesforce-commerce-cloud) - [Adobe Commerce and Magento Open Sourcerecipe](#p-adobe-commerce) Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.