--- title: "VERCEL-ROBOTS-EXCLUSIONS: Robots exclusions | Agent Ready Checklist" description: "VERCEL-ROBOTS-EXCLUSIONS (VERCEL, Find layer): Robots exclusions. What it asks, the evidence, the command that verifies it and the fix on 19 platforms." canonical: https://agentreadychecklist.com/checklist/vercel-robots-exclusions site: "Agent Ready Checklist" last_updated: 2026-11-05 section: "Checklist" --- # VERCEL-ROBOTS-EXCLUSIONS: Robots exclusions | Agent Ready Checklist > VERCEL-ROBOTS-EXCLUSIONS (VERCEL, Find layer): Robots exclusions. What it asks, the evidence, the command that verifies it and the fix on 19 platforms. Source: https://agentreadychecklist.com/checklist/vercel-robots-exclusions Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # VERCEL-ROBOTS-EXCLUSIONS: Robots exclusions VERCEL-ROBOTS-EXCLUSIONS is an item from the Vercel guide on the find layer: Use robots.txt only for deliberate exclusions: a public site does not need a robots.txt to permit crawling, so audit existing rules rather than adding agent-specific ones by default. It applies to every site. 19 of 19 platforms have a recipe below. 3 cited facts back it. ## The requirement Identifier VERCEL-ROBOTS-EXCLUSIONS Level VERCEL Layer Find Applies to every site Checked by Ora, matched on this site, Verify Kit robots-verdicts Ora check [robots-ai-policy-quality, robots-agent-user-policy](#ora) The statement Use robots.txt only for deliberate exclusions: a public site does not need a robots.txt to permit crawling, so audit existing rules rather than adding agent-specific ones by default. ## Why - The Vercel guide says public sites do not need a robots.txt file to permit crawling and that robots.txt was not evaluated in its studies.primary source, 2026-08-28: [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04 - The AgentReady spec instead recommends one wildcard rule for all agents with a Sitemap line.primary source: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 - Cloudflare scores the presence of robots.txt and of explicit AI bot rules, the opposite default.primary source: [isitagentready.com/.well-known/agent-skills/ai-rules/SKILL.md](https://isitagentready.com/.well-known/agent-skills/ai-rules/SKILL.md), read 2026-11-04 ## How to verify ### [robots.txt verdict per agent token](https://agentreadychecklist.com/kit/robots-verdicts/) ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=robots-verdicts ``` Parses /robots.txt per RFC 9309 group matching and prints allow or disallow for / for each of 17 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Applebot, Applebot-Extended, Bingbot, Amazonbot, Meta-ExternalAgent, DuckAssistBot, MistralAI-User), plus any Content-Signal and Sitemap lines. FAIL when a search or user-initiated token is disallowed at /. **It cannot prove:** User-initiated fetchers may ignore robots.txt; a disallow for a training crawler can be a deliberate, valid choice; the parser checks / only unless given a path. ### The one-line version curl -s /robots.txt and read every Disallow; each must have a reason you can state. ## Verify with Ora 2 checks in Ora's catalogue (contract 1.38.0, read 2026-11-05) read this row. Each entry says what the check looks at and why this page lists it. - `robots-ai-policy-quality` robots.txt AI crawler policy Reads whether robots.txt lets the AI search crawlers OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages. Blocking training crawlers such as GPTBot or ClaudeBot never lowers it. Discovery, 2 points, required; catalogue applicability domain-only. Listed because: same name, `robots.txt`. - `robots-agent-user-policy` robots.txt agent-user policy Reads the permission robots.txt declares for the homepage to ChatGPT-User, Claude-User and Perplexity-User. A policy that cannot be verified reads as not applicable. Access, 2 points, required; catalogue applicability domain-only. Listed because: same name, `robots.txt`. ### Run just these checks ```bash curl -s -X POST https://ora.ai/api/scan/checks -H 'content-type: application/json' -d '{"url":"https://example.com","checkIds":["robots-ai-policy-quality","robots-agent-user-policy"]}' ``` This calls the endpoint Ora documents for re-verifying a fix. It sends no cache, returns a result for each id and no total score, and spends one scan from the daily budget. Replace the URL with yours. [How to re-verify one fix](https://agentreadychecklist.com/guides/re-verify-one-fix-with-ora/). ### Or audit the whole site ```bash npx ax@0.10 audit https://example.com --min-score 70 ``` Exit code 1 means the score is below the number you passed, so the same line gates a deploy: [the CI recipe](https://agentreadychecklist.com/guides/gate-deploys-on-agent-readiness-in-ci/). [Why this site recommends Ora, and where it falls short](https://agentreadychecklist.com/verify-with-ora/). ## In the crawl This row has one crawl rate. Each panel splits the 2026-11-04 crawl into ten bands of 1,000 ranks, on a 0 to 100 percent scale. robots.txt blocks no search or user-fetch token 73.2% overall Reachable sites in the Tranco top 10,000 on which the check ran, 2026-11-04. Left is ranks 1 to 1,000. Numbers behind the chart | Rate | 1 to 1,000 | 1,001 to 2,000 | 2,001 to 3,000 | 3,001 to 4,000 | 4,001 to 5,000 | 5,001 to 6,000 | 6,001 to 7,000 | 7,001 to 8,000 | 8,001 to 9,000 | 9,001 to 10,000 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | robots.txt blocks no search or user-fetch token | 70% | 69% | 76% | 76% | 74% | 74% | 71% | 77% | 73% | 71% | ## Fix it on your platform ### Next.js (App Router)recipe Open the robots.txt that Next.js serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: app/robots.ts returning MetadataRoute.Robots generates /robots.txt. Next.js 16.x (npm latest 16.4.5, read 2026-11-04); docs at nextjs.org/docs/app. Derived from Next.js's documentation read on 2026-11-04. [nextjs.org/docs/app/api-reference/file-conventions/metadata/robots](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Next.js (App Router) recipes](https://agentreadychecklist.com/platforms/nextjs-app-router/#vercel-robots-exclusions) ### Astrorecipe Open the robots.txt that Astro serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: No robots helper; write public/robots.txt or a src/pages/robots.txt.ts static endpoint. Astro 7.x (npm latest astro 7.4.5, @astrojs/sitemap 3.7.4, read 2026-11-04). Derived from Astro's documentation read on 2026-11-04. [docs.astro.build/en/guides/endpoints/](https://docs.astro.build/en/guides/endpoints/) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Astro recipes](https://agentreadychecklist.com/platforms/astro/#vercel-robots-exclusions) ### Nuxtrecipe Open the robots.txt that Nuxt serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Nuxt Robots module (listed in the official module directory) generates robots.txt and X-Robots-Tag; or write public/robots.txt. Nuxt 4.x (npm latest nuxt 4.6.4, @nuxtjs/robots 7.0.2, @nuxtjs/sitemap 9.1.0, read 2026-11-04). Derived from Nuxt's documentation read on 2026-11-04. [nuxt.com/modules/robots](https://nuxt.com/modules/robots) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Nuxt recipes](https://agentreadychecklist.com/platforms/nuxt/#vercel-robots-exclusions) ### SvelteKitrecipe Open the robots.txt that SvelteKit serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: No first-party doc: SvelteKit has no robots helper. Put robots.txt in static/ or write src/routes/robots.txt/+server.ts. SvelteKit (npm latest @sveltejs/kit 3.1.1, read 2026-11-04); docs at svelte.dev/docs/kit. Derived from SvelteKit's documentation read on 2026-11-04. [svelte.dev/docs/kit/project-structure](https://svelte.dev/docs/kit/project-structure) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All SvelteKit recipes](https://agentreadychecklist.com/platforms/sveltekit/#vercel-robots-exclusions) ### Remix and React Router (framework mode)recipe Open the robots.txt that Remix and React Router serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: No robots helper; a resource route (loader without default export) at app/routes/robots\[.\]txt.ts returns text. React Router framework mode (npm latest react-router 8.5.1; Remix v2 @remix-run/node 2.17.5, read 2026-11-04); docs at reactrouter.com. Derived from Remix and React Router's documentation read on 2026-11-04. [reactrouter.com/how-to/resource-routes](https://reactrouter.com/how-to/resource-routes) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Remix and React Router (framework mode) recipes](https://agentreadychecklist.com/platforms/remix-react-router/#vercel-robots-exclusions) ### Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront)recipe Open the robots.txt that Plain static hosting serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: No first-party doc: Hosts serve robots.txt as a plain file from the publish root; no host-specific doc needed. Host documentation as published on 2026-11-04 (no version numbers; config files \_headers, \_redirects, vercel.json, 404.html). Derived from Plain static hosting's documentation read on 2026-11-04. [developers.cloudflare.com/pages/configuration/serving-pages/](https://developers.cloudflare.com/pages/configuration/serving-pages/) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront) recipes](https://agentreadychecklist.com/platforms/static-hosting/#vercel-robots-exclusions) ### WordPress and Elementorrecipe Open the robots.txt that WordPress and Elementor serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Core serves a virtual robots.txt through do\_robots(); the robots\_txt filter edits its output. A physical robots.txt in the web root overrides it. WordPress 7.1.4 (core release from the version-check API), Elementor 4.3.6 (plugin directory, updated 2026-11-02), Yoast SEO 28.8 noted for llms.txt; docs read 2026-11-04. Derived from WordPress and Elementor's documentation read on 2026-11-04. [developer.wordpress.org/reference/hooks/robots\_txt/](https://developer.wordpress.org/reference/hooks/robots_txt/) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All WordPress and Elementor recipes](https://agentreadychecklist.com/platforms/wordpress-elementor/#vercel-robots-exclusions) ### Shopify (Online Store 2.0 themes)recipe Open the robots.txt that Shopify serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Shopify generates robots.txt; a templates/robots.txt.liquid file replaces the output and supports the robots, group, rule, user\_agent, sitemap and request objects. Hosted SaaS, Online Store 2.0 Liquid themes, docs read 2026-11-04; storefront behaviour probed live on one Shopify store the same day. Derived from Shopify's documentation read on 2026-11-04. [shopify.dev/docs/storefronts/themes/seo/robots-txt](https://shopify.dev/docs/storefronts/themes/seo/robots-txt) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Shopify (Online Store 2.0 themes) recipes](https://agentreadychecklist.com/platforms/shopify/#vercel-robots-exclusions) ### Webflowrecipe Open the robots.txt that Webflow serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Site settings > SEO > Indexing holds the robots.txt rules; paid Site plan or paid Workspace needed. Hosted SaaS, help center articles read 2026-11-04 (updated dates 2024-10-14 to 2026-10-29). Derived from Webflow's documentation read on 2026-11-04. [help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules](https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Webflow recipes](https://agentreadychecklist.com/platforms/webflow/#vercel-robots-exclusions) ### Framerrecipe Open the robots.txt that Framer serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Generated for every published site; a robots.txt uploaded as a Static File (Pro and Enterprise) replaces it. Hosted SaaS, help articles read 2026-11-04 (marked updated 2026-09-15). Derived from Framer's documentation read on 2026-11-04. [framer.com/help/articles/how-can-i-access-the-robots-txt-file/](https://www.framer.com/help/articles/how-can-i-access-the-robots-txt-file/) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Framer recipes](https://agentreadychecklist.com/platforms/framer/#vercel-robots-exclusions) ### Wix (Wix Editor and Wix Studio)recipe Open the robots.txt that Wix serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: SEO & GEO > Tools and settings > Robots.txt Editor; Wix includes the sitemap URL in the file. Hosted SaaS, help center read 2026-11-04. Derived from Wix's documentation read on 2026-11-04. [support.wix.com/en/article/editing-your-sites-robotstxt-file](https://support.wix.com/en/article/editing-your-sites-robotstxt-file) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Wix (Wix Editor and Wix Studio) recipes](https://agentreadychecklist.com/platforms/wix/#vercel-robots-exclusions) ### Squarespace (version 7.1)recipe Open the robots.txt that Squarespace serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: robots.txt is generated. The cited article says Squarespace provides two options related to crawlers, both written into robots.txt: the 'Block known artificial intelligence crawlers' box under Settings > Crawlers, and hiding the site from search results, which the article links to as a separate guide. Hosted SaaS, version 7.1 sites, help center read 2026-11-04 (articles updated 2026-07-17 to 2026-11-03). Derived from Squarespace's documentation read on 2026-11-04. [support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site](https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Squarespace (version 7.1) recipes](https://agentreadychecklist.com/platforms/squarespace/#vercel-robots-exclusions) ### HubSpot CMS (Content Hub)recipe Open the robots.txt that HubSpot CMS serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Settings > Content > Pages > choose domain > SEO & Crawlers > Robots.txt; default or per-domain. Hosted SaaS, Content Hub, knowledge base and developer docs read 2026-11-04. Derived from HubSpot CMS's documentation read on 2026-11-04. [knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results](https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All HubSpot CMS (Content Hub) recipes](https://agentreadychecklist.com/platforms/hubspot-cms/#vercel-robots-exclusions) ### Docusaurusrecipe Open the robots.txt that Docusaurus serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: No robots.txt generator. Files in static/ are copied to the build root, so static/robots.txt becomes /robots.txt. 3.10.2 (npm @docusaurus/core latest, read 2026-11-04). Derived from Docusaurus's documentation read on 2026-11-04. [docusaurus.io/docs/static-assets](https://docusaurus.io/docs/static-assets) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Docusaurus recipes](https://agentreadychecklist.com/platforms/docusaurus/#vercel-robots-exclusions) ### Mintlifyrecipe Open the robots.txt that Mintlify serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: robots.txt is generated and includes Content-Signal directives; a robots.txt at the project root overrides it. Hosted SaaS, docs read 2026-11-04. Derived from Mintlify's documentation read on 2026-11-04. [mintlify.com/docs/optimize/seo](https://mintlify.com/docs/optimize/seo) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Mintlify recipes](https://agentreadychecklist.com/platforms/mintlify/#vercel-robots-exclusions) ### GitBookrecipe Open the robots.txt that GitBook serves for your site and read every Disallow; each one must have a reason you can state. The live gitbook.com/docs/robots.txt read on 2026-11-04 holds one wildcard group that allows everything, a Content-Signal line and a Sitemap line. GitBook documents no robots.txt editor; to own the file, serve the docs under a subdirectory of your own domain and write the root robots.txt there. Hosted SaaS, docs read 2026-11-04. Derived from GitBook's documentation read on 2026-11-04. [gitbook.com/docs/robots.txt](https://gitbook.com/docs/robots.txt) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All GitBook recipes](https://agentreadychecklist.com/platforms/gitbook/#vercel-robots-exclusions) ### ReadMerecipe Open the robots.txt that ReadMe serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Content Signals toggles (training, search indexing, prompt context) in AI > AI Discoverability. Live docs.readme.com/robots.txt (2026-11-04) has User-agent: \*, Allow: /, Sitemap and Content-Signal lines. Hosted SaaS, docs read 2026-11-04. Derived from ReadMe's documentation read on 2026-11-04. [docs.readme.com/main/docs/ai-discoverability](https://docs.readme.com/main/docs/ai-discoverability) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All ReadMe recipes](https://agentreadychecklist.com/platforms/readme/#vercel-robots-exclusions) ### Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront)recipe Open the robots.txt that your PWA Kit storefront serves and read every Disallow; each one must have a reason you can state. The PWA Kit sitemap guide says to keep the sitemap line and the crawl directives in the robots.txt file of the PWA Kit or headless site. Hosted SaaS (B2C Commerce help release 264.0.0); PWA Kit retail-react-app 10.3.0 and pwa-kit-runtime 3.21.0 on npm, read 2026-11-04. Derived from Salesforce B2C Commerce's documentation read on 2026-11-04. [developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html](https://developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront) recipes](https://agentreadychecklist.com/platforms/salesforce-commerce-cloud/#vercel-robots-exclusions) ### Adobe Commerce and Magento Open Sourcerecipe Open the robots.txt that Adobe Commerce and Magento Open Source serves for your site and read every Disallow; each one must have a reason you can state. How the file is produced here: Content > Design > Configuration > Global > Search Engine Robots: Default Robots and Edit custom instruction of robots.txt File. robots.txt is generated by a controller when no physical file exists. 2.4.9 (GitHub tag published 2026-05-12; 2.4.10-alpha1 tagged 2026-10-13); release notes read 2026-11-04. Derived from Adobe Commerce and Magento Open Source's documentation read on 2026-11-04. [experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview](https://experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview) and [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents). [All Adobe Commerce and Magento Open Source recipes](https://agentreadychecklist.com/platforms/adobe-commerce/#vercel-robots-exclusions) ## What it does not do - Does not say delete robots.txt; it says do not add rules you have not decided on. ## Common mistakes - Copying a long AI-bot blocklist from a template without a decision behind each line. ## Related - Term: [MUST](https://agentreadychecklist.com/glossary/must/) - Term: [robots.txt](https://agentreadychecklist.com/glossary/robots-txt/) ## Sources - [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents)read 2026-11-04 - [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04 - [isitagentready.com/.well-known/agent-skills/ai-rules/SKILL.md](https://isitagentready.com/.well-known/agent-skills/ai-rules/SKILL.md)read 2026-11-04 - [nextjs.org/docs/app/api-reference/file-conventions/metadata/robots](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)read 2026-11-04 - [docs.astro.build/en/guides/endpoints/](https://docs.astro.build/en/guides/endpoints/)read 2026-11-04 - [nuxt.com/modules/robots](https://nuxt.com/modules/robots)read 2026-11-04 - [svelte.dev/docs/kit/project-structure](https://svelte.dev/docs/kit/project-structure)read 2026-11-04 - [reactrouter.com/how-to/resource-routes](https://reactrouter.com/how-to/resource-routes)read 2026-11-04 - [developers.cloudflare.com/pages/configuration/serving-pages/](https://developers.cloudflare.com/pages/configuration/serving-pages/)read 2026-11-04 - [developer.wordpress.org/reference/hooks/robots\_txt/](https://developer.wordpress.org/reference/hooks/robots_txt/)read 2026-11-04 - [shopify.dev/docs/storefronts/themes/seo/robots-txt](https://shopify.dev/docs/storefronts/themes/seo/robots-txt)read 2026-11-04 - [help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules](https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules)read 2026-11-04 - [framer.com/help/articles/how-can-i-access-the-robots-txt-file/](https://www.framer.com/help/articles/how-can-i-access-the-robots-txt-file/)read 2026-11-04 - [support.wix.com/en/article/editing-your-sites-robotstxt-file](https://support.wix.com/en/article/editing-your-sites-robotstxt-file)read 2026-11-04 - [support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site](https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site)read 2026-11-04 - [knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results](https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results)read 2026-11-04 - [docusaurus.io/docs/static-assets](https://docusaurus.io/docs/static-assets)read 2026-11-04 - [mintlify.com/docs/optimize/seo](https://mintlify.com/docs/optimize/seo)read 2026-11-04 - [gitbook.com/docs/robots.txt](https://gitbook.com/docs/robots.txt)read 2026-11-04 - [docs.readme.com/main/docs/ai-discoverability](https://docs.readme.com/main/docs/ai-discoverability)read 2026-11-04 - [developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html](https://developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html)read 2026-11-04 - [experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview](https://experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/seo-overview)read 2026-11-04 - [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04 - [ora.ai/api/checks, contract 1.38.0](https://ora.ai/api/checks?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=requirement)read 2026-11-05 - [ora.ai/llms-full.txt](https://ora.ai/llms-full.txt?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=requirement)read 2026-11-05 - [registry.npmjs.org/ax, the ax README](https://registry.npmjs.org/ax)read 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Fix it on - [Next.js (App Router)recipe](#p-nextjs-app-router) - [Astrorecipe](#p-astro) - [Nuxtrecipe](#p-nuxt) - [SvelteKitrecipe](#p-sveltekit) - [Remix and React Router (framework mode)recipe](#p-remix-react-router) - [Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront)recipe](#p-static-hosting) - [WordPress and Elementorrecipe](#p-wordpress-elementor) - [Shopify (Online Store 2.0 themes)recipe](#p-shopify) - [Webflowrecipe](#p-webflow) - [Framerrecipe](#p-framer) - [Wix (Wix Editor and Wix Studio)recipe](#p-wix) - [Squarespace (version 7.1)recipe](#p-squarespace) - [HubSpot CMS (Content Hub)recipe](#p-hubspot-cms) - [Docusaurusrecipe](#p-docusaurus) - [Mintlifyrecipe](#p-mintlify) - [GitBookrecipe](#p-gitbook) - [ReadMerecipe](#p-readme) - [Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront)recipe](#p-salesforce-commerce-cloud) - [Adobe Commerce and Magento Open Sourcerecipe](#p-adobe-commerce) Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.