--- title: "AR-FIND-03: Sitemap listed in robots.txt | Agent Ready Checklist" description: "AR-FIND-03 (SHOULD, Find layer): Sitemap listed in robots.txt. What it asks, the evidence, the command that verifies it and the fix on 19 platforms." canonical: https://agentreadychecklist.com/checklist/ar-find-03 site: "Agent Ready Checklist" last_updated: 2026-11-05 section: "Checklist" --- # AR-FIND-03: Sitemap listed in robots.txt | Agent Ready Checklist > AR-FIND-03 (SHOULD, Find layer): Sitemap listed in robots.txt. What it asks, the evidence, the command that verifies it and the fix on 19 platforms. Source: https://agentreadychecklist.com/checklist/ar-find-03 Site: Agent Ready Checklist Last reviewed: 2026-11-05 Agent endpoints: https://agentreadychecklist.com/llms.txt (index), https://agentreadychecklist.com/openapi.json (API), https://agentreadychecklist.com/mcp (MCP server), https://agentreadychecklist.com/ask (search) Access: https://agentreadychecklist.com/auth.md (no key). Pricing: https://agentreadychecklist.com/pricing.md (free). # AR-FIND-03: Sitemap listed in robots.txt AR-FIND-03 is a SHOULD on the find layer: List the site pages in a sitemap.xml and reference it from robots.txt with a Sitemap: line. It applies to every site. 19 of 19 platforms have a recipe below. The kit check sitemap-listed tests it with one command. 4 cited facts back it, 1 labelled as a vendor claim. ## The requirement Identifier AR-FIND-03 Level SHOULD Layer Find Applies to every site Checked by Ora, matched on this site, Cloudflare sitemap, Verify Kit sitemap-listed Ora check [sitemap](#ora) The statement List the site pages in a sitemap.xml and reference it from robots.txt with a Sitemap: line. ## Why - Ora's lab found (published 2026-10-21): agents fetched sitemap.xml directly in 4% of runs, one of the files they touch least; its value is for crawlers and search indexes.vendor claim, 2026-10-21: [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md), read 2026-11-04 - 43 of 1,033 traced runs fetched sitemap.xml.primary source, 2026-08-28: [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 2026-11-04 - The sitemaps protocol caps one file at 50,000 URLs and 50MB uncompressed; larger sites use a sitemap index.primary source: [sitemaps.org/protocol.html](https://www.sitemaps.org/protocol.html), read 2026-11-04 - RFC 9309 lets robots.txt carry Sitemap lines alongside the groups.primary source: [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309), read 2026-11-04 ## How to verify ### [Sitemap referenced and lists the page](https://agentreadychecklist.com/kit/sitemap-listed/) ```bash curl -fsSL https://agentreadychecklist.com/kit/verify.sh | bash -s -- https://example.com --only=sitemap-listed ``` Reads Sitemap: lines from robots.txt (falls back to /sitemap.xml), follows one level of sitemap index, and checks the URL appears in a . **It cannot prove:** Does not walk gzip sitemaps beyond one level or prove search engines have read it. ### The one-line version curl -s /robots.txt | grep -i "^sitemap:" then curl the sitemap and grep for the page URL. ## Verify with Ora One check in Ora's catalogue (contract 1.38.0, read 2026-11-05) read this row. Each entry says what the check looks at and why this page lists it. - `sitemap` Sitemap exists Looks for an XML sitemap. **Partial.** Looks for an XML sitemap. It does not read the Sitemap line in robots.txt that the row also asks for. Access, 2 points, required; catalogue applicability domain-only. Listed because: same name, `sitemap`. ### Run just these checks ```bash curl -s -X POST https://ora.ai/api/scan/checks -H 'content-type: application/json' -d '{"url":"https://example.com","checkIds":["sitemap"]}' ``` This calls the endpoint Ora documents for re-verifying a fix. It sends no cache, returns a result for each id and no total score, and spends one scan from the daily budget. Replace the URL with yours. [How to re-verify one fix](https://agentreadychecklist.com/guides/re-verify-one-fix-with-ora/). ### Or audit the whole site ```bash npx ax@0.10 audit https://example.com --min-score 70 ``` ### Related checks This check reads something near the row but asks for more than the row does. A site that meets the row can score nothing or lose points on it, so it is not counted as a test of the row and is left out of the command above. - `sitemap-lastmod` Sitemap freshness (lastmod) Reads whether the lastmod dates in the sitemap parse and whether the newest is recent. The row asks for a sitemap and a Sitemap line in robots.txt, not for fresh dates, so a site that meets the row scores nothing here. Access, 1 point, bonus, recommended; catalogue applicability domain-only. Exit code 1 means the score is below the number you passed, so the same line gates a deploy: [the CI recipe](https://agentreadychecklist.com/guides/gate-deploys-on-agent-readiness-in-ci/). [Why this site recommends Ora, and where it falls short](https://agentreadychecklist.com/verify-with-ora/). ## Fix it on your platform ### Next.js (App Router)recipe Create app/sitemap.ts exporting a default function that returns MetadataRoute.Sitemap entries ({ url, lastModified }). Reference it from app/robots.ts with sitemap: "https://example.com/sitemap.xml". Next.js 16.x (npm latest 16.4.5, read 2026-11-04); docs at nextjs.org/docs/app. Read 2026-11-04. [nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap). [All Next.js (App Router) recipes](https://agentreadychecklist.com/platforms/nextjs-app-router/#ar-find-03) ### Astrorecipe npx astro add sitemap, set site: "https://example.com" in astro.config.mjs, then add "Sitemap: https://example.com/sitemap-index.xml" to robots.txt. Astro 7.x (npm latest astro 7.4.5, @astrojs/sitemap 3.7.4, read 2026-11-04). Read 2026-11-04. [docs.astro.build/en/guides/integrations-guide/sitemap/](https://docs.astro.build/en/guides/integrations-guide/sitemap/). [All Astro recipes](https://agentreadychecklist.com/platforms/astro/#ar-find-03) ### Nuxtrecipe npx nuxi module add sitemap, set site.url in nuxt.config.ts; the module serves /sitemap.xml. Add the Sitemap line to robots.txt (the Robots module adds it when both are installed). Nuxt 4.x (npm latest nuxt 4.6.4, @nuxtjs/robots 7.0.2, @nuxtjs/sitemap 9.1.0, read 2026-11-04). Read 2026-11-04. [nuxt.com/modules/sitemap](https://nuxt.com/modules/sitemap). [All Nuxt recipes](https://agentreadychecklist.com/platforms/nuxt/#ar-find-03) ### SvelteKitrecipe Create src/routes/sitemap.xml/+server.ts: export async function GET() { return new Response(xml, { headers: { "Content-Type": "application/xml" } }); }, add export const prerender = true, and add a Sitemap line to robots.txt. SvelteKit (npm latest @sveltejs/kit 3.1.1, read 2026-11-04); docs at svelte.dev/docs/kit. Read 2026-11-04. [svelte.dev/docs/kit/seo](https://svelte.dev/docs/kit/seo). [All SvelteKit recipes](https://agentreadychecklist.com/platforms/sveltekit/#ar-find-03) ### Remix and React Router (framework mode)recipe Add app/routes/sitemap\[.\]xml.ts with export async function loader() { return new Response(xml, { headers: { "Content-Type": "application/xml" } }); } and a Sitemap line in robots.txt. React Router framework mode (npm latest react-router 8.5.1; Remix v2 @remix-run/node 2.17.5, read 2026-11-04); docs at reactrouter.com. Read 2026-11-04. [reactrouter.com/how-to/resource-routes](https://reactrouter.com/how-to/resource-routes). [All Remix and React Router (framework mode) recipes](https://agentreadychecklist.com/platforms/remix-react-router/#ar-find-03) ### Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront)recipe Upload sitemap.xml from your generator and add "Sitemap: https://example.com/sitemap.xml" to robots.txt. Host documentation as published on 2026-11-04 (no version numbers; config files \_headers, \_redirects, vercel.json, 404.html). Read 2026-11-04. [sitemaps.org/protocol.html](https://www.sitemaps.org/protocol.html). [All Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront) recipes](https://agentreadychecklist.com/platforms/static-hosting/#ar-find-03) ### WordPress and Elementorrecipe Core publishes /wp-sitemap.xml and adds `Sitemap: https://example.com/wp-sitemap.xml` to the virtual robots.txt. If an SEO plugin replaces it with /sitemap\_index.xml, confirm the robots.txt line points at the live index. WordPress 7.1.4 (core release from the version-check API), Elementor 4.3.6 (plugin directory, updated 2026-11-02), Yoast SEO 28.8 noted for llms.txt; docs read 2026-11-04. Read 2026-11-04. [make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/](https://make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/). [All WordPress and Elementor recipes](https://agentreadychecklist.com/platforms/wordpress-elementor/#ar-find-03) ### Shopify (Online Store 2.0 themes)recipe Nothing to do: /sitemap.xml is generated and the default groups print it. In a custom robots.txt.liquid keep the `{%- if group.sitemap != blank -%}{{ group.sitemap }}{%- endif -%}` branch. Hosted SaaS, Online Store 2.0 Liquid themes, docs read 2026-11-04; storefront behaviour probed live on one Shopify store the same day. Read 2026-11-04. [shopify.dev/docs/storefronts/themes/seo/robots-txt](https://shopify.dev/docs/storefronts/themes/seo/robots-txt). [All Shopify (Online Store 2.0 themes) recipes](https://agentreadychecklist.com/platforms/shopify/#ar-find-03) ### Webflowrecipe Turn on Auto-generate sitemap in Site settings > SEO > Sitemap, then add `Sitemap: https://example.com/sitemap.xml` to the robots.txt rules and publish. Hosted SaaS, help center articles read 2026-11-04 (updated dates 2024-10-14 to 2026-10-29). Read 2026-11-04. [help.webflow.com/hc/en-us/articles/33961355371667-Create-a-sitemap-in-Webflow](https://help.webflow.com/hc/en-us/articles/33961355371667-Create-a-sitemap-in-Webflow). [All Webflow recipes](https://agentreadychecklist.com/platforms/webflow/#ar-find-03) ### Framerrecipe /sitemap.xml is automatic. Open /robots.txt; if it lacks a Sitemap line, upload a custom robots.txt via Static Files that includes `Sitemap: https://example.com/sitemap.xml`. Hosted SaaS, help articles read 2026-11-04 (marked updated 2026-09-15). Read 2026-11-04. [framer.com/help/articles/how-can-i-access-the-sitemap-xml-file/](https://www.framer.com/help/articles/how-can-i-access-the-sitemap-xml-file/). [All Framer recipes](https://agentreadychecklist.com/platforms/framer/#ar-find-03) ### Wix (Wix Editor and Wix Studio)recipe Nothing to build: /sitemap.xml is automatic and Wix writes its URL into robots.txt. Confirm the line is still there after any robots.txt edit. Hosted SaaS, help center read 2026-11-04. Read 2026-11-04. [support.wix.com/en/article/editing-your-sites-robotstxt-file](https://support.wix.com/en/article/editing-your-sites-robotstxt-file). [All Wix (Wix Editor and Wix Studio) recipes](https://agentreadychecklist.com/platforms/wix/#ar-find-03) ### Squarespace (version 7.1)recipe Nothing to build: /sitemap.xml is automatic and squarespace.com's own robots.txt lists its sitemap (read 2026-11-04). Make the site Public so the sitemap loads. Hosted SaaS, version 7.1 sites, help center read 2026-11-04 (articles updated 2026-07-17 to 2026-11-03). Read 2026-11-04. [support.squarespace.com/hc/en-us/articles/206543547-View-your-site-map](https://support.squarespace.com/hc/en-us/articles/206543547-View-your-site-map). [All Squarespace (version 7.1) recipes](https://agentreadychecklist.com/platforms/squarespace/#ar-find-03) ### HubSpot CMS (Content Hub)recipe The sitemap is automatic; add landing pages to it by hand and add `Sitemap: https://www.example.com/sitemap.xml` to the robots.txt field. Hosted SaaS, Content Hub, knowledge base and developer docs read 2026-11-04. Read 2026-11-04. [knowledge.hubspot.com/domains-and-urls/view-and-edit-a-hubspot-hosted-domain-sitemap](https://knowledge.hubspot.com/domains-and-urls/view-and-edit-a-hubspot-hosted-domain-sitemap). [All HubSpot CMS (Content Hub) recipes](https://agentreadychecklist.com/platforms/hubspot-cms/#ar-find-03) ### Docusaurusrecipe Keep preset-classic sitemap on (default). Configure in docusaurus.config.js presets: \['classic', { sitemap: { lastmod: 'date', changefreq: 'weekly', ignorePatterns: \['/tags/\*\*'\], filename: 'sitemap.xml' } }\]. Add the Sitemap: line to static/robots.txt. 3.10.2 (npm @docusaurus/core latest, read 2026-11-04). Read 2026-11-04. [docusaurus.io/docs/api/plugins/@docusaurus/plugin-sitemap](https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-sitemap). [All Docusaurus recipes](https://agentreadychecklist.com/platforms/docusaurus/#ar-find-03) ### Mintlifyrecipe Nothing to do: sitemap.xml is generated and referenced. To override, commit sitemap.xml at the project root. Hosted SaaS, docs read 2026-11-04. Read 2026-11-04. [mintlify.com/docs/optimize/seo](https://mintlify.com/docs/optimize/seo). [All Mintlify recipes](https://agentreadychecklist.com/platforms/mintlify/#ar-find-03) ### GitBookrecipe Nothing to do for public sites: sitemap generated. Do not set the site to unlisted, which disables the sitemap. Hosted SaaS, docs read 2026-11-04. Read 2026-11-04. [gitbook.com/docs/publish/seo](https://gitbook.com/docs/publish/seo). [All GitBook recipes](https://agentreadychecklist.com/platforms/gitbook/#ar-find-03) ### ReadMerecipe Nothing to do: the generated robots.txt references /sitemap.xml (seen live on docs.readme.com, 2026-11-04). Hosted SaaS, docs read 2026-11-04. Read 2026-11-04. [docs.readme.com/main/docs/ai-discoverability](https://docs.readme.com/main/docs/ai-discoverability). [All ReadMe recipes](https://agentreadychecklist.com/platforms/readme/#ar-find-03) ### Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront)recipe Generate the sitemap in Business Manager (sitemap\_index.xml), add Sitemap: https://example.com/sitemap\_index.xml to robots.txt. PWA Kit: app.get('/sitemap\_index.xml', runtime.serveStaticFile('static/sitemap\_index.xml')) in ssr.js and add 'static/sitemap\_index.xml' to ssrShared in config/default.js. Hosted SaaS (B2C Commerce help release 264.0.0); PWA Kit retail-react-app 10.3.0 and pwa-kit-runtime 3.21.0 on npm, read 2026-11-04. Read 2026-11-04. [developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html](https://developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html). [All Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront) recipes](https://agentreadychecklist.com/platforms/salesforce-commerce-cloud/#ar-find-03) ### Adobe Commerce and Magento Open Sourcerecipe Marketing > SEO & Search > Site Map > Add Sitemap, Save & Generate. Stores > Settings > Configuration > Catalog > XML Sitemap > Search Engine Submission Settings > Enable Submission to Robots.txt = Yes. On cloud, write the file to the /media/ path. 2.4.9 (GitHub tag published 2026-05-12; 2.4.10-alpha1 tagged 2026-10-13); release notes read 2026-11-04. Read 2026-11-04. [experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/sitemap-xml](https://experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/sitemap-xml). [All Adobe Commerce and Magento Open Source recipes](https://agentreadychecklist.com/platforms/adobe-commerce/#ar-find-03) ## What it does not do - A sitemap does not make an agent read a page; agents arrive through links and search. - Listing a URL does not override a robots.txt Disallow for it. ## Common mistakes - Sitemap lists URLs that redirect or 404. - robots.txt has no Sitemap line, so crawlers must guess the path. ## Related - Guide: [The one-day agent-readiness sprint](https://agentreadychecklist.com/guides/one-day-agent-readiness-sprint/) - Term: [robots.txt](https://agentreadychecklist.com/glossary/robots-txt/) - Term: [Sitemap](https://agentreadychecklist.com/glossary/sitemap/) ## Sources - [github.com/agentready-org/standard/blob/main/spec/spec.md](https://github.com/agentready-org/standard/blob/main/spec/spec.md)read 2026-11-04 - [vercel.com/kb/guide/make-your-site-readable-by-ai-agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents)read 2026-11-04 - [sitemaps.org/protocol.html](https://www.sitemaps.org/protocol.html)read 2026-11-04 - [rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)read 2026-11-04 - [nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap)read 2026-11-04 - [docs.astro.build/en/guides/integrations-guide/sitemap/](https://docs.astro.build/en/guides/integrations-guide/sitemap/)read 2026-11-04 - [nuxt.com/modules/sitemap](https://nuxt.com/modules/sitemap)read 2026-11-04 - [svelte.dev/docs/kit/seo](https://svelte.dev/docs/kit/seo)read 2026-11-04 - [reactrouter.com/how-to/resource-routes](https://reactrouter.com/how-to/resource-routes)read 2026-11-04 - [make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/](https://make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/)read 2026-11-04 - [shopify.dev/docs/storefronts/themes/seo/robots-txt](https://shopify.dev/docs/storefronts/themes/seo/robots-txt)read 2026-11-04 - [help.webflow.com/hc/en-us/articles/33961355371667-Create-a-sitemap-in-Webflow](https://help.webflow.com/hc/en-us/articles/33961355371667-Create-a-sitemap-in-Webflow)read 2026-11-04 - [framer.com/help/articles/how-can-i-access-the-sitemap-xml-file/](https://www.framer.com/help/articles/how-can-i-access-the-sitemap-xml-file/)read 2026-11-04 - [support.wix.com/en/article/editing-your-sites-robotstxt-file](https://support.wix.com/en/article/editing-your-sites-robotstxt-file)read 2026-11-04 - [support.squarespace.com/hc/en-us/articles/206543547-View-your-site-map](https://support.squarespace.com/hc/en-us/articles/206543547-View-your-site-map)read 2026-11-04 - [knowledge.hubspot.com/domains-and-urls/view-and-edit-a-hubspot-hosted-domain-sitemap](https://knowledge.hubspot.com/domains-and-urls/view-and-edit-a-hubspot-hosted-domain-sitemap)read 2026-11-04 - [docusaurus.io/docs/api/plugins/@docusaurus/plugin-sitemap](https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-sitemap)read 2026-11-04 - [mintlify.com/docs/optimize/seo](https://mintlify.com/docs/optimize/seo)read 2026-11-04 - [gitbook.com/docs/publish/seo](https://gitbook.com/docs/publish/seo)read 2026-11-04 - [docs.readme.com/main/docs/ai-discoverability](https://docs.readme.com/main/docs/ai-discoverability)read 2026-11-04 - [developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html](https://developer.salesforce.com/docs/commerce/pwa-kit-managed-runtime/guide/create-a-sitemap.html)read 2026-11-04 - [experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/sitemap-xml](https://experienceleague.adobe.com/en/docs/commerce-admin/marketing/seo/sitemap-xml)read 2026-11-04 - [ora.ai/api/checks, contract 1.38.0](https://ora.ai/api/checks?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=requirement)read 2026-11-05 - [ora.ai/llms-full.txt](https://ora.ai/llms-full.txt?utm_source=agentreadychecklist.com&utm_medium=referral&utm_campaign=requirement)read 2026-11-05 - [registry.npmjs.org/ax, the ax README](https://registry.npmjs.org/ax)read 2026-11-05 Verify with Ora The scanner this site recommends for the whole checklist. 132 checks, a score out of 100, no account. `npx ax@0.10 audit --min-score ` [Why Ora, and its limits](https://agentreadychecklist.com/verify-with-ora/) Fix it on - [Next.js (App Router)recipe](#p-nextjs-app-router) - [Astrorecipe](#p-astro) - [Nuxtrecipe](#p-nuxt) - [SvelteKitrecipe](#p-sveltekit) - [Remix and React Router (framework mode)recipe](#p-remix-react-router) - [Plain static hosting (Netlify, Vercel static, Cloudflare Pages, GitHub Pages, S3 and CloudFront)recipe](#p-static-hosting) - [WordPress and Elementorrecipe](#p-wordpress-elementor) - [Shopify (Online Store 2.0 themes)recipe](#p-shopify) - [Webflowrecipe](#p-webflow) - [Framerrecipe](#p-framer) - [Wix (Wix Editor and Wix Studio)recipe](#p-wix) - [Squarespace (version 7.1)recipe](#p-squarespace) - [HubSpot CMS (Content Hub)recipe](#p-hubspot-cms) - [Docusaurusrecipe](#p-docusaurus) - [Mintlifyrecipe](#p-mintlify) - [GitBookrecipe](#p-gitbook) - [ReadMerecipe](#p-readme) - [Salesforce B2C Commerce (SFRA and PWA Kit / Composable Storefront)recipe](#p-salesforce-commerce-cloud) - [Adobe Commerce and Magento Open Sourcerecipe](#p-adobe-commerce) Last reviewed 2026-11-05. AgentReady v2.0, read 2026-11-04.