robots.txt and sitemap are accessible
Why it matters
Googlebot, Bingbot, and every AI-crawler (GPTBot, ClaudeBot, PerplexityBot) check robots.txt first to learn what they're allowed to fetch, and they discover deep URLs through sitemap.xml. Missing both forces crawlers to spider links one hop at a time from the homepage, which means deep pages get crawled late or not at all. For GEO (Generative Engine Optimization), an unreachable sitemap means your pages don't make it into AI answer sets.
Severity rationale
Info because crawlers will eventually find most pages anyway, but missing files slow indexing and hurt AI discoverability.
Remediation
Ship robots.txt and sitemap.xml at the site root. For Next.js, generate the sitemap from a route handler at app/sitemap.ts and app/robots.ts so URLs stay in sync with routes. For static sites, place these files in public/:
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Verify both return HTTP 200 in production, not a SPA-rewritten HTML response.
DetectionPro
The detection method for this pattern is part of AuditBuffet Pro.
The full method (what to look for, pass and fail conditions, and how results get reported) ships inside the Site Health Check audit. The pattern definition, severity rationale, and remediation remain openly licensed under CC-BY-4.0.
Taxons
History
- 2026-04-18·v1.0.0·Initial import from site-health-check·automated