# Crawl budget is not wasted on no-index content

- **Pattern:** `ab-001687` (`marketing-advanced-seo.technical-seo.crawl-budget-efficiency`)
- **Severity:** info
- **Lifecycle:** active
- **Last modified:** 2026-04-18
- **Canonical URL:** https://auditbuffet.com/patterns/ab-001687
- **License:** CC-BY-4.0 — attribute to AuditBuffet Pattern Catalog (https://auditbuffet.com/patterns/ab-001687)

## Why it matters

Googlebot allocates a fixed crawl budget per site, and every request to a noindex admin or debug route is a request it did not spend on a page you actually want ranked. Noindex URLs in the sitemap send contradictory signals — submit this URL, but do not index it — which Google's Search Console flags as a coverage error and which slows indexation of legitimate new content.

## Severity rationale

Info because crawl-budget inefficiency mostly affects large sites and rarely suppresses individual page rankings directly.

## Remediation

Filter noindex pages out of the sitemap generator and remove links to consistently-noindex routes from the public navigation and footer. Scope admin/dashboard links to authenticated sessions so they never appear in public HTML. Verify the result by requesting the sitemap and confirming zero noindex URLs are listed.

```tsx
// app/sitemap.ts
const pages = allPages.filter(p => !p.noindex)
return pages.map(p => ({ url: p.url, lastModified: p.updatedAt }))
```

## Detection

- **ID:** `crawl-budget-efficiency`
- **Severity:** `info`
- **What to look for:** Count all noindex pages that appear in the sitemap or are linked from navigation. Enumerate wasted crawl paths. Check whether pages or sections marked as `noindex` are still reachable via internal links from indexable pages. Look for: (1) authenticated/admin route sections (e.g., `/admin/*`, `/dashboard/*`) that have noindex but are linked from public pages, (2) utility pages (`/api-docs`, `/debug`) that are noindex but appear in the site navigation or footer, (3) pages with `noindex` directives that appear as URL entries in the XML sitemap (contradictory signals).
- **Pass criteria:** No-index pages are not linked from indexed pages (or are linked only from pages that are themselves noindex), and no more than 0 noindex pages appear in the XML sitemap. Zero noindex pages should be included in the sitemap or linked from primary navigation.
- **Fail criteria:** Public indexed pages contain links to consistently-noindex pages. OR noindex pages appear as URL entries in the XML sitemap.
- **Skip (N/A) when:** No pages with noindex directives detected in the project.

- **Cross-reference:** For sitemap completeness, see the seo-advanced audit `sitemap-completeness` check.
- **Detail on fail:** `"Crawl budget inefficiency: /admin and /api-docs are set noindex but are linked from the public footer. These pages consume crawl budget without contributing to indexation. Also: /api-docs appears in sitemap.xml despite being noindex."`
- **Remediation:** For large sites, crawl budget matters. For smaller sites, this is lower priority but still worth cleaning up. Remove links to consistently-noindex pages from your public navigation. Remove noindex URLs from your sitemap. If admin or tool pages must appear in the nav, consider scoping those links to authenticated sessions only.

  ```tsx
  // app/sitemap.ts — exclude noindex pages
  const pages = allPages.filter(p => !p.noindex)
  return pages.map(p => ({ url: p.url, lastModified: p.updatedAt }))
  ```

Taxons: findability

HTML version: https://auditbuffet.com/patterns/ab-001687