# Robots.txt accessible and doesn't block Googlebot from content, CSS, or JS

- **Pattern:** `ab-002490` (`seo-advanced.crawlability.robots-txt-open`)
- **Severity:** medium
- **Lifecycle:** active
- **Last modified:** 2026-04-18
- **Canonical URL:** https://auditbuffet.com/patterns/ab-002490
- **License:** CC-BY-4.0 — attribute to AuditBuffet Pattern Catalog (https://auditbuffet.com/patterns/ab-002490)

## Why it matters

A `Disallow` rule on `/_next/static/` or `/assets/` blocks Googlebot from fetching the CSS and JavaScript it needs to render the page — Google then sees unstyled, scriptless markup and may classify your pages as broken or low-quality. Mobile-first indexing amplifies this: rendered content that depends on blocked resources gets scored as if it were missing, dropping rankings on the pages you care about most.

## Severity rationale

Medium because blocked render resources degrade ranking signals but core HTML still reaches the crawler.

## Remediation

Keep `public/robots.txt` narrow — disallow only admin, API, and private paths; never block static asset directories. Reference the sitemap explicitly:

```
User-agent: *
Disallow: /admin
Disallow: /api/internal
Sitemap: https://yoursite.com/sitemap.xml
```

Verify with Search Console's URL Inspection tool after changes.

## Detection

- **ID:** `robots-txt-open`
- **Severity:** `medium`
- **What to look for:** Read the `public/robots.txt` file. Count all `Disallow` rules. For each rule, verify it does not block content paths, CSS files, JavaScript files, or images needed for rendering. Enumerate any rules that block Googlebot specifically (via `User-agent: Googlebot` sections).
- **Pass criteria:** robots.txt exists in `public/robots.txt`, is parseable, has no more than 20 `Disallow` rules total, and zero of those rules block Googlebot from indexable content, CSS (`/_next/static/`), or JavaScript paths. Report: "robots.txt has X Disallow rules; 0 block essential resources."
- **Fail criteria:** Googlebot is blocked from indexable content, CSS, or JS by any `Disallow` rule, or robots.txt is missing or unparseable.
- **Skip (N/A) when:** No `public/robots.txt` file exists and no `app/robots.ts` generates one.
- **Detail on fail:** `"robots.txt has 4 Disallow rules; 1 blocks /assets/ path, preventing CSS and JS rendering"`.
- **Remediation:** Ensure `public/robots.txt` allows crawling of essential resources:

  ```
  User-agent: *
  Allow: /
  Disallow: /admin
  Disallow: /private

  Allow: /assets/
  Allow: /scripts/

  Sitemap: https://yoursite.com/sitemap.xml
  ```

Taxons: findability

HTML version: https://auditbuffet.com/patterns/ab-002490