Robots.txt accessible and doesn't block Googlebot from content, CSS, or JS

A Disallow rule on /_next/static/ or /assets/ blocks Googlebot from fetching the CSS and JavaScript it needs to render the page — Google then sees unstyled, scriptless markup and may classify your pages as broken or low-quality. Mobile-first indexing amplifies this: rendered content that depends on blocked resources gets scored as if it were missing, dropping rankings on the pages you care about most.

Keep public/robots.txt narrow — disallow only admin, API, and private paths; never block static asset directories. Reference the sitemap explicitly:

User-agent: *
Disallow: /admin
Disallow: /api/internal
Sitemap: https://yoursite.com/sitemap.xml

Verify with Search Console's URL Inspection tool after changes.

• ID: robots-txt-open

• Severity: medium

• What to look for: Read the public/robots.txt file. Count all Disallow rules. For each rule, verify it does not block content paths, CSS files, JavaScript files, or images needed for rendering. Enumerate any rules that block Googlebot specifically (via User-agent: Googlebot sections).

• Pass criteria: robots.txt exists in public/robots.txt, is parseable, has no more than 20 Disallow rules total, and zero of those rules block Googlebot from indexable content, CSS (/_next/static/), or JavaScript paths. Report: "robots.txt has X Disallow rules; 0 block essential resources."

• Fail criteria: Googlebot is blocked from indexable content, CSS, or JS by any Disallow rule, or robots.txt is missing or unparseable.

• Skip (N/A) when: No public/robots.txt file exists and no app/robots.ts generates one.

• Detail on fail: "robots.txt has 4 Disallow rules; 1 blocks /assets/ path, preventing CSS and JS rendering".

• Remediation: Ensure public/robots.txt allows crawling of essential resources:

  User-agent: *
  Allow: /
  Disallow: /admin
  Disallow: /private
  
  Allow: /assets/
  Allow: /scripts/
  
  Sitemap: https://yoursite.com/sitemap.xml