robots.txt exists and is properly configured

A missing robots.txt means crawlers operate without guidance: Googlebot and other search engines will crawl everything by default, including staging pages, admin routes, or duplicate parameter URLs that dilute crawl budget and can create duplicate-content penalties. A misconfigured Disallow: / blocks all crawlers from indexing the entire site, which causes complete de-indexing — the site disappears from search results within days. RFC 9309 formalizes the robots.txt protocol; non-compliance means crawler behavior is undefined across different search engines.

Create public/robots.txt with at minimum an allow-all rule and a sitemap reference:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

In Next.js App Router, generate it programmatically from app/robots.ts so the sitemap URL stays in sync with your environment:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/', disallow: ['/admin/', '/api/'] },
    sitemap: `${process.env.NEXT_PUBLIC_SITE_URL}/sitemap.xml`,
  }
}

Verify the live output at https://yoursite.com/robots.txt after deploying. Never set Disallow: / in production.