Search and citation crawlers are not blocked in robots.txt

A small set of robots.txt tokens controls whether AI answer engines can surface your site at all, and every vendor documents its own. OpenAI is categorical — sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers." Anthropic states that blocking Claude-SearchBot or Claude-User "may reduce your site's visibility" in Claude's search results. Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results." Google's AI Overviews and AI Mode are built on the ordinary Search index, so a Googlebot block removes you from those too, and Bing's index powers Microsoft Copilot and is a named third-party provider behind ChatGPT search. Blocking these search and citation crawlers silently removes your product from the fastest-growing discovery channel. Blocking training-only tokens like GPTBot or ClaudeBot is a different, documented choice that does not affect search visibility.

Remove Disallow rules targeting search and citation crawlers (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, bingbot) from public/robots.txt or app/robots.ts. OpenAI notes robots.txt changes take about 24 hours to propagate, and Anthropic requires the opt-out (or opt-in) per subdomain. A page blocked from OAI-SearchBot can still appear in ChatGPT as a bare title-and-link if discovered via a third-party search provider; only noindex removes that.

// app/robots.ts
export default function robots() {
  return {
    rules: [
      // One wildcard rule is enough. If you want to opt out of AI TRAINING
      // without losing search visibility, add Disallow entries only for the
      // training tokens (GPTBot, ClaudeBot, Google-Extended) — every vendor
      // documents these as independent of search appearance. Note that
      // blocking Google-Extended also opts you out of Gemini app grounding.
      { userAgent: '*', allow: '/', disallow: ['/api/', '/auth/'] },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}