No aggressive bot detection on content pages

Cloudflare Turnstile, reCAPTCHA, and user-agent-sniffing middleware that challenge every request will block OAI-SearchBot, PerplexityBot, Claude-SearchBot, and the other crawlers that feed AI answers, alongside malicious traffic. This failure mode is serious enough that the vendors document the fix themselves — OpenAI asks site owners to "ensure your site host and/or content delivery network allows traffic from our published IP addresses," and Perplexity's crawler docs include step-by-step Cloudflare and AWS WAF allowlisting instructions. A one-click "block AI bots" toggle in a CDN dashboard silently overrides a perfectly open robots.txt. When your blog and docs return a JavaScript challenge instead of HTML, AI systems record the page as uncrawlable and stop retrying — your content vanishes from generative answers even though it loads fine for humans.

Restrict bot protection middleware to API, auth, and form-submission routes using an explicit matcher, and leave content pages unprotected at the application layer. If a CDN or edge firewall enforces challenges, allowlist the AI search crawlers using the vendors' published IP ranges — openai.com/searchbot.json, perplexity.com/perplexitybot.json, claude.com/crawling/bots.json — or exclude content-only paths; OpenAI and Perplexity both publish first-party WAF-allowlisting instructions. Check CDN dashboards for one-click "block AI bots" toggles (Cloudflare, Vercel) — they override robots.txt silently. Update middleware.ts.

export const config = {
  matcher: ['/api/:path*', '/auth/:path*'],
}