robots.txt parser present and enforced for scrapers

Scraping without checking robots.txt is a legal and ethical violation that exposes your company to GDPR Art. 6 liability (no lawful basis for collecting personal data from disallowed pages) and CCPA §1798.100 scrutiny. CFAA and DMCA claims have been upheld against scrapers that ignored robots.txt. Beyond legal risk, platforms detect unconstrained scrapers and block your IP ranges, poisoning data quality for legitimate sources. CWE-749 (Exposed Dangerous Method or Function) applies when scraping code makes unchecked requests to external systems.

Wrap all outbound scraping requests in a robots.txt enforcement function. Cache the fetched robots.txt for at least 1 hour to avoid excessive fetching, and abort any request to a disallowed path before it executes.

import robotsParser from 'robots-parser'

const robotsCache = new Map<string, { robots: ReturnType<typeof robotsParser>; cachedAt: number }>()

async function isAllowed(url: string, userAgent: string): Promise<boolean> {
  const { protocol, host } = new URL(url)
  const robotsUrl = `${protocol}//${host}/robots.txt`
  const cached = robotsCache.get(host)
  let robots: ReturnType<typeof robotsParser>

  if (cached && Date.now() - cached.cachedAt < 3_600_000) {
    robots = cached.robots
  } else {
    const res = await fetch(robotsUrl)
    const text = res.ok ? await res.text() : ''
    robots = robotsParser(robotsUrl, text)
    robotsCache.set(host, { robots, cachedAt: Date.now() })
  }
  return robots.isAllowed(url, userAgent) ?? true
}

// In every scraper:
if (!(await isAllowed(targetUrl, 'MyBot/1.0'))) {
  logger.warn('robots_blocked', { url: targetUrl })
  return
}

This check must run at 100% of scrapers — a single scraper path without it is a fail.