Dead source detection with alerting

A scraper or API integration that stops producing records with no alert will silently starve your pipeline for days. You will not know until someone notices list growth has plateaued or a manual check reveals the last ingest timestamp is a week old. NIST CSF 2.0 DE.CM-1 requires continuous monitoring of system components, which extends to data ingestion pipelines. Without dead-source detection, outages in data acquisition are invisible until they become business-impacting.

Add a scheduled health-check job that queries the maximum acquired_at per source and alerts when any source exceeds its expected silence threshold.

// cron: every hour via cron job or Inngest function
async function checkSourceHealth() {
  const stale = await db.query(`
    SELECT source_id, MAX(acquired_at) AS last_seen
    FROM contacts
    GROUP BY source_id
    HAVING MAX(acquired_at) < NOW() - INTERVAL '24 hours'
  `)
  if (stale.rows.length > 0) {
    await sendAlert('Dead data sources detected', stale.rows)
  }
}

Set the silence threshold per source type — scraping sources may warrant a 6-hour threshold, purchased-list imports a 7-day threshold.