Ingestion throughput monitoring

Ingestion batch jobs with no structured output metrics are black boxes. You cannot tell whether a 30-minute job processed 10,000 records successfully or crashed after 100. NIST CSF 2.0 DE.AE-3 requires that event data be aggregated to support detection of anomalies. Without per-run throughput metrics — records processed, records quarantined, duration — you have no baseline against which to detect degradation, and no data to drive SLA decisions about ingestion pipeline capacity.

Emit a structured summary log at the end of every ingestion batch run. Include at minimum: source_id, processed, failed, quarantined, and duration_ms.

// src/lib/ingestion/batch.ts
export async function runIngestionBatch(sourceId: string, records: unknown[]) {
  const start = Date.now()
  let processed = 0, failed = 0, quarantined = 0

  for (const record of records) {
    try {
      await ingestContact(record)
      processed++
    } catch (err) {
      await quarantine(record, String(err), sourceId)
      quarantined++
      failed++
    }
  }

  logger.info('ingestion_batch_complete', {
    source_id:   sourceId,
    processed,
    failed,
    quarantined,
    duration_ms: Date.now() - start,
  })
}

If you use a job queue (BullMQ, pg-boss), write these metrics to the job result payload as well so they are queryable from the queue dashboard.