Malformed records quarantined, not silently discarded

Silently discarding records that fail validation destroys forensic visibility into data quality problems. A catch block that logs an error and continues leaves no persistent artifact of what was rejected, why, or how many records were affected. CWE-390 (Detection of Error Condition Without Action) applies directly — the error is detected but no remediation path exists. ISO 25010:2011 §4.2.5 (reliability) requires that fault conditions produce recoverable states. Without a quarantine store, a systematic problem in a purchased list (wrong column mapping, encoding issue) will produce thousands of silent discards that are invisible until someone notices the list's expected contact count does not match what was inserted.

Create a contact_quarantine table and write every rejected record to it with the raw payload and rejection reason preserved. Never discard a record that has been received by your ingestion pipeline.

CREATE TABLE contact_quarantine (
  id               TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
  raw_payload      JSONB NOT NULL,
  rejection_reason TEXT NOT NULL,
  quarantined_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  source_id        TEXT,
  reviewed         BOOLEAN NOT NULL DEFAULT FALSE
);

// src/lib/ingestion/quarantine.ts
export async function quarantine(
  raw: unknown,
  reason: string,
  sourceId?: string
) {
  await db.contactQuarantine.create({
    data: {
      raw_payload:      raw as Prisma.InputJsonValue,
      rejection_reason: reason,
      source_id:        sourceId ?? null,
    },
  })
}

Add a reviewed flag and a simple admin interface so operations teams can inspect, correct, and reprocess quarantined records.

• ID: data-sourcing-provenance.ingestion-pipeline.malformed-record-quarantine

• Severity: medium

• What to look for: Enumerate all error handling paths in the ingestion pipeline and classify each as "quarantine" or "discard." When a record fails validation or causes an error during insert, does the system: write it to a quarantine table or dead-letter queue for review, or silently catch and discard it? Look for catch blocks that simply continue or return without recording the rejected record. Also look for a quarantine table, dead_letter_queue, or error log that captures the raw rejected payload.

• Pass criteria: Malformed records that fail validation or insertion are written to at least 1 quarantine store (table, queue, or log) with the reason for rejection. The rejected record's raw payload is preserved. Operations teams can review and remediate quarantined records.

• Fail criteria: Validation failures are caught and discarded with only a log line (no persistent storage of the rejected payload). Malformed records are lost permanently with no way to recover or investigate.

• Skip (N/A) when: Not applicable — all ingestion pipelines should handle malformed records explicitly.

• Detail on fail: "Validation errors are logged but the raw record is discarded — malformed records cannot be investigated or reprocessed" or "catch block in ingestion loop calls continue with no quarantine".

• Remediation: Add a quarantine table and write to it on failure:

  CREATE TABLE contact_quarantine (
    id             TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
    raw_payload    JSONB NOT NULL,
    rejection_reason TEXT NOT NULL,
    quarantined_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    source_id      TEXT,
    reviewed       BOOLEAN NOT NULL DEFAULT FALSE
  );
  

  async function quarantine(raw: unknown, reason: string, sourceId?: string) {
    await db.contactQuarantine.create({
      data: {
        raw_payload:      raw as Prisma.InputJsonValue,
        rejection_reason: reason,
        source_id:        sourceId ?? null,
      },
    })
  }