# Malformed records quarantined, not silently discarded

- **Pattern:** `ab-000894` (`data-sourcing-provenance.ingestion-pipeline.malformed-record-quarantine`)
- **Severity:** medium
- **Lifecycle:** active
- **Last modified:** 2026-04-18
- **Canonical URL:** https://auditbuffet.com/patterns/ab-000894
- **License:** CC-BY-4.0 — attribute to AuditBuffet Pattern Catalog (https://auditbuffet.com/patterns/ab-000894)

## Why it matters

Silently discarding records that fail validation destroys forensic visibility into data quality problems. A `catch` block that logs an error and continues leaves no persistent artifact of what was rejected, why, or how many records were affected. CWE-390 (Detection of Error Condition Without Action) applies directly — the error is detected but no remediation path exists. ISO 25010:2011 §4.2.5 (reliability) requires that fault conditions produce recoverable states. Without a quarantine store, a systematic problem in a purchased list (wrong column mapping, encoding issue) will produce thousands of silent discards that are invisible until someone notices the list's expected contact count does not match what was inserted.

## Severity rationale

Medium because silent discards make systematic ingestion failures invisible, prevent reprocessing when the root cause is fixed, and violate the principle that all received data should have an auditable disposition.

## Remediation

Create a `contact_quarantine` table and write every rejected record to it with the raw payload and rejection reason preserved. Never discard a record that has been received by your ingestion pipeline.

```sql
CREATE TABLE contact_quarantine (
  id               TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
  raw_payload      JSONB NOT NULL,
  rejection_reason TEXT NOT NULL,
  quarantined_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  source_id        TEXT,
  reviewed         BOOLEAN NOT NULL DEFAULT FALSE
);
```

```ts
// src/lib/ingestion/quarantine.ts
export async function quarantine(
  raw: unknown,
  reason: string,
  sourceId?: string
) {
  await db.contactQuarantine.create({
    data: {
      raw_payload:      raw as Prisma.InputJsonValue,
      rejection_reason: reason,
      source_id:        sourceId ?? null,
    },
  })
}
```

Add a `reviewed` flag and a simple admin interface so operations teams can inspect, correct, and reprocess quarantined records.

## Detection

- **ID:** `malformed-record-quarantine`
- **Severity:** `medium`
- **What to look for:** Enumerate all error handling paths in the ingestion pipeline and classify each as "quarantine" or "discard." When a record fails validation or causes an error during insert, does the system: write it to a quarantine table or dead-letter queue for review, or silently catch and discard it? Look for `catch` blocks that simply `continue` or `return` without recording the rejected record. Also look for a `quarantine` table, `dead_letter_queue`, or error log that captures the raw rejected payload.
- **Pass criteria:** Malformed records that fail validation or insertion are written to at least 1 quarantine store (table, queue, or log) with the reason for rejection. The rejected record's raw payload is preserved. Operations teams can review and remediate quarantined records.
- **Fail criteria:** Validation failures are caught and discarded with only a log line (no persistent storage of the rejected payload). Malformed records are lost permanently with no way to recover or investigate.
- **Skip (N/A) when:** Not applicable — all ingestion pipelines should handle malformed records explicitly.
- **Detail on fail:** `"Validation errors are logged but the raw record is discarded — malformed records cannot be investigated or reprocessed"` or `"catch block in ingestion loop calls continue with no quarantine"`.
- **Remediation:** Add a quarantine table and write to it on failure:

  ```sql
  CREATE TABLE contact_quarantine (
    id             TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
    raw_payload    JSONB NOT NULL,
    rejection_reason TEXT NOT NULL,
    quarantined_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    source_id      TEXT,
    reviewed       BOOLEAN NOT NULL DEFAULT FALSE
  );
  ```

  ```ts
  async function quarantine(raw: unknown, reason: string, sourceId?: string) {
    await db.contactQuarantine.create({
      data: {
        raw_payload:      raw as Prisma.InputJsonValue,
        rejection_reason: reason,
        source_id:        sourceId ?? null,
      },
    })
  }
  ```

## External references

- cwe CWE-390
- iso-25010 reliability.availability

Taxons: data-integrity, observability

HTML version: https://auditbuffet.com/patterns/ab-000894