Dedup check before insert
Why it matters
Duplicate contact records from multiple ingestion runs inflate your list size, distort per-source quality metrics, and cause repeated outreach to the same person — a direct CAN-SPAM and GDPR Art. 5(1)(c) (data minimization) violation. A raw INSERT that relies on a unique constraint exception for dedup handling is not graceful deduplication — it is crash-driven dedup that requires explicit exception handling at every call site or risks silent swallowing of duplicates. CWE-694 (Use of Multiple Resources with Duplicate Identifier) applies when duplicate identity keys are not resolved at the insertion boundary.
Severity rationale
High because ungraceful duplicate handling either corrupts the contacts table with true duplicates or creates unhandled exceptions in ingestion pipelines, both of which degrade data integrity and pipeline reliability.
Remediation
Replace raw INSERTs with an explicit upsert pattern that handles the duplicate case intentionally — updating only non-provenance fields on conflict so the original provenance record is preserved.
// Prisma upsert — provenance fields are in `create`, not `update`
await db.contacts.upsert({
where: { email: contact.email },
create: {
email: contact.email,
source_type: contact.source_type,
source_id: contact.source_id,
acquired_at: contact.acquired_at,
},
update: {
last_seen_at: new Date(), // Only update non-provenance fields
},
})
-- Raw SQL alternative
INSERT INTO contacts (email, source_type, source_id, acquired_at)
VALUES ($1, $2, $3, $4)
ON CONFLICT (email) DO UPDATE
SET last_seen_at = NOW();
A unique constraint on email remains required as a database-level safety net, but it should never be the primary dedup mechanism.
DetectionPro
The detection method for this pattern is part of AuditBuffet Pro.
The full method (what to look for, pass and fail conditions, and how results get reported) ships inside the Data Sourcing & Provenance Audit. The pattern definition, severity rationale, and remediation remain openly licensed under CC-BY-4.0.
External references
- cwe · CWE-694 — Use of Multiple Resources with Duplicate Identifier
- iso-25010:2011 · functional-suitability.functional-correctness — Functional Correctness — duplicate records violate data integrity
- gdpr · Art. 5(1)(c) — Data minimisation
Taxons
History
- 2026-04-18·v1.0.0·Initial import from data-sourcing-provenance·automated