Control group is maintained throughout the experiment
Why it matters
An auto-winner mechanism that re-assigns control contacts to the leading variant partway through an experiment corrupts the experiment's results in two ways: it reduces the control group size mid-run (inflating variance) and introduces a survivor bias into the final comparison. Contacts who remained in the control group through the auto-switch may be systematically different from those who were switched. If the experiment hasn't reached required sample size when the auto-switch fires, you've now lost the ability to recover a clean result. You've shipped a variant based on preliminary data that may reverse at full sample — the classic peeking outcome — with no way to rerun the experiment cleanly.
Severity rationale
High because premature control-group dissolution forfeits the experiment's ability to produce a valid result, and the corrupted data still looks like a conclusive win, making the failure invisible without a methodological audit.
Remediation
Conclude experiments only through an explicit human action after significance is confirmed. Never auto-reassign historical contacts — apply the winner only to future sends:
async function concludeExperiment(
experimentId: string,
winnerId: string,
concludedBy: string
) {
const exp = await db.experiments.findUnique({ where: { id: experimentId } })
if (exp.status !== 'active') throw new Error('Experiment is not active')
await db.experiments.update({
where: { id: experimentId },
data: {
status: 'concluded',
winner_variant_id: winnerId,
concluded_at: new Date(),
concluded_by: concludedBy
}
})
// Apply winner to future sends only — historical assignments are immutable
await db.campaignDefaults.upsert({
where: { campaign_id: exp.campaign_id },
update: { default_variant_id: winnerId },
create: { campaign_id: exp.campaign_id, default_variant_id: winnerId }
})
}
Remove any timer-based or threshold-based auto-conclusion logic from your experiment scheduler.
DetectionPro
The detection method for this pattern is part of AuditBuffet Pro.
The full method (what to look for, pass and fail conditions, and how results get reported) ships inside the Campaign Analytics & Attribution Audit. The pattern definition, severity rationale, and remediation remain openly licensed under CC-BY-4.0.
External references
- iso-25010:2011 · functional-suitability — Functional suitability — functional correctness
Taxons
History
- 2026-04-18·v1.0.0·Initial import from campaign-analytics-attribution·automated