Control group is maintained throughout the experiment

An auto-winner mechanism that re-assigns control contacts to the leading variant partway through an experiment corrupts the experiment's results in two ways: it reduces the control group size mid-run (inflating variance) and introduces a survivor bias into the final comparison. Contacts who remained in the control group through the auto-switch may be systematically different from those who were switched. If the experiment hasn't reached required sample size when the auto-switch fires, you've now lost the ability to recover a clean result. You've shipped a variant based on preliminary data that may reverse at full sample — the classic peeking outcome — with no way to rerun the experiment cleanly.

Conclude experiments only through an explicit human action after significance is confirmed. Never auto-reassign historical contacts — apply the winner only to future sends:

async function concludeExperiment(
  experimentId: string,
  winnerId: string,
  concludedBy: string
) {
  const exp = await db.experiments.findUnique({ where: { id: experimentId } })
  if (exp.status !== 'active') throw new Error('Experiment is not active')

  await db.experiments.update({
    where: { id: experimentId },
    data: {
      status: 'concluded',
      winner_variant_id: winnerId,
      concluded_at: new Date(),
      concluded_by: concludedBy
    }
  })

  // Apply winner to future sends only — historical assignments are immutable
  await db.campaignDefaults.upsert({
    where:  { campaign_id: exp.campaign_id },
    update: { default_variant_id: winnerId },
    create: { campaign_id: exp.campaign_id, default_variant_id: winnerId }
  })
}

Remove any timer-based or threshold-based auto-conclusion logic from your experiment scheduler.

• ID: campaign-analytics-attribution.ab-testing.control-group-maintained

• Severity: high

• What to look for: Examine how the control group is handled once an experiment is running. Look for whether contacts assigned to the control variant continue to receive the baseline treatment for the duration of the experiment, or whether early results cause the control group to be abandoned prematurely. Check whether winner selection automatically re-assigns the losing group (which would corrupt results for any ongoing measurement). Look for "auto-winner" logic that switches all contacts to the winning variant before the experiment formally concludes.

• Pass criteria: Control group assignments are persisted and remain stable throughout the experiment. No logic auto-reassigns contacts from control to variant during the experiment run. Winner selection only occurs as an explicit action after the experiment is formally concluded. Enumerate all code paths that modify experiment assignments and count how many include a status check — no more than 0 should bypass the concluded check.

• Fail criteria: Control group contacts are automatically moved to the winning variant during the experiment run. Assignments are not persisted, allowing re-assignment on subsequent sends within the same experiment. Auto-winner selection changes treatment mid-experiment.

• Skip (N/A) when: The project does not run A/B tests.

• Detail on fail: Example: "Auto-winner logic re-assigns control contacts to the leading variant after 48 hours — experiment concludes early without significance check" or "Assignments not persisted — control contacts may receive variant treatment in follow-up sends"

• Remediation: Persist assignments and prevent premature closure:

  // Experiment can only be concluded through explicit action, not automated timer
  async function concludeExperiment(experimentId: string, winnerId: string, concludedBy: string) {
    const exp = await db.experiments.findUnique({ where: { id: experimentId } })
    if (exp.status !== 'active') throw new Error('Experiment is not active')
  
    // Mark concluded without reassigning historical contacts
    await db.experiments.update({
      where: { id: experimentId },
      data: {
        status: 'concluded',
        winner_variant_id: winnerId,
        concluded_at: new Date(),
        concluded_by: concludedBy
      }
    })
  
    // Future sends use the winner — past assignments are never touched
    await db.campaignDefaults.upsert({
      where: { campaign_id: exp.campaign_id },
      update: { default_variant_id: winnerId },
      create: { campaign_id: exp.campaign_id, default_variant_id: winnerId }
    })
  }