Experiments are isolated — one active experiment per contact at a time, or interaction effects are handled

When a contact is simultaneously enrolled in a subject-line test and a send-time test that both use open rate as the outcome metric, the results of each experiment are contaminated by the other. You cannot isolate the effect of the subject line from the effect of the send time. Reported lifts from both experiments are artifacts of the interaction, not the individual treatments. In practice this leads to shipping changes that performed well in a contaminated test but have no effect or a negative effect in production. The data-integrity failure is structural: the experiment framework appears to produce results, but those results are not attributable to the variable being tested.

Implement exclusion groups so contacts enrolled in one experiment sharing a metric are blocked from enrollment in another. Check at assignment time, not at analysis time:

async function canEnrollContact(
  contactId: string,
  experimentId: string
): Promise<boolean> {
  const experiment = await db.experiments.findUnique({ where: { id: experimentId } })
  if (!experiment.exclusion_group) return true

  const conflict = await db.experimentAssignments.findFirst({
    where: {
      contact_id: contactId,
      experiment: {
        exclusion_group: experiment.exclusion_group,
        status: 'active',
        id: { not: experimentId }
      }
    }
  })

  return conflict === null
}

Tag every experiment with an exclusion_group string ('open-rate', 'click-rate', etc.) at creation time. Experiments testing independent levers (e.g., subject line vs. footer copy) can share a group if they share the primary outcome metric.

• ID: campaign-analytics-attribution.ab-testing.experiment-isolation

• Severity: medium

• What to look for: Look at how multiple simultaneous experiments interact. If a contact can be enrolled in multiple A/B tests at the same time, check whether the system either (a) prevents simultaneous enrollment in tests that affect the same metric (e.g., open rate tests and subject-line tests), or (b) explicitly models interaction effects when reporting results. Check for experiment exclusion groups, mutual exclusion logic, or documentation of overlapping experiment handling. The failure mode is silently enrolling contacts in multiple experiments that share the same metric without accounting for the interaction.

• Pass criteria: Experiments use mutual exclusion groups (contacts in experiment A cannot be in experiment B for overlapping metrics), OR the system tracks which experiments a contact is enrolled in and uses interaction-aware analysis, OR experiments are designed to test independent levers with documented independence assumptions. Count all active experiment configurations and enumerate which ones share metrics — no more than 1 pair should share an outcome metric without an exclusion group.

• Fail criteria: Contacts can be enrolled in multiple simultaneous experiments affecting the same metric with no isolation or interaction modeling. No experiment exclusion logic found.

• Skip (N/A) when: The project runs at most one experiment at a time (serial experimentation), or does not run A/B tests.

• Detail on fail: Example: "No experiment exclusion found — contacts can simultaneously be in a subject-line test and a send-time test, contaminating both results" or "Multiple active experiments share the same metric with no isolation layer"

• Remediation: Implement experiment exclusion groups:

  // Before enrolling a contact in an experiment, check exclusions
  async function canEnrollContact(contactId: string, experimentId: string): Promise<boolean> {
    const experiment = await db.experiments.findUnique({ where: { id: experimentId } })
    if (!experiment.exclusion_group) return true // No exclusion group set
  
    // Check if contact is already in another experiment in the same exclusion group
    const existingEnrollment = await db.experimentAssignments.findFirst({
      where: {
        contact_id: contactId,
        experiment: {
          exclusion_group: experiment.exclusion_group,
          status: 'active',
          id: { not: experimentId }
        }
      }
    })
  
    return existingEnrollment === null
  }