Experiment Agent

Runs A/B tests on subject lines, send times, and copy variants. Decides when a winner is statistically significant and auto-promotes it.

Defining an experiment

{
  "name": "subject-line-renewal-q2",
  "metric": "click_rate",
  "variants": [
    { "id": "A", "templateSlug": "renewal-v1" },
    { "id": "B", "templateSlug": "renewal-v2" }
  ],
  "trafficSplit": { "A": 0.5, "B": 0.5 },
  "minSampleSize": 500,
  "maxDurationDays": 14
}

How it decides

Hourly: check if both arms have hit minSampleSize. If yes, run a two-sample z-test on the chosen metric. p < 0.05 + practical lift > 5% ⇒ winner. p > 0.5 + sample > 2x min ⇒ called as no-effect.

Auto-promotion

When a winner is declared, the experiment's campaign step is rewritten to use the winning template and the experiment is marked completed. The losing template stays in the registry — easy to revive.

Schedule

Every hour via /api/cron/experiment-agent. Allocation happens at send-time (sticky to the profile so a user always sees the same variant).