Read conversion by variant
Compare each variant's conversion with the control, read what the verdict means and know when the honest answer is to wait.
- Time
- About 8 min
- Who can do it
- Anyone with the metrics.read permission
- Where
- Flags, a flag's Analytics tab, Outcomes by variant
- You finish with
- A conversion comparison you can read with care
What and why#
Once your app reports exposures and outcomes, FlagSwift can tell you, for each variant, what share of the people shown it went on to do something, and whether that share differs from the control by more than chance would explain. This guide is about reading that table correctly, because a table full of green badges is easy to over-trust.
How the rate is calculated#
The rate is the number of people who did the event after being shown a variant, divided by the number of people shown that variant in the period. Each person counts once per variant and event, so a person who buys five times is one person. Repeats and the summed values you send are kept separately and feed the Per person column.
What the control is#
Every variant is compared with one control. For a flag without variants the control is off. For a flag with variants it is the variant the environment serves when the flag is off (its off variant), or the first variant if none is set. Because it depends on the environment, the control can differ between Staging and Production.
How the test works#
The comparison is a two-proportion z-test. It pools both groups to ask how surprising the gap would be if the variants were really the same, giving a two-sided p-value, and reports a 95% confidence interval for the difference in percentage points. A p-value below 0.05 is called a real difference. Nothing is run until both groups have at least 100 people.
Before you start#
- Outcome analytics is on and your app reports exposures and outcomes. See Measure a feature's impact.
- A flag that has been showing variants to real traffic for a while.
- The
metrics.readpermission, which every built-in role has. - You have decided which event is the one that matters, before you open the page.
Walk through it#
Open the outcomes
Open Flags, choose the flag and open the Analytics tab. Scroll to Outcomes by variant. Choose the environment (Production is chosen first if the flag has it) and a range of 24 hours, 7 days (the default) or 30 days.
You should see. A line listing each variant with how many people were shown it, such asgreen (control): 1,000 shown, then one table per event.Find the control
The control is marked
(control)next to its name in the line above the tables, and its row says Control in the last column. Every other variant is compared with it.Read one event at a time
Each event has its own table, with the most common events first.
One table per event. Each person counts once per variant and event. The purple variant has fewer than 100 people shown, so it says Too early and shows only the gap so far. Columns Column Description People How many people did the event after being shown the variant. Each person counts once. Rate People who did it, divided by people shown the variant. Per person The summed value divided by the people who did it, when you send values. A dash when nobody did it or no value was sent. Against control A verdict, the difference in percentage points, a 95% interval and a p-value. Read the verdict
Verdicts Verdict Description No data One of the two groups has nobody shown yet. No numbers are given. Too early A group has fewer than 100 people, so no test is run. The difference so far is shown, with no interval or p-value. No clear difference Enough people, and the gap could be chance (p-value of 0.05 or more). Better / Worse The gap is unlikely to be chance (p-value below 0.05), and the variant is above or below the control. Read a worked example
Suppose 1,000 people saw the control and 100 of them bought (10.0%). 1,000 saw
greenand 130 bought (13.0%). The row forgreenreadsBetter,+3.0 pts (95%: +0.2 to +5.8) · p = 0.035. The interval says the true gain is plausibly anywhere from a fifth of a point to nearly six. The lower end is barely above zero, so this is a hint that deserves more data, not a mandate to ship.
Treat it as a signal
Read it from the API#
The same report is available from the dashboard API, which uses your signed-in session cookie and needs metrics.read.
Do it with the API
environment is required. days runs from 1 to 90 and is capped by the 90 days rows are kept; without it the report covers 14 days. The console's own selector offers 1, 7 and 30. Each row also carries count, valueSum, valuePerConverter and relativeLift, left out above. Up to 25 events are listed, and truncated is true when a period has more outcomes than one report reads (20,000).
curl -s -b jar.txt \
"https://flags.example.com/api/flags/$FLAG_ID/analytics?environment=production&days=14"{
"status": "success",
"data": {
"enabled": true,
"flagKey": "button-colour",
"environment": "production",
"days": 14,
"control": "blue",
"variants": [
{ "variant": "blue", "exposed": 1000 },
{ "variant": "green", "exposed": 1000 }
],
"events": [{
"event": "purchase",
"rows": [
{ "variant": "blue", "converters": 100, "conversionRate": 0.1, "comparison": null },
{ "variant": "green", "converters": 130, "conversionRate": 0.13,
"comparison": { "verdict": "better", "difference": 0.03, "interval": [0.002, 0.058], "pValue": 0.0355 } }
]
}],
"truncated": false,
"eventsTruncated": false
}
}Good to know#
Wait for a full cycle
Why does a variant say Too early when it has plenty of conversions?
Can I compare two non-control variants with each other?
Why is Per person blank?
value with the outcome, or nobody in that variant did the event.Why are numbers lower than I expect?
What to check#
- Both groups have at least 100 people before you take any verdict seriously.
- You decided which event matters before you looked, not after.
- The window covers a full week of normal behaviour, if your traffic changes by day.
- Exposures are being sent for every variant, not only the new one. A missing control exposure leaves the control group at zero.
If something goes wrong#
The section says outcome analysis is switched off
Why: The server is missing ANALYTICS_ENABLED=true.
Fix: Ask your operator to set it. Nothing sent before that was kept.
No outcomes reported for this flag in this period
Why: No exposure and outcome pair matched in the window, or the environment selected is not the one your app reports to.
Fix: Switch to the right environment and range, then check that exposures were sent first with the same user key.
Every row says No data
Why: Either the control or the variant has nobody shown.
Fix: Make sure your app reports exposures for every variant, including the control.
API call returns 400 about days
Why: days must be a whole number from 1 to 90.
Fix: Send an integer in that range.
API call returns 404 Environment does not exist for this flag
Why: The environment name is wrong, or the flag is not configured there.
Fix: Use the environment name exactly as it appears in Environments.