Skip to content
FlagSwiftDocs

Read conversion by variant

Compare each variant's conversion with the control, read what the verdict means and know when the honest answer is to wait.

Time
About 8 min
Who can do it
Anyone with the metrics.read permission
Where
Flags, a flag's Analytics tab, Outcomes by variant
You finish with
A conversion comparison you can read with care

What and why#

Once your app reports exposures and outcomes, FlagSwift can tell you, for each variant, what share of the people shown it went on to do something, and whether that share differs from the control by more than chance would explain. This guide is about reading that table correctly, because a table full of green badges is easy to over-trust.

How the rate is calculated#

The rate is the number of people who did the event after being shown a variant, divided by the number of people shown that variant in the period. Each person counts once per variant and event, so a person who buys five times is one person. Repeats and the summed values you send are kept separately and feed the Per person column.

What the control is#

Every variant is compared with one control. For a flag without variants the control is off. For a flag with variants it is the variant the environment serves when the flag is off (its off variant), or the first variant if none is set. Because it depends on the environment, the control can differ between Staging and Production.

How the test works#

The comparison is a two-proportion z-test. It pools both groups to ask how surprising the gap would be if the variants were really the same, giving a two-sided p-value, and reports a 95% confidence interval for the difference in percentage points. A p-value below 0.05 is called a real difference. Nothing is run until both groups have at least 100 people.

Before you start#

  • Outcome analytics is on and your app reports exposures and outcomes. See Measure a feature's impact.
  • A flag that has been showing variants to real traffic for a while.
  • The metrics.read permission, which every built-in role has.
  • You have decided which event is the one that matters, before you open the page.

Walk through it#

  1. Open the outcomes

    Open Flags, choose the flag and open the Analytics tab. Scroll to Outcomes by variant. Choose the environment (Production is chosen first if the flag has it) and a range of 24 hours, 7 days (the default) or 30 days.

    You should see. A line listing each variant with how many people were shown it, such as green (control): 1,000 shown, then one table per event.
  2. Find the control

    The control is marked (control) next to its name in the line above the tables, and its row says Control in the last column. Every other variant is compared with it.

  3. Read one event at a time

    Each event has its own table, with the most common events first.

    One table per event. Each person counts once per variant and event. The purple variant has fewer than 100 people shown, so it says Too early and shows only the gap so far.
    Columns
    ColumnDescription
    PeopleHow many people did the event after being shown the variant. Each person counts once.
    RatePeople who did it, divided by people shown the variant.
    Per personThe summed value divided by the people who did it, when you send values. A dash when nobody did it or no value was sent.
    Against controlA verdict, the difference in percentage points, a 95% interval and a p-value.
  4. Read the verdict

    Verdicts
    VerdictDescription
    No dataOne of the two groups has nobody shown yet. No numbers are given.
    Too earlyA group has fewer than 100 people, so no test is run. The difference so far is shown, with no interval or p-value.
    No clear differenceEnough people, and the gap could be chance (p-value of 0.05 or more).
    Better / WorseThe gap is unlikely to be chance (p-value below 0.05), and the variant is above or below the control.
  5. Read a worked example

    Suppose 1,000 people saw the control and 100 of them bought (10.0%). 1,000 saw green and 130 bought (13.0%). The row for green reads Better, +3.0 pts (95%: +0.2 to +5.8) · p = 0.035. The interval says the true gain is plausibly anywhere from a fifth of a point to nearly six. The lower end is barely above zero, so this is a hint that deserves more data, not a mandate to ship.

Treat it as a signal

The p-value does not account for checking many events, or for checking again and again while data comes in. Both make a chance difference look real more often than the number says. There are no stopping rules or sample-size planning, and no automatic winner. If you look at ten events, expect about one in twenty chance differences to show as Better or Worse.

Read it from the API#

The same report is available from the dashboard API, which uses your signed-in session cookie and needs metrics.read.

Do it with the API

environment is required. days runs from 1 to 90 and is capped by the 90 days rows are kept; without it the report covers 14 days. The console's own selector offers 1, 7 and 30. Each row also carries count, valueSum, valuePerConverter and relativeLift, left out above. Up to 25 events are listed, and truncated is true when a period has more outcomes than one report reads (20,000).

Request
curl -s -b jar.txt \
  "https://flags.example.com/api/flags/$FLAG_ID/analytics?environment=production&days=14"
Response
{
  "status": "success",
  "data": {
    "enabled": true,
    "flagKey": "button-colour",
    "environment": "production",
    "days": 14,
    "control": "blue",
    "variants": [
      { "variant": "blue", "exposed": 1000 },
      { "variant": "green", "exposed": 1000 }
    ],
    "events": [{
      "event": "purchase",
      "rows": [
        { "variant": "blue", "converters": 100, "conversionRate": 0.1, "comparison": null },
        { "variant": "green", "converters": 130, "conversionRate": 0.13,
          "comparison": { "verdict": "better", "difference": 0.03, "interval": [0.002, 0.058], "pValue": 0.0355 } }
      ]
    }],
    "truncated": false,
    "eventsTruncated": false
  }
}

Good to know#

Wait for a full cycle

If behaviour differs between weekdays and weekends, choose a window that covers at least one whole week, and do not decide on the first day that a badge turns green.
Why does a variant say Too early when it has plenty of conversions?
The 100 is people shown the variant, not people who converted. Both the control and the variant need at least 100 people shown.
Can I compare two non-control variants with each other?
Not directly. Every variant is compared only with the control.
Why is Per person blank?
You did not send a value with the outcome, or nobody in that variant did the event.
Why are numbers lower than I expect?
Only the window you picked is read, and rows older than 90 days are deleted. If a period has more than 20,000 outcome rows, the figures cover the most recent ones and a note says so.

What to check#

  • Both groups have at least 100 people before you take any verdict seriously.
  • You decided which event matters before you looked, not after.
  • The window covers a full week of normal behaviour, if your traffic changes by day.
  • Exposures are being sent for every variant, not only the new one. A missing control exposure leaves the control group at zero.

If something goes wrong#

The section says outcome analysis is switched off

Why: The server is missing ANALYTICS_ENABLED=true.

Fix: Ask your operator to set it. Nothing sent before that was kept.

No outcomes reported for this flag in this period

Why: No exposure and outcome pair matched in the window, or the environment selected is not the one your app reports to.

Fix: Switch to the right environment and range, then check that exposures were sent first with the same user key.

Every row says No data

Why: Either the control or the variant has nobody shown.

Fix: Make sure your app reports exposures for every variant, including the control.

API call returns 400 about days

Why: days must be a whole number from 1 to 90.

Fix: Send an integer in that range.

API call returns 404 Environment does not exist for this flag

Why: The environment name is wrong, or the flag is not configured there.

Fix: Use the environment name exactly as it appears in Environments.