Measure a feature's impact with exposures and outcomes
Switch on outcome analytics, report what people were shown and what they did next, and read the result on the flag.
- Time
- About 10 min
- Who can do it
- An operator turns it on; anyone with metrics access can read
- Where
- Server settings, your app, and a flag's Analytics tab
- You finish with
- Conversion by variant on the flag
What and why#
Evaluation counts tell you how often a flag was read. They cannot tell you whether the people who saw the green button bought more than the people who saw the blue one. Outcome analytics answers that. Your app reports two things, and FlagSwift joins them for you.
Exposures and outcomes#
- An exposure says a person was shown a flag: the flag key and the variant (or on and off for a flag without variants).
- An outcome says a person did something that matters: an event name such as
purchaseand, optionally, a number such as the order total.
How an outcome is credited#
An outcome is credited to the variants that person was already shown in that environment. FlagSwift looks up the person's exposures, keeps the most recent variant for each flag, and credits the outcome to the 20 flags they were shown most recently. A person who was never shown a flag counts for no flag, so nobody can be credited to a variant they did not see. Each person counts once per flag, variant and event however often they repeat it. The number of repeats and the summed values are kept separately.
What is stored#
Never the key you send. FlagSwift keeps a keyed hash of it (HMAC-SHA-256 with your JWT_SECRET, scoped to the project), so the same person in two projects has two hashes. Only counts are kept, and rows are deleted after the retention period.
What this is and is not
Before you start#
- Access to the server settings. Nothing is recorded unless
ANALYTICS_ENABLED=trueis set. - A server or client key, and a flag with variants if you want to compare more than on and off. Reporting uses the
evaluation:readscope, which every SDK key has. - A stable key for each person that your app can use for both exposures and outcomes, up to 255 characters.
- To read the result you need the
metrics.readpermission. Every built-in role has it.
Walk through it#
Turn it on
Set
ANALYTICS_ENABLED=trueand restart the server. It is off by default. Rows are kept for 90 days. Old rows are removed in the background, at most once every six hours per instance.Do not rotate JWT_SECRET casually
The hashes are made withJWT_SECRET. Changing it changes every hash, so everyone starts over and earlier exposures can no longer be matched to new outcomes.You should see. With the setting off, reporting calls reply{ "enabled": false, "recorded": 0, "ignored": 0 }and keep nothing. With it on,enabledis true.Report an exposure
When a person is shown the flag, send an exposure with the same key you will use for outcomes. For a flag with variants, name the variant. For a flag without variants, send
enabledinstead.Exposure body Field Type Description userKey* string Who was shown the flag. Up to 255 characters, trimmed. Also accepted as user. exposures* array Between 1 and 50 entries per request. Duplicate entries in one request are collapsed. exposures[].flag* string The flag key. exposures[].variant string A variant key that exists on the flag, for a flag with variants. exposures[].enabled boolean What was served, for a flag without variants. Recorded as on or off. Each entry needs a
variantorenabled. An entry that names an unknown flag or variant is ignored, not rejected, so a client cannot invent rows.Report an outcome
When the person does something that matters, send an outcome.
Outcome body Field Type Description userKey* string The same key used for the exposure. event* string Letters, numbers, spaces and . _ / + - only, up to 64 characters. Other characters are removed. value number Optional, such as an order total. It must be finite and between -1,000,000,000 and 1,000,000,000. Values are summed per variant. Send the exposure first
An outcome is credited only to variants the person was already shown. An outcome with no earlier exposure is credited to nothing, and the reply saysrecorded: 0.Or use an SDK
Every SDK has
trackExposure,trackandflushAnalyticshelpers, so you do not call the endpoints yourself. Exposures are batched per person and sent about two seconds later. An outcome is sent straight away, after that person's exposures. Automatic exposures are off unless you turn onautoExposures. CallflushAnalyticsbefore a serverless function returns.Look at the result
Open Flags, choose the flag and open the Analytics tab. Scroll to Outcomes by variant, choose the environment and a range of 24 hours, 7 days or 30 days. It lists how many people were shown each variant, then one table per event.
What the section looks like once exposures are in but no outcome has been reported yet. After the first outcome, an event table appears under the variant line. You should see. The line above the tables reads likegreen: 412 shown, with the control marked. After a test purchase the event appears as its own table.
Do it with the API
Both endpoints answer 202. env is required for a project-wide key and optional for a key limited to one environment (naming a different one is a 403). For an exposure, recorded counts rows written and ignored counts unknown flags or variants. For an outcome, recorded is the number of flags it was credited to.
# Report that user-42 was shown the green variant
curl -X POST "https://flags.example.com/api/v1/analytics/exposures?env=production" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"userKey":"user-42","exposures":[{"flag":"button-colour","variant":"green"},{"flag":"checkout-v2","enabled":true}]}'
# Later, report that they bought something
curl -X POST "https://flags.example.com/api/v1/analytics/outcomes?env=production" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"userKey":"user-42","event":"purchase","value":49.9}'{ "enabled": true, "recorded": 2, "ignored": 0 }Or from an SDK
Go, Python, PHP, .NET, Flutter and Swift have the same three calls in their own style. A call with no identifier for the person is ignored.
// Browser, React, Vue, React Native, Next.js client: the current user
flags.trackExposure("button-colour");
await flags.track("purchase", 49.9);
// Node and other server SDKs: you say who
flags.trackExposure("button-colour", { identifier: user.id });
await flags.track("purchase", { identifier: user.id }, 49.9);
await flags.flushAnalytics(); // before a serverless function returnsGood to know#
Use one key for exposures and outcomes
Do I have to send an exposure for every request?
What if a person is shown two variants of one flag?
Does this slow my app down?
Can a browser report outcomes?
What to check#
- The exposure reply says how many were recorded and how many were ignored. Ignored means an unknown flag or variant.
- With analytics off, the reply is
{ "enabled": false }and nothing is kept. - The Analytics tab shows a message that outcome analysis is switched off if the server setting is missing.
- After a test purchase, the event appears under the flag. If it does not, check that the exposure used the same key and was sent first.
If something goes wrong#
The reply is { enabled: false } and the tab says outcomes are switched off
Why: The server is not running with ANALYTICS_ENABLED=true.
Fix: Set it, restart, and send the exposures again. Nothing was kept while it was off.
An exposure reply says ignored: 1
Why: The flag key does not exist in that environment, or the variant is not one of the flag's variants.
Fix: Check the exact key and variant spelling. For a flag without variants, send enabled instead of variant.
An outcome reply says recorded: 0
Why: That person has no exposure in this environment, or the exposure used a different key.
Fix: Send the exposure first, with the same userKey, and use the same env on both calls.
400 INVALID_EVENT or INVALID_VALUE
Why: The event name was empty after cleaning, or value was not a finite number within one billion.
Fix: Use letters, numbers and . _ / + - up to 64 characters, and send value as a number, not text.
The tab says This period has more outcomes than one report reads
Why: The period holds more outcome rows than a single report reads, so the figures cover the most recent ones. A second note says Only the most common events are listed when there are more than 25 events.
Fix: Choose a shorter range, or report fewer distinct event names.
400 ENV_REQUIRED
Why: A project-wide key was used without ?env=.
Fix: Add ?env=production (or the environment name), or use a key limited to one environment.