Monitor your flags
See how flags are being used, get told when something drifts and send the numbers to the monitoring you already run.
- Time
- About 10 min
- Who can do it
- metrics.read to look; an operator for Prometheus and logs
- Where
- Metrics, Errors, Alerts and the server
- You finish with
- Dashboards, alerts and a scrape target
What and why#
Feature flags sit on the hot path of your app, so you want to know three things: are my flags being read, is anything failing, and how fast is the API answering. FlagSwift answers these in the console and also exposes them to the tools your operators already use.
Four sources of signals#
| Source | Description |
|---|---|
| Metrics page | Evaluation volume, share served on, busiest flags and evaluation reasons, from hourly counts of server evaluations. |
| Errors page | API requests, error rate, latency (p50, p95, average), failed calls by type, flag, SDK, environment and endpoint, plus errors SDKs report about themselves. |
| Alerts | Rules over error rate, latency, volume and failed releases. See Set up alerts. |
| Prometheus, health and logs | Per-process counters and gauges, readiness and liveness endpoints, and structured JSON logs with a request id. |
What is counted#
Only counts are stored, never who was evaluated. Evaluation counts come from server evaluations. Flags an SDK evaluates locally from the downloaded configuration are not counted, so a flag can look unused while a backend uses it. Counts are hourly, kept for 30 days, and METRICS_ENABLED=false stops collection.
Before you start#
- Traffic that evaluates flags through the server endpoint, with a key.
- The
metrics.readpermission to open Metrics and Errors. Every built-in role has it. - To use Prometheus, set
METRICS_TOKENon the server. Without it the endpoint returns 404.
Walk through it#
Read Metrics and Errors
Metrics shows evaluation volume per project. Errors shows failures and latency. Both have a project selector, an environment selector (All environments by default) and a range of 24 hours, 7 days or 30 days. Check both after any release.
Tiles on the Errors page Tile Description API requests Calls made with an API key in the range. Error rate Failed calls divided by requests, with the count of failed calls. Latency p95 Estimated from latency buckets, so it is the upper bound of the bucket that holds the 95th percentile. Median and average are shown underneath. Reported by SDKs Errors SDKs met while evaluating, such as timeouts or unparsable configuration. The Metrics page: three tiles, then the busiest flags with the share served on. The Errors page for one project over the last 7 days: four tiles, then the breakdowns. You should see. Both pages move within a minute of real traffic. Counts are written about every 10 seconds and pages cache for 15 seconds.Create alert rules
Open Alerts and add rules for error rate, 95th percentile latency, a drop or spike in volume, or failed releases. Pick a window from 1 to 24 hours and a threshold. A rule with too few observations shows Not enough data rather than OK, so quiet periods do not raise false alarms. The full list of fields is in Set up alerts.
Alert rules are checked every minute. Each card shows its state, its current value and a History list. Where alerts go
Rules always show in the console and on the Overview page. If your operator setsALERT_WEBHOOK_URL, a message is also posted there, and with an email provider configured, people who can manage alerts are emailed. For paging or anything richer, scrape Prometheus and alert from your own system.Scrape Prometheus
Point Prometheus at
/api/prometheuswith the token as a bearer credential. It does not use a session or an API key. Values are per process and reset on restart, so scrape every instance and let Prometheus add them up.prometheus.yml scrape_configs: - job_name: flagswift metrics_path: /api/prometheus authorization: credentials: ${METRICS_TOKEN} static_configs: - targets: ["flags.example.com"]Series you get Series Description flagswift_evaluations_total Flag evaluations served, by outcome. flagswift_evaluation_errors_total Failed evaluation API calls and SDK-reported errors, by error type. flagswift_api_requests_total Evaluation API requests by endpoint and status. flagswift_api_request_duration_ms Request duration histogram in milliseconds, buckets 5 to 5000. flagswift_flag_changes_total, flagswift_release_changes_total Changes to flags and release events, by audit action. flagswift_realtime_connections Open streaming connections on this instance. flagswift_buffered_counters Counts waiting to be written, by recorder. A growing number means the database is slow or down. flagswift_cache_hits_total, flagswift_cache_misses_total Configuration and key cache lookups. flagswift_uptime_seconds Seconds since this instance started. Trace a single request
Every API response carries
X-Request-Id, and the server writes structured JSON log lines and audit entries with the same id. Send your ownX-Request-Id(8 to 128 letters, digits, dots, underscores or dashes) to follow a request from your app. Set the detail withLOG_LEVEL: debug, info (the default), warn, error or silent.Let SDKs report their own errors
SDKs report errors they meet with
POST /api/v1/evaluation/errors, which is how Reported by SDKs fills in. You normally do not call it yourself.
Do it with the API
The scrape returns 200 text, 401 with a wrong token and 404 when METRICS_TOKEN is not set. The errors report accepts up to 100 entries, each with errorType (required, 40 characters), and optional flagKey, count (1 to 10,000) and sdk. The Errors page API needs metrics.read.
# Scrape the metrics by hand
curl -s https://flags.example.com/api/prometheus -H "Authorization: Bearer $METRICS_TOKEN"
# Evaluation counts for a project, as the Metrics page reads them (dashboard API, session cookie)
curl -s -b jar.txt "https://flags.example.com/api/metrics?projectId=$PROJECT_ID&environment=production&days=7"
# Errors for a project, as the Errors page reads them (dashboard API, session cookie)
curl -s -b jar.txt "https://flags.example.com/api/observability/errors?projectId=$PROJECT_ID&days=7"
# An SDK-style report
curl -X POST "https://flags.example.com/api/v1/evaluation/errors?env=production" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"errors":[{"flagKey":"checkout","errorType":"TIMEOUT"}]}'{ "accepted": 1 }Good to know#
Put readiness on the load balancer
/health/ready answers 200 when the database responds within 2 seconds and 503 otherwise, and /health/live never touches the database. /health returns both with the uptime. They say only ok or down, never why; the reason is in the logs as health.database_down.Why are latency numbers always round?
Does Metrics include flags my backend evaluates locally?
Why do Prometheus counters drop after a deploy?
rate() and increase(), which handle resets.Can viewers see Errors and Metrics?
metrics.read, which every built-in role has.What to check#
- Hit the Prometheus endpoint with and without the token. Only the first should succeed.
- Metrics moves when you evaluate a flag through the server endpoint.
- A test alert with a low threshold fires, and the Alerts firing count on the Overview page changes.
- The
X-Request-Idyou send comes back on the response and appears in the log line for that call.
If something goes wrong#
Prometheus gets 404
Why: METRICS_TOKEN is not set on the server, which turns the endpoint off.
Fix: Set a long random token, restart, and send it as a bearer credential.
Prometheus gets 401
Why: The token does not match, or the scrape uses basic auth instead of a bearer credential.
Fix: Use authorization.credentials as shown and check for trailing spaces.
Metrics and Errors are empty
Why: Nobody has called the server endpoint with a key in that range, the environment filter excludes the traffic, or METRICS_ENABLED=false.
Fix: Widen the range, choose All environments and check the server setting.
Numbers differ between instances
Why: Prometheus values are per process.
Fix: Scrape every instance and sum, instead of scraping through the load balancer.
flagswift_buffered_counters keeps growing
Why: Counts cannot be written to the database fast enough, or the database is unreachable.
Fix: Check /health/ready and the database. Counts are held in memory in the meantime and retried.