Skip to content
FlagSwiftDocs

Monitor your flags

See how flags are being used, get told when something drifts and send the numbers to the monitoring you already run.

Time
About 10 min
Who can do it
metrics.read to look; an operator for Prometheus and logs
Where
Metrics, Errors, Alerts and the server
You finish with
Dashboards, alerts and a scrape target

What and why#

Feature flags sit on the hot path of your app, so you want to know three things: are my flags being read, is anything failing, and how fast is the API answering. FlagSwift answers these in the console and also exposes them to the tools your operators already use.

Four sources of signals#

Where each signal lives
SourceDescription
Metrics pageEvaluation volume, share served on, busiest flags and evaluation reasons, from hourly counts of server evaluations.
Errors pageAPI requests, error rate, latency (p50, p95, average), failed calls by type, flag, SDK, environment and endpoint, plus errors SDKs report about themselves.
AlertsRules over error rate, latency, volume and failed releases. See Set up alerts.
Prometheus, health and logsPer-process counters and gauges, readiness and liveness endpoints, and structured JSON logs with a request id.

What is counted#

Only counts are stored, never who was evaluated. Evaluation counts come from server evaluations. Flags an SDK evaluates locally from the downloaded configuration are not counted, so a flag can look unused while a backend uses it. Counts are hourly, kept for 30 days, and METRICS_ENABLED=false stops collection.

Before you start#

  • Traffic that evaluates flags through the server endpoint, with a key.
  • The metrics.read permission to open Metrics and Errors. Every built-in role has it.
  • To use Prometheus, set METRICS_TOKEN on the server. Without it the endpoint returns 404.

Walk through it#

  1. Read Metrics and Errors

    Metrics shows evaluation volume per project. Errors shows failures and latency. Both have a project selector, an environment selector (All environments by default) and a range of 24 hours, 7 days or 30 days. Check both after any release.

    Tiles on the Errors page
    TileDescription
    API requestsCalls made with an API key in the range.
    Error rateFailed calls divided by requests, with the count of failed calls.
    Latency p95Estimated from latency buckets, so it is the upper bound of the bucket that holds the 95th percentile. Median and average are shown underneath.
    Reported by SDKsErrors SDKs met while evaluating, such as timeouts or unparsable configuration.
    The Metrics page: three tiles, then the busiest flags with the share served on.
    The Errors page for one project over the last 7 days: four tiles, then the breakdowns.
    You should see. Both pages move within a minute of real traffic. Counts are written about every 10 seconds and pages cache for 15 seconds.
  2. Create alert rules

    Open Alerts and add rules for error rate, 95th percentile latency, a drop or spike in volume, or failed releases. Pick a window from 1 to 24 hours and a threshold. A rule with too few observations shows Not enough data rather than OK, so quiet periods do not raise false alarms. The full list of fields is in Set up alerts.

    Alert rules are checked every minute. Each card shows its state, its current value and a History list.

    Where alerts go

    Rules always show in the console and on the Overview page. If your operator sets ALERT_WEBHOOK_URL, a message is also posted there, and with an email provider configured, people who can manage alerts are emailed. For paging or anything richer, scrape Prometheus and alert from your own system.
  3. Scrape Prometheus

    Point Prometheus at /api/prometheus with the token as a bearer credential. It does not use a session or an API key. Values are per process and reset on restart, so scrape every instance and let Prometheus add them up.

    prometheus.yml
    scrape_configs:
      - job_name: flagswift
        metrics_path: /api/prometheus
        authorization:
          credentials: ${METRICS_TOKEN}
        static_configs:
          - targets: ["flags.example.com"]
    Series you get
    SeriesDescription
    flagswift_evaluations_totalFlag evaluations served, by outcome.
    flagswift_evaluation_errors_totalFailed evaluation API calls and SDK-reported errors, by error type.
    flagswift_api_requests_totalEvaluation API requests by endpoint and status.
    flagswift_api_request_duration_msRequest duration histogram in milliseconds, buckets 5 to 5000.
    flagswift_flag_changes_total, flagswift_release_changes_totalChanges to flags and release events, by audit action.
    flagswift_realtime_connectionsOpen streaming connections on this instance.
    flagswift_buffered_countersCounts waiting to be written, by recorder. A growing number means the database is slow or down.
    flagswift_cache_hits_total, flagswift_cache_misses_totalConfiguration and key cache lookups.
    flagswift_uptime_secondsSeconds since this instance started.
  4. Trace a single request

    Every API response carries X-Request-Id, and the server writes structured JSON log lines and audit entries with the same id. Send your own X-Request-Id (8 to 128 letters, digits, dots, underscores or dashes) to follow a request from your app. Set the detail with LOG_LEVEL: debug, info (the default), warn, error or silent.

  5. Let SDKs report their own errors

    SDKs report errors they meet with POST /api/v1/evaluation/errors, which is how Reported by SDKs fills in. You normally do not call it yourself.

Do it with the API

The scrape returns 200 text, 401 with a wrong token and 404 when METRICS_TOKEN is not set. The errors report accepts up to 100 entries, each with errorType (required, 40 characters), and optional flagKey, count (1 to 10,000) and sdk. The Errors page API needs metrics.read.

Request
# Scrape the metrics by hand
curl -s https://flags.example.com/api/prometheus -H "Authorization: Bearer $METRICS_TOKEN"

# Evaluation counts for a project, as the Metrics page reads them (dashboard API, session cookie)
curl -s -b jar.txt "https://flags.example.com/api/metrics?projectId=$PROJECT_ID&environment=production&days=7"

# Errors for a project, as the Errors page reads them (dashboard API, session cookie)
curl -s -b jar.txt "https://flags.example.com/api/observability/errors?projectId=$PROJECT_ID&days=7"

# An SDK-style report
curl -X POST "https://flags.example.com/api/v1/evaluation/errors?env=production" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"errors":[{"flagKey":"checkout","errorType":"TIMEOUT"}]}'
SDK report response (202)
{ "accepted": 1 }

Good to know#

Put readiness on the load balancer

/health/ready answers 200 when the database responds within 2 seconds and 503 otherwise, and /health/live never touches the database. /health returns both with the uptime. They say only ok or down, never why; the reason is in the logs as health.database_down.
Why are latency numbers always round?
They are estimated from buckets (5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000 ms), so a percentile is the upper bound of its bucket. Anything slower than 5 seconds is counted as 10,000 ms.
Does Metrics include flags my backend evaluates locally?
No. Only server evaluations are counted. Use code references or the lifecycle page before concluding a flag is unused.
Why do Prometheus counters drop after a deploy?
Counters live in each process and reset on restart. Use rate() and increase(), which handle resets.
Can viewers see Errors and Metrics?
Yes. The pages need metrics.read, which every built-in role has.

What to check#

  • Hit the Prometheus endpoint with and without the token. Only the first should succeed.
  • Metrics moves when you evaluate a flag through the server endpoint.
  • A test alert with a low threshold fires, and the Alerts firing count on the Overview page changes.
  • The X-Request-Id you send comes back on the response and appears in the log line for that call.

If something goes wrong#

Prometheus gets 404

Why: METRICS_TOKEN is not set on the server, which turns the endpoint off.

Fix: Set a long random token, restart, and send it as a bearer credential.

Prometheus gets 401

Why: The token does not match, or the scrape uses basic auth instead of a bearer credential.

Fix: Use authorization.credentials as shown and check for trailing spaces.

Metrics and Errors are empty

Why: Nobody has called the server endpoint with a key in that range, the environment filter excludes the traffic, or METRICS_ENABLED=false.

Fix: Widen the range, choose All environments and check the server setting.

Numbers differ between instances

Why: Prometheus values are per process.

Fix: Scrape every instance and sum, instead of scraping through the load balancer.

flagswift_buffered_counters keeps growing

Why: Counts cannot be written to the database fast enough, or the database is unreachable.

Fix: Check /health/ready and the database. Counts are held in memory in the meantime and retried.