Skip to content
FlagSwiftDocs

Run FlagSwift in production

Set up the database, secrets, health checks and limits so a self-hosted server stays up and behaves the same on every instance.

Time
About 12 min
Who can do it
Whoever operates the server
Where
Environment variables, your database and your proxy
You finish with
A server you can restart, scale and watch

What and why#

A FlagSwift server is a single Next.js application plus a database. Instances hold no durable state of their own, so you can run one or several behind a load balancer. Everything that must survive lives in the database: flags, keys, roles, the audit log, metrics, releases and analytics. This guide covers what to set so that is safe to operate.

What is shared and what is per instance#

State and where it lives
StateDescription
Flags, keys, members, audit log, releases, metricsIn the database. Shared by every instance.
Configuration and key cachesIn each process, for 30 seconds. After a change, another instance may serve the previous value until its cache expires.
Role lookupsCached per process for 10 seconds. A role change is immediate on the instance that handled it and takes up to this long elsewhere.
StreamsA change reaches the streams open on the instance that handled it. SDKs keep a slow safety-net poll to catch the rest.
Prometheus counters and buffered countsPer process. Scrape every instance.
Rate limitsIn the database so instances share them; per instance only while the database cannot be reached.

What runs in the background#

Each instance runs a release scheduler (checks for due steps every 15 seconds) and an alert evaluator (every minute) unless SCHEDULER_ENABLED=false. Both are safe on several instances: a due step or a state change is claimed with a conditional update, so it happens once. Metrics and analytics are buffered in memory and written about every 10 seconds. A crash loses at most that buffer.

Before you start#

  • A PostgreSQL or MongoDB database you can back up, and a host for the app (Node version as in .nvmrc).
  • HTTPS in front of the app. Sign-in cookies and bearer keys must never travel in clear text.
  • A secrets manager for JWT_SECRET, API_KEY_ENCRYPTION_KEY and METRICS_TOKEN.
  • A reverse proxy or load balancer that sets X-Forwarded-For and does not buffer text/event-stream.

Walk through it#

  1. Set the required variables

    Set DB_TYPE to postgres or mongodb and give the matching connection string (DATABASE_URL and DIRECT_URL for PostgreSQL, or MONGODB_URI). Set JWT_SECRET to a long random value. The server refuses to sign anyone in without it. Set APP_URL to the public address so invitation and email links are correct. On start the server logs env.missing for any of these that is not set.

    bash
    # One way to make a strong secret
    openssl rand -base64 48

    JWT_SECRET is not freely changeable

    It signs console sessions and keys the outcome-analytics hashes. Changing it signs everyone out and starts analytics over. Pick it once and keep it.
  2. Run migrations

    On PostgreSQL, apply every pending migration, first against a copy of the database, then against production. MongoDB has no migrations, but the app does not build its indexes: run npm run db:init-mongo once on a new database and again after an upgrade that changes an index. To move an existing install to the other database, see Switch from PostgreSQL to MongoDB or Switch from MongoDB to PostgreSQL.

    bash
    npx prisma generate
    npx prisma migrate deploy
    npm run build
    npm start

    Migrations should use a direct connection, not a transaction pooler. Set DIRECT_URL to it and keep DATABASE_URL for the pooled connection the app uses.

  3. Size the connection pool

    The readiness checklist suggests adding connection_limit=10, pool_timeout=20 and connect_timeout=15 as query parameters on DATABASE_URL. Tune them to your database, remembering that each instance gets its own pool. Keep the app in the same region as the database, because distance shows up as slow queries (hundreds of milliseconds per query across regions). Watch the logs for P1017 and “Timed out fetching a new connection”. Scheduler ticks retry connection drops; ordinary request queries do not. Queries slower than 500 milliseconds are logged with their duration.

    bash
    DATABASE_URL="postgresql://user:pass@db.example.com:6543/flags?pgbouncer=true&connection_limit=10&pool_timeout=20&connect_timeout=15"
    DIRECT_URL="postgresql://user:pass@db.example.com:5432/flags"
  4. Wire up health checks

    Point your load balancer at /health/ready, which answers 200 when the database responds within 2 seconds and 503 otherwise. Point your process manager at /health/live, which does not touch the database, so a database outage does not restart the process. /health returns both with the uptime. They report only ok or down; the reason goes to the logs as health.database_down.

    You should see. With the database up, /health/ready returns { "status": "ok", "checks": { "database": "ok" } }.
    What a load balancer sees. With the database down the same path answers 503 with status unavailable and database down.
  5. Decide which instances run schedulers

    Each instance runs the release and alert schedulers unless SCHEDULER_ENABLED=false. At least one instance must keep them on, or scheduled releases never run and alert rules never update. Turn them off on instances that should only serve requests.

  6. Check rate limits behind your proxy

    Make the proxy set X-Forwarded-For, because rate limiting uses it to tell callers apart; without it every caller can look like the proxy. The limits are 600 API requests per key per minute and 30 bad-key attempts per minute per address. They are fixed and counted in the database so instances share them; the per-key limiter runs on every SDK request and adds database round trips, so watch database load at high volume. RATE_LIMIT_ENABLED=false turns limits off entirely. Failed console sign-ins are limited to 10 per 15 minutes for one address and email and five times that for one address overall. Password reset is limited too: 3 requests an hour for one email from one address, 20 an hour from one address, and 10 reset-link submissions per 15 minutes. The full list is in Self-hosting. A key call over the limit gets 429 with Retry-After.

  7. Turn on monitoring

    Set METRICS_TOKEN to enable /api/prometheus and scrape every instance, since counters are per process. Set LOG_LEVEL as you need. Create alert rules for Production. See Monitor your flags.

  8. Confirm the settings in the console

    Open Settings and expand System. Its badge reads Looks good or how many things there are to check. Open it for any notices, a live check of the instance health and a read-only list of how this deployment is set up, shown only to people with organization.update. Values come from environment variables and change only when you redeploy, never from this page.

    The System panel, expanded. Each row names the variable that controls it.
    You should see. The Background scheduler row reads On on at least one instance, and Prometheus endpoint reads On once METRICS_TOKEN is set. Any row that disagrees with what you set means the process has not been restarted with the new environment.
  9. Back up and plan upgrades

    Back up the database on a schedule and test a restore. For an upgrade, take a backup, apply new migrations against a copy first, then production, then roll the instances one at a time while /health/ready gates traffic.

    bash
    # PostgreSQL example
    pg_dump "$DIRECT_URL" --format=custom --file=flagswift-$(date +%F).dump

Settings you may want#

Optional settings
VariableDescription
API_KEY_ENCRYPTION_KEY16 or more characters. Lets administrators copy new keys again. Changing it makes stored keys uncopyable; they keep working.
STREAM_ENABLEDDefault on. Behind a proxy, do not buffer text/event-stream responses. SDKs fall back to polling when it is off. Stream limits are fixed: 100 per key, 5,000 per instance, 5 minutes each.
METRICS_ENABLEDEvaluation and API counts. Default on, kept 30 days.
ANALYTICS_ENABLEDDefault off. Rows are kept 90 days.
ALLOW_SELF_APPROVALDefault false. Leave it that way if approval is meant to be a second pair of eyes. Requests nobody reviews expire after 72 hours.
ALLOWED_ORIGINSExtra origins allowed to send changing requests to the console API. The app's own address is always allowed.
CSP_CONNECT_SRC, CSP_IMG_SRCExtra hosts the console may connect to or load images from. The content security policy is always enforced and HSTS is always sent in production.
RESEND_API_KEY, MAIL_FROMRequired: one email provider. New accounts cannot sign in until they open the verification email, so it must deliver (use a verified sending domain and set APP_URL). Both are needed for Resend (or SMTP_HOST and SMTP_FROM for your own SMTP server; Resend is used when its key is set). Without one the server shows a setup screen and does not start.
ALERT_WEBHOOK_URLOptional chat or incident webhook for alerts.

Things to know with more than one instance

With several instances, each one checks the database for changes every 5 seconds by default (DELIVERY_SYNC_INTERVAL_SECONDS; 0 turns it off) and drops what changed, so a flag change or a revoked key reaches the others in seconds. If a check is missed, the 30 second cache lifetime still applies, and role changes can take up to 10 seconds. Lifetimes, limits and retention are fixed; see Self-hosting for the list. If the database cannot be reached, rate limits fall back to counting per instance.

Not yet confirmed on a real deployment

The readiness checklist in the repository still lists several items open: long idle runs, a load test and a handful of failing tests. No load test has been run, and audit writes are transactional only for flags, roles and API keys. Treat the first weeks as a trial.

Verify it#

Run these against your public address. Replace the host and token.

bash
# Readiness and liveness
curl -s -o /dev/null -w "%{http_code}\n" https://flags.example.com/health/ready   # 200
curl -s https://flags.example.com/health/live

# An SDK call with a real key: flag data plus rate-limit headers
curl -si "https://flags.example.com/api/v1/evaluation/server?env=production" \
  -H "Authorization: Bearer $SERVER_KEY" | grep -iE "^HTTP|x-ratelimit"

# Metrics: 401 without the token, 200 with it
curl -s -o /dev/null -w "%{http_code}\n" https://flags.example.com/api/prometheus
curl -s -o /dev/null -w "%{http_code}\n" https://flags.example.com/api/prometheus -H "Authorization: Bearer $METRICS_TOKEN"
  • Sign in, create a flag, toggle it and find each step in the audit log.
  • Export the configuration and import the same file as a dry run. It should report nothing to change.
  • Stop the database and confirm /health/ready returns 503 while /health/live stays 200.
  • A restore of your latest backup works. Flags, keys, the audit log and metrics all live in the database.
  • Leave the server running 30 to 60 minutes, then check the logs for P1017 after idle.

Good to know#

How many instances do I need?
One works. Run at least two for availability. Each takes its own database connections, so multiply the pool size by the number of instances and check your database's connection limit.
Why does a flag change take up to 30 seconds on one instance?
Each instance caches the configuration for 30 seconds. The instance that handled the change updates at once. Turn on streaming in the SDK if you need faster convergence.
Can I turn rate limiting off?
With RATE_LIMIT_ENABLED=false, but then bad-key guessing and sign-in attempts are not slowed. Watch database load if the per-key limiter's round trips become a problem.
Where do invitation links go without email?
They appear only in the console, for the person who invited, to copy and send themselves.

If something goes wrong#

Nobody can sign in, and logs mention JWT_SECRET

Why: JWT_SECRET is not set in the server's environment.

Fix: Set it, restart, and keep the same value on every instance.

/health/ready returns 503

Why: The database did not answer within 2 seconds, or the connection string is wrong.

Fix: Check DATABASE_URL, network access and the logs for health.database_down.

Errors with P1017 or Timed out fetching a new connection

Why: The pool is exhausted or connections were dropped by a pooler or idle timeout.

Fix: Add the pool parameters to DATABASE_URL, put the app near the database, and check the total connections across all instances.

Rate limits seem to block everyone, or none

Why: The proxy is not setting X-Forwarded-For, so callers cannot be told apart.

Fix: Configure the proxy to set it, then confirm X-RateLimit headers on a key call.

Scheduled releases never run, alerts never update

Why: SCHEDULER_ENABLED=false on every instance.

Fix: Leave it on for at least one instance.

migrate deploy hangs or fails behind a pooler

Why: Migrations are running over a transaction pooler.

Fix: Set DIRECT_URL to the direct connection.