Run FlagSwift in production
Set up the database, secrets, health checks and limits so a self-hosted server stays up and behaves the same on every instance.
- Time
- About 12 min
- Who can do it
- Whoever operates the server
- Where
- Environment variables, your database and your proxy
- You finish with
- A server you can restart, scale and watch
What and why#
A FlagSwift server is a single Next.js application plus a database. Instances hold no durable state of their own, so you can run one or several behind a load balancer. Everything that must survive lives in the database: flags, keys, roles, the audit log, metrics, releases and analytics. This guide covers what to set so that is safe to operate.
What is shared and what is per instance#
| State | Description |
|---|---|
| Flags, keys, members, audit log, releases, metrics | In the database. Shared by every instance. |
| Configuration and key caches | In each process, for 30 seconds. After a change, another instance may serve the previous value until its cache expires. |
| Role lookups | Cached per process for 10 seconds. A role change is immediate on the instance that handled it and takes up to this long elsewhere. |
| Streams | A change reaches the streams open on the instance that handled it. SDKs keep a slow safety-net poll to catch the rest. |
| Prometheus counters and buffered counts | Per process. Scrape every instance. |
| Rate limits | In the database so instances share them; per instance only while the database cannot be reached. |
What runs in the background#
Each instance runs a release scheduler (checks for due steps every 15 seconds) and an alert evaluator (every minute) unless SCHEDULER_ENABLED=false. Both are safe on several instances: a due step or a state change is claimed with a conditional update, so it happens once. Metrics and analytics are buffered in memory and written about every 10 seconds. A crash loses at most that buffer.
Before you start#
- A PostgreSQL or MongoDB database you can back up, and a host for the app (Node version as in
.nvmrc). - HTTPS in front of the app. Sign-in cookies and bearer keys must never travel in clear text.
- A secrets manager for
JWT_SECRET,API_KEY_ENCRYPTION_KEYandMETRICS_TOKEN. - A reverse proxy or load balancer that sets
X-Forwarded-Forand does not buffertext/event-stream.
Walk through it#
Set the required variables
Set
DB_TYPEtopostgresormongodband give the matching connection string (DATABASE_URLandDIRECT_URLfor PostgreSQL, orMONGODB_URI). SetJWT_SECRETto a long random value. The server refuses to sign anyone in without it. SetAPP_URLto the public address so invitation and email links are correct. On start the server logsenv.missingfor any of these that is not set.bash # One way to make a strong secret openssl rand -base64 48JWT_SECRET is not freely changeable
It signs console sessions and keys the outcome-analytics hashes. Changing it signs everyone out and starts analytics over. Pick it once and keep it.Run migrations
On PostgreSQL, apply every pending migration, first against a copy of the database, then against production. MongoDB has no migrations, but the app does not build its indexes: run
npm run db:init-mongoonce on a new database and again after an upgrade that changes an index. To move an existing install to the other database, see Switch from PostgreSQL to MongoDB or Switch from MongoDB to PostgreSQL.bash npx prisma generate npx prisma migrate deploy npm run build npm startMigrations should use a direct connection, not a transaction pooler. Set
DIRECT_URLto it and keepDATABASE_URLfor the pooled connection the app uses.Size the connection pool
The readiness checklist suggests adding
connection_limit=10,pool_timeout=20andconnect_timeout=15as query parameters onDATABASE_URL. Tune them to your database, remembering that each instance gets its own pool. Keep the app in the same region as the database, because distance shows up as slow queries (hundreds of milliseconds per query across regions). Watch the logs forP1017and “Timed out fetching a new connection”. Scheduler ticks retry connection drops; ordinary request queries do not. Queries slower than 500 milliseconds are logged with their duration.bash DATABASE_URL="postgresql://user:pass@db.example.com:6543/flags?pgbouncer=true&connection_limit=10&pool_timeout=20&connect_timeout=15" DIRECT_URL="postgresql://user:pass@db.example.com:5432/flags"Wire up health checks
Point your load balancer at
/health/ready, which answers 200 when the database responds within 2 seconds and 503 otherwise. Point your process manager at/health/live, which does not touch the database, so a database outage does not restart the process./healthreturns both with the uptime. They report only ok or down; the reason goes to the logs ashealth.database_down.You should see. With the database up,/health/readyreturns{ "status": "ok", "checks": { "database": "ok" } }.What a load balancer sees. With the database down the same path answers 503 with status unavailable and database down. Decide which instances run schedulers
Each instance runs the release and alert schedulers unless
SCHEDULER_ENABLED=false. At least one instance must keep them on, or scheduled releases never run and alert rules never update. Turn them off on instances that should only serve requests.Check rate limits behind your proxy
Make the proxy set
X-Forwarded-For, because rate limiting uses it to tell callers apart; without it every caller can look like the proxy. The limits are 600 API requests per key per minute and 30 bad-key attempts per minute per address. They are fixed and counted in the database so instances share them; the per-key limiter runs on every SDK request and adds database round trips, so watch database load at high volume.RATE_LIMIT_ENABLED=falseturns limits off entirely. Failed console sign-ins are limited to 10 per 15 minutes for one address and email and five times that for one address overall. Password reset is limited too: 3 requests an hour for one email from one address, 20 an hour from one address, and 10 reset-link submissions per 15 minutes. The full list is in Self-hosting. A key call over the limit gets429withRetry-After.Turn on monitoring
Set
METRICS_TOKENto enable/api/prometheusand scrape every instance, since counters are per process. SetLOG_LEVELas you need. Create alert rules for Production. See Monitor your flags.Confirm the settings in the console
Open Settings and expand System. Its badge reads Looks good or how many things there are to check. Open it for any notices, a live check of the instance health and a read-only list of how this deployment is set up, shown only to people with
organization.update. Values come from environment variables and change only when you redeploy, never from this page.The System panel, expanded. Each row names the variable that controls it. You should see. The Background scheduler row reads On on at least one instance, and Prometheus endpoint reads On onceMETRICS_TOKENis set. Any row that disagrees with what you set means the process has not been restarted with the new environment.Back up and plan upgrades
Back up the database on a schedule and test a restore. For an upgrade, take a backup, apply new migrations against a copy first, then production, then roll the instances one at a time while
/health/readygates traffic.bash # PostgreSQL example pg_dump "$DIRECT_URL" --format=custom --file=flagswift-$(date +%F).dump
Settings you may want#
| Variable | Description |
|---|---|
| API_KEY_ENCRYPTION_KEY | 16 or more characters. Lets administrators copy new keys again. Changing it makes stored keys uncopyable; they keep working. |
| STREAM_ENABLED | Default on. Behind a proxy, do not buffer text/event-stream responses. SDKs fall back to polling when it is off. Stream limits are fixed: 100 per key, 5,000 per instance, 5 minutes each. |
| METRICS_ENABLED | Evaluation and API counts. Default on, kept 30 days. |
| ANALYTICS_ENABLED | Default off. Rows are kept 90 days. |
| ALLOW_SELF_APPROVAL | Default false. Leave it that way if approval is meant to be a second pair of eyes. Requests nobody reviews expire after 72 hours. |
| ALLOWED_ORIGINS | Extra origins allowed to send changing requests to the console API. The app's own address is always allowed. |
| CSP_CONNECT_SRC, CSP_IMG_SRC | Extra hosts the console may connect to or load images from. The content security policy is always enforced and HSTS is always sent in production. |
| RESEND_API_KEY, MAIL_FROM | Required: one email provider. New accounts cannot sign in until they open the verification email, so it must deliver (use a verified sending domain and set APP_URL). Both are needed for Resend (or SMTP_HOST and SMTP_FROM for your own SMTP server; Resend is used when its key is set). Without one the server shows a setup screen and does not start. |
| ALERT_WEBHOOK_URL | Optional chat or incident webhook for alerts. |
Things to know with more than one instance
DELIVERY_SYNC_INTERVAL_SECONDS; 0 turns it off) and drops what changed, so a flag change or a revoked key reaches the others in seconds. If a check is missed, the 30 second cache lifetime still applies, and role changes can take up to 10 seconds. Lifetimes, limits and retention are fixed; see Self-hosting for the list. If the database cannot be reached, rate limits fall back to counting per instance.Not yet confirmed on a real deployment
Verify it#
Run these against your public address. Replace the host and token.
# Readiness and liveness
curl -s -o /dev/null -w "%{http_code}\n" https://flags.example.com/health/ready # 200
curl -s https://flags.example.com/health/live
# An SDK call with a real key: flag data plus rate-limit headers
curl -si "https://flags.example.com/api/v1/evaluation/server?env=production" \
-H "Authorization: Bearer $SERVER_KEY" | grep -iE "^HTTP|x-ratelimit"
# Metrics: 401 without the token, 200 with it
curl -s -o /dev/null -w "%{http_code}\n" https://flags.example.com/api/prometheus
curl -s -o /dev/null -w "%{http_code}\n" https://flags.example.com/api/prometheus -H "Authorization: Bearer $METRICS_TOKEN"- Sign in, create a flag, toggle it and find each step in the audit log.
- Export the configuration and import the same file as a dry run. It should report nothing to change.
- Stop the database and confirm
/health/readyreturns 503 while/health/livestays 200. - A restore of your latest backup works. Flags, keys, the audit log and metrics all live in the database.
- Leave the server running 30 to 60 minutes, then check the logs for
P1017after idle.
Good to know#
How many instances do I need?
Why does a flag change take up to 30 seconds on one instance?
Can I turn rate limiting off?
RATE_LIMIT_ENABLED=false, but then bad-key guessing and sign-in attempts are not slowed. Watch database load if the per-key limiter's round trips become a problem.Where do invitation links go without email?
If something goes wrong#
Nobody can sign in, and logs mention JWT_SECRET
Why: JWT_SECRET is not set in the server's environment.
Fix: Set it, restart, and keep the same value on every instance.
/health/ready returns 503
Why: The database did not answer within 2 seconds, or the connection string is wrong.
Fix: Check DATABASE_URL, network access and the logs for health.database_down.
Errors with P1017 or Timed out fetching a new connection
Why: The pool is exhausted or connections were dropped by a pooler or idle timeout.
Fix: Add the pool parameters to DATABASE_URL, put the app near the database, and check the total connections across all instances.
Rate limits seem to block everyone, or none
Why: The proxy is not setting X-Forwarded-For, so callers cannot be told apart.
Fix: Configure the proxy to set it, then confirm X-RateLimit headers on a key call.
Scheduled releases never run, alerts never update
Why: SCHEDULER_ENABLED=false on every instance.
Fix: Leave it on for at least one instance.
migrate deploy hangs or fails behind a pooler
Why: Migrations are running over a transaction pooler.
Fix: Set DIRECT_URL to the direct connection.