// Deployment
Deploying Fleeger
How to stand up the Fleeger hub — the control plane + admin UI — from container
images with Docker Compose. For how those images are built/published see
Container images; for running from source in
development, see the just recipes (just dev-up, just dev-help).
What runs
The Compose stack in deploy/compose/ brings up:
- Dependencies — SurrealDB (state), NATS/JetStream (hub↔spoke messaging),
VictoriaMetrics (metrics), and Loki + Promtail (hub-service log
aggregation). Loki stays internal-only and is reached through the gateway's
authenticated
/api/logsproxy; Promtail tails the containers' stdout/stderr. - Services —
auth-svc,inventory-svc,gateway-svc,ingestion-svc,alerting-svc,control-svc(each first runs its DB migrations via a one-shot init). The gateway additionally proxies service logs (/api/logs), live agent NATS connections (/api/agents/connz) for the Services page, and control actions (/api/control/*, gated on thecontrol.*permission). - control-svc is the write path: it dispatches control actions to a
location's spoke-agent over NATS and journals every action in the hub-wide
audit log inventory-svc keeps (
db=audit), so it reaches inventory-svc ([inventory] url). It gets a dedicated least-privilege NATS user — the sole publisher ofcommands.>. - auth-svc journals every user and role change in that audit log too
(
[inventory] url): IAM changes answer 503 while inventory-svc cannot take their row; sign-in never depends on it. - alerting-svc journals every alert rule, notification channel and mute
change there as well (its existing
[inventory] url): those writes answer 503 while inventory-svc cannot take their row; evaluation and delivery never depend on it. - UI — the
frontendstatic SPA, behind an edge nginx that serves it and proxies/apito the gateway. The whole thing is reached at http://localhost:8080.
Agents (spoke-agent) run at your locations, not in this stack — you onboard
them from the UI (see Onboarding a new location).
Prerequisites
- Docker + Docker Compose.
- The service/UI images — either build them locally (
docker compose build, needs the repo checked out) or pull them from a registry (setREGISTRY/TAG).
Steps
All commands run from deploy/compose/.
1. Generate the NATS auth material
Fleeger uses NATS decentralized (operator) auth. Generate the operator, SPOKES account, and hub/SYS credentials once:
../nats/gen-auth.shThis writes gitignored material under deploy/nats/auth/ (regenerating rotates
all NATS credentials — every agent must
re-onboard). See
Agent credentials & revocation for what it creates and why.
2. Configure
cp .env.example .envEdit .env and set FLEEGER_CREDENTIAL_KEY — a 32-byte base64 key that encrypts
stored secrets (infra credentials, alert channel configs):
head -c 32 /dev/urandom | base64Back this key up. Losing it makes every encrypted credential unreadable, even
with a perfect database restore. It is required: while it is unset or empty,
every docker compose command refuses to run and names it, rather than
inventory-svc and alerting-svc restarting in a loop. Set it in .env: a key in
the [credential] key field of a service's TOML does not satisfy compose (that
field is a fallback for running a service locally, not for deployments).
Four more secrets are required — the stack refuses to start without them:
SURREAL_PASS— the SurrealDB root password (head -c 24 /dev/urandom | base64). The services and the SurrealDB server read it from the environment; it is never written into a config file or onto a command line. Anyone who can rundocker inspector read/proc/<pid>/environ(root and thedockergroup) can still read it; other local users cannot. SurrealDB uses it only when the datastore is first created; changing it later does not change the password (see rotating it, step 2). While the two match,docker compose exec surrealdb /surreal sql --endpoint http://localhost:8000logs in as root without--user/--pass, because the container has both in its environment.FLEEGER_INTERNAL_TOKEN— the shared secret the gateway uses to authenticate to the backend services (head -c 24 /dev/urandom | base64).FLEEGER_VM_DELETE_KEYandFLEEGER_VM_SNAPSHOT_KEY— VictoriaMetrics' admin-API keys, two differentopenssl rand -hex 32, hex only. The delete key goes to VictoriaMetrics and ingestion-svc, which alone deletes series; without it anyone who reaches VictoriaMetrics deletes every series with one request. The snapshot key guards the snapshot and other maintenance endpoints;scripts/backup.shneeds it indocker-execmode (see Backup & disaster recovery). ingestion-svc checks at startup that VictoriaMetrics refuses a delete without the key and accepts its own, and logs an error if not. No surrounding whitespace or newline in either key: VictoriaMetrics does not trim them, so a padded key never matches.
Required for backups, though the stack starts without them, and read
only by the --profile backup runner: FLEEGER_BACKUP_SIGNING_KEY,
the Ed25519 key it signs each backup set with, and
FLEEGER_BACKUP_VERIFY_KEYS, the public keys scripts/restore.sh accepts
sets from. Without the signing key each set is written unsigned (with an
ERROR; older sets are not pruned), restored only with --allow-unsigned,
and no backup-success metric is pushed, so the backup dead-man alert fires
until the key is set. .env.example has the commands; keep a copy of
the public key off the host (see
Backup & disaster recovery).
If pulling images, also set REGISTRY/TAG.
3. Bring the stack up
docker compose build # skip if pulling published images
docker compose up -dMigrations run automatically (the *-migrate init containers) before each
service starts. Each run also checks the schema against the migrations and
defines again what differs: it replays every
migration of the service into a scratch database, which it creates and
removes again, so a start takes a few seconds more than the migrations
alone. A --no-deps recreate (below) skips the one-shot, and with it this
check.
Upgrading a single service: redeploy it without
--no-deps(docker compose up -d <svc>), so its*-migrateone-shot runs first. Recreating a schema-owning service with--no-depsskips migrations, and if the new image adds one the service refuses to boot withDatabase is not up to date. To recover, run the one-shot manually:docker compose run --rm <svc>-migrate. For inventory-svc, stop it first (docker compose stop inventory-svc, thendocker compose up -d inventory-svc):up -drunsinventory-migratewhile the old container still runs, and a migration that builds an index (0040, the unique discovery key) can miss a row the old container writes meanwhile.
4. Load the SPOKES account into the NATS resolver
The NATS account resolver starts empty; publish the SPOKES account to it (the services will auth once it's loaded — they retry until then):
../nats/push-accounts.sh5. Create the first admin
docker compose exec auth-svc /usr/local/bin/service \
user add -u admin -e admin@example.com -r adminIt is journalled in the audit log as user.create before the
user is created, so run it once docker compose up -d has run the
inventory-migrate one-shot (inventory-svc db migrate, which migrates the
audit database); before that it refuses and creates nothing.
It prompts for the password without echoing it. Leave -p out: a password
given on the command line is readable by every local user with ps while the
command runs.
6. Open the UI
Go to http://localhost:8080 and log in as admin. From there: add a
location, onboard an agent to it, and metrics start flowing.
Operating
- Config lives in
deploy/compose/config/<service>.toml(mounted at/etc/fleeger/<service>.toml). URLs point at the Compose service names. - Backups — see Backup & disaster recovery
(
scripts/backup.shdumps SurrealDB + VictoriaMetrics). - Logs —
docker compose logs -f <service>. - Restarts — every long-running service, SurrealDB, NATS and
VictoriaMetrics included, runs with
restart: unless-stopped, so after a Docker or host restart the stack comes back without adocker compose up -d. A service you stopped withdocker compose stopstays stopped. The*-migrateandbackupone-shots arerestart: "no". render-charts checks both. - Database reconnects: every service checks its SurrealDB connection
every 5 s. After two checks in a row get no answer within 5 s, it opens a
new connection, logging "Database connection presumed dead" and then
"Database connection replaced". This covers a connection that died without
being closed, as when SurrealDB moves to a new IP, so no service needs a
restart after SurrealDB restarts. A call already waiting on the dead
connection is not cut short. It ends at its caller's deadline: the
gateway's
timeout_secs, or 60 s for an agent's inventory request. - Gateway time budgets — an optional
[http]section ingateway-svc.tomlbounds every call the gateway makes to the services, VictoriaMetrics, Loki and the NATS monitor:timeout_secs(default60, whole request),connect_timeout_secs(default10) andbackup_timeout_secs(default3600, for a backup download, as the edge allows, including relaying the archive to the browser; a restore no longer goes through the gateway). Each must be at least1, or the gateway refuses to start. Keeptimeout_secsabove control-svc'scommand_timeout_secs(default20). The bundled Loki gives up on a log query after 30s (query_timeout) and answers 504 "Request timed out…", well before this budget would cut the request; a Loki you run yourself should also time queries out belowtimeout_secs. - Self-monitoring (the "Hub services" dashboard panel, Loki logs) is wired in
the dev
docker-compose.yml; it's omitted from this minimal hub stack.
Behind a reverse proxy
The edge keys its /api/auth/login rate limit on the client's address and hands
that address to the gateway for the audit log. Out of the box the stack is
reached directly (ports: 8080:80), the connection's peer is the client,
and the edge trusts no forwarded header.
Put a TLS terminator or reverse proxy in front (Caddy, Traefik, nginx on the
host, a cloud LB, cloudflared for a Cloudflare Tunnel, tailscale serve /
Funnel) and every request arrives from that one hop. Name it in
deploy/compose/edge.nginx.conf:
uncomment the set_real_ip_from / real_ip_header / real_ip_recursive block
at server level and list only the proxy's address, as the edge sees it.
Read it off the 503s the edge logs (the first field of each line of
docker compose logs --no-log-prefix edge) instead of guessing. Trusting
anything wider lets whoever can reach the edge forge an address. Where it comes from depends on how the proxy
connects:
A proxy on the host going to
127.0.0.1:8080(Caddy, nginx,tailscale serve,cloudflaredas a host service) arrives from the compose network's gateway (e.g.172.18.0.1). Docker's own port relay hands other connections to the published port that same address: IPv6 clients ([::]:8080) on Linux, and on Docker Desktop (macOS, Windows) every client. So trusting the gateway trusts them too, and they can forge an address straight past the proxy. Publish the edge on loopback only, in acompose.override.yamlnext tocompose.yaml(!overrideand!resetneed Docker Compose 2.24.4 or newer):services: edge: ports: !override - "127.0.0.1:8080:80"A proxy container on the compose network (Traefik, Caddy,
cloudflared) arrives from its container address, which changes when it is recreated. Name the service instead,set_real_ip_from caddy;: nginx looks the name up once at start. Start the edge after it (depends_on: [caddy]onedgein the override; a name that does not resolve stops the edge from starting) and rundocker compose restart edgewhenever the proxy container is recreated. Then the edge needs no published port at all: setports: !reset []onedgein the override.A proxy on another machine going to the host's LAN address arrives from that machine's address. Firewall port 8080 to that machine.
cloudflared also sends CF-Connecting-IP, which real_ip_header can name
instead of X-Forwarded-For. Then:
docker compose restart edgeCheck that a page loads through the proxy. If it answers 503 with "Fleeger is behind a proxy the edge does not trust", the address does not match: correct it and restart the edge again.
Until the proxy is trusted, everything through it gets a 503. A
request that carries X-Forwarded-For or X-Real-IP came through a proxy; if
the edge did not take the client's address from it, the proxy is not covered by
set_real_ip_from (not set, or an address that misses it), and the edge answers
503 with what to set. Without that check every client would share the proxy's
address: one login bucket for everybody (10 attempts a minute, which an
unauthenticated attacker can exhaust to lock every operator out) and the
proxy's address in the audit log. This also stops agents connecting through
wss://<host>/nats behind that proxy, and auth-svc when its [nats] public_url
(config/auth-svc.toml) is that URL: then revoking or offboarding an agent
fails, so the agent stays admitted. Requests without either header are served
as before. A 503 on login can also be the rate limit; the body says which.
Behind the edge's per-address limit, auth-svc caps the sign-ins it runs at once, since each costs an argon2 hash of about 19 MiB in SurrealDB, whether or
not the name exists. A sign-in over the cap waits for a slot, then is refused
with 429 and Retry-After: 1. The cap is per auth-svc instance, set in
config/auth-svc.toml:
[login]
max_concurrent = 4 # sign-ins in flight at once (default 4)
max_wait_ms = 2000 # how long one waits for a slot before the 429 (default 2000)
- A trusted proxy that does not send the header
real_ip_headernames gets the 503 too. - A proxy that sets neither header (a plain TCP forwarder, or one that sends
only
ForwardedorCF-Connecting-IP) is not noticed, and the edge sees only its address. Use an HTTP-level proxy, or configure the PROXY protocol withset_real_ip_fromyourself. - A client that sends one of these headers itself, for example through a forward proxy in your office, gets the 503 when it reaches the edge directly.
- If you keep your own copy of
edge.nginx.conf, carry over themap … $fleeger_refuseblock and theif ($fleeger_refuse)check; without them nothing is checked.
Upgrading
Read the notes for the release you are upgrading to under Upgrading a running Fleeger hub to 1.0.0-rc.5 first; they list every step that needs a hand. Three apply to every compose upgrade from before 1.0.0-rc1:
- Take
compose.yamland the images from the same release. The*-migrateone-shots no longer receiveFLEEGER_CREDENTIAL_KEY, but images built before that change still check for the key before migrating, so a currentcompose.yamlrun with older images (an olderTAG) makesalerting-migrateandinventory-migrateexitInvalid credential master key, and those two services never start. Pull or build the images of the release whosecompose.yamlyou run. - Bound the ingestion streams once. A hub that ran a release which created
the
metricsandlogsJetStream streams unbounded keeps them until they are migrated, and fills the NATS account's disk with them.ingestion-svclogsthe 'metrics' stream already exists with Limits retentionat every start while one is left. After the upgrade, withingestion-svcrunning, runIt loses no message that has not been forwarded; the Helm chart reference's Ingestion streams section covers what it does and when it refuses.docker compose exec ingestion-svc /usr/local/bin/service stream migrate - Agent revocations are re-asserted. auth-svc checks every 10 seconds that
the NATS resolver's SPOKES account carries every revoked agent credential,
and pushes it again when it does not. A resolver that lost its store
(the
nats-datavolume) therefore re-admits offboarded agents only until auth-svc's next pass, and for as long as auth-svc is down.
Upgrading an existing (pre-hardening) deployment
A stack deployed before the security hardening needs a few one-time steps beyond
docker compose up -d:
- Add the two new required secrets to
.env:SURREAL_PASSandFLEEGER_INTERNAL_TOKEN(see step 2 above). - Rotate the SurrealDB root password on the existing datastore. SurrealDB
sets the root user from
SURREAL_USER/SURREAL_PASSonly when the datastore is first created; they are then ignored, so a running DB keepsroot:root. Set the new password against the live DB before restarting the services:docker compose exec surrealdb /surreal sql --endpoint http://localhost:8000 \ --user root --pass root --auth-level root \ <<< "DEFINE USER OVERWRITE root ON ROOT PASSWORD '$SURREAL_PASS' ROLES OWNER;" - Regenerate the NATS auth material (
../nats/gen-auth.sh) — the per-service creds replace the old singlehub.creds. This rotates all credentials, so agents must re-onboard. Then../nats/push-accounts.sh. - Redeploy and re-onboard every spoke agent. Two things changed for agents:
their creds were rotated (step 3), and the per-location inbox subject changed
(
_INBOX_<loc>→_INBOX.<loc>), which is a wire-protocol change — an old agent binary can't use new creds. Rebuild/pull the newspoke-agentimage and re-onboard each location (fresh registration token, ordocker compose exec -T auth-svc /usr/local/bin/service mint-creds --location <location-id> > <location-id>.creds, see minting by hand), removing its old credentials first (re-onboarding). - Self-signed Redfish/iDRAC hosts: TLS verification is on by default (M7). A
host with a self-signed cert (most iDRACs) needs Allow invalid/self-signed
TLS certificates ticked on the resource (the checkbox on the iDRAC /
Proxmox resource form — stored as
allow_invalid_certson the resource), or hardware collection fails with a cert error. This is a connection property of the endpoint, not the secret: it lived on the credential in an earlier build, where toggling it did nothing unless the secret was also re-entered.
Stored credentials keep decrypting with your existing FLEEGER_CREDENTIAL_KEY —
nothing to migrate there.
Rotating the credential key
Stored secrets are sealed under FLEEGER_CREDENTIAL_KEY. Each ciphertext records
which key sealed it, so you can roll the key without downtime:
- Generate a new key (
head -c 32 /dev/urandom | base64). - In
.env, setFLEEGER_CREDENTIAL_KEYto the new key andFLEEGER_CREDENTIAL_KEYS_RETIREDto the old one (comma-separate multiple retired keys). Recreate the services:docker compose up -d. New writes use the new key; existing secrets still decrypt via the retired key. - Re-encrypt everything under the new key, in both services that hold secrets:
docker compose exec inventory-svc /usr/local/bin/service rekey(credentials) anddocker compose exec alerting-svc /usr/local/bin/service rekey(notification-channel secrets). A secret edited while the rekey runs keeps the edit: the rekey only writes over the ciphertext it read, reports such rows asedited or deleted meanwhile, and re-encrypts one that was edited under the retired key. If a row keeps being edited under the retired key, it stops with an error naming it; run it again. Each run is journalled in the audit log assecrets.rekey(never a key) before it re-encrypts anything; when the audit log cannot take the row it refuses and changes nothing. - Make sure every replica of both services runs with the new key (a replica
still on the old one seals what it writes under the retired key). Then run
rekeyagain in each; once a run reports0 ... re-encrypted, dropFLEEGER_CREDENTIAL_KEYS_RETIREDand recreate. Keep a backup of the old key until the rekey is confirmed — anything not yet re-encrypted still needs it. - Keep the old key, off-box, for as long as you keep any backup taken before
both rekeys were confirmed. The rekey re-encrypts the live databases only.
Every backup taken before it finished can still hold the old key's
ciphertexts, even one taken after step 2 that is sealed under the new key. A
UI
.fbkfrom before step 2 is also sealed as a whole under the old key. Without the old key such a.fbkcannot be restored at all, and any other such backup brings back credentials and alert-channel secrets that nothing can decrypt. To restore one later, supply the old key again asFLEEGER_CREDENTIAL_KEYS_RETIRED, restore, and run bothrekeycommands again; see Restoring a backup from before a key rotation finished.
Production note. This Compose stack is a straightforward way to run Fleeger on one host. For Kubernetes, use either the Helm chart in Helm chart: fleeger-hub or the plain manifests in Fleeger on Kubernetes — plain manifests — see Deploying Fleeger on Kubernetes.