Zum Inhalt springen
Produkte

// Deployment

Deploying Fleeger

How to stand up the Fleeger hub — the control plane + admin UI — from container images with Docker Compose. For how those images are built/published see Container images; for running from source in development, see the just recipes (just dev-up, just dev-help).

What runs

The Compose stack in deploy/compose/ brings up:

  • Dependencies — SurrealDB (state), NATS/JetStream (hub↔spoke messaging), VictoriaMetrics (metrics), and Loki + Promtail (hub-service log aggregation). Loki stays internal-only and is reached through the gateway's authenticated /api/logs proxy; Promtail tails the containers' stdout/stderr.
  • Services — auth-svc, inventory-svc, gateway-svc, ingestion-svc, alerting-svc, control-svc (each first runs its DB migrations via a one-shot init). The gateway additionally proxies service logs (/api/logs), live agent NATS connections (/api/agents/connz) for the Services page, and control actions (/api/control/*, gated on the control.* permission).
  • control-svc is the write path: it dispatches control actions to a location's spoke-agent over NATS and journals every action in the hub-wide audit log inventory-svc keeps (db=audit), so it reaches inventory-svc ([inventory] url). It gets a dedicated least-privilege NATS user — the sole publisher of commands.>.
  • auth-svc journals every user and role change in that audit log too ([inventory] url): IAM changes answer 503 while inventory-svc cannot take their row; sign-in never depends on it.
  • alerting-svc journals every alert rule, notification channel and mute change there as well (its existing [inventory] url): those writes answer 503 while inventory-svc cannot take their row; evaluation and delivery never depend on it.
  • UI — the frontend static SPA, behind an edge nginx that serves it and proxies /api to the gateway. The whole thing is reached at http://localhost:8080.

Agents (spoke-agent) run at your locations, not in this stack — you onboard them from the UI (see Onboarding a new location).

Prerequisites

  • Docker + Docker Compose.
  • The service/UI images — either build them locally (docker compose build, needs the repo checked out) or pull them from a registry (set REGISTRY/TAG).

Steps

All commands run from deploy/compose/.

1. Generate the NATS auth material

Fleeger uses NATS decentralized (operator) auth. Generate the operator, SPOKES account, and hub/SYS credentials once:

../nats/gen-auth.sh

This writes gitignored material under deploy/nats/auth/ (regenerating rotates all NATS credentials — every agent must re-onboard). See Agent credentials & revocation for what it creates and why.

2. Configure

cp .env.example .env

Edit .env and set FLEEGER_CREDENTIAL_KEY — a 32-byte base64 key that encrypts stored secrets (infra credentials, alert channel configs):

head -c 32 /dev/urandom | base64

Back this key up. Losing it makes every encrypted credential unreadable, even with a perfect database restore. It is required: while it is unset or empty, every docker compose command refuses to run and names it, rather than inventory-svc and alerting-svc restarting in a loop. Set it in .env: a key in the [credential] key field of a service's TOML does not satisfy compose (that field is a fallback for running a service locally, not for deployments).

Four more secrets are required — the stack refuses to start without them:

  • SURREAL_PASS — the SurrealDB root password (head -c 24 /dev/urandom | base64). The services and the SurrealDB server read it from the environment; it is never written into a config file or onto a command line. Anyone who can run docker inspect or read /proc/<pid>/environ (root and the docker group) can still read it; other local users cannot. SurrealDB uses it only when the datastore is first created; changing it later does not change the password (see rotating it, step 2). While the two match, docker compose exec surrealdb /surreal sql --endpoint http://localhost:8000 logs in as root without --user/--pass, because the container has both in its environment.
  • FLEEGER_INTERNAL_TOKEN — the shared secret the gateway uses to authenticate to the backend services (head -c 24 /dev/urandom | base64).
  • FLEEGER_VM_DELETE_KEY and FLEEGER_VM_SNAPSHOT_KEY — VictoriaMetrics' admin-API keys, two different openssl rand -hex 32, hex only. The delete key goes to VictoriaMetrics and ingestion-svc, which alone deletes series; without it anyone who reaches VictoriaMetrics deletes every series with one request. The snapshot key guards the snapshot and other maintenance endpoints; scripts/backup.sh needs it in docker-exec mode (see Backup & disaster recovery). ingestion-svc checks at startup that VictoriaMetrics refuses a delete without the key and accepts its own, and logs an error if not. No surrounding whitespace or newline in either key: VictoriaMetrics does not trim them, so a padded key never matches.

Required for backups, though the stack starts without them, and read only by the --profile backup runner: FLEEGER_BACKUP_SIGNING_KEY, the Ed25519 key it signs each backup set with, and FLEEGER_BACKUP_VERIFY_KEYS, the public keys scripts/restore.sh accepts sets from. Without the signing key each set is written unsigned (with an ERROR; older sets are not pruned), restored only with --allow-unsigned, and no backup-success metric is pushed, so the backup dead-man alert fires until the key is set. .env.example has the commands; keep a copy of the public key off the host (see Backup & disaster recovery).

If pulling images, also set REGISTRY/TAG.

3. Bring the stack up

docker compose build      # skip if pulling published images
docker compose up -d

Migrations run automatically (the *-migrate init containers) before each service starts. Each run also checks the schema against the migrations and defines again what differs: it replays every migration of the service into a scratch database, which it creates and removes again, so a start takes a few seconds more than the migrations alone. A --no-deps recreate (below) skips the one-shot, and with it this check.

Upgrading a single service: redeploy it without --no-deps (docker compose up -d <svc>), so its *-migrate one-shot runs first. Recreating a schema-owning service with --no-deps skips migrations, and if the new image adds one the service refuses to boot with Database is not up to date. To recover, run the one-shot manually: docker compose run --rm <svc>-migrate. For inventory-svc, stop it first (docker compose stop inventory-svc, then docker compose up -d inventory-svc): up -d runs inventory-migrate while the old container still runs, and a migration that builds an index (0040, the unique discovery key) can miss a row the old container writes meanwhile.

4. Load the SPOKES account into the NATS resolver

The NATS account resolver starts empty; publish the SPOKES account to it (the services will auth once it's loaded — they retry until then):

../nats/push-accounts.sh

5. Create the first admin

docker compose exec auth-svc /usr/local/bin/service \
  user add -u admin -e admin@example.com -r admin

It is journalled in the audit log as user.create before the user is created, so run it once docker compose up -d has run the inventory-migrate one-shot (inventory-svc db migrate, which migrates the audit database); before that it refuses and creates nothing.

It prompts for the password without echoing it. Leave -p out: a password given on the command line is readable by every local user with ps while the command runs.

6. Open the UI

Go to http://localhost:8080 and log in as admin. From there: add a location, onboard an agent to it, and metrics start flowing.

Operating

  • Config lives in deploy/compose/config/<service>.toml (mounted at /etc/fleeger/<service>.toml). URLs point at the Compose service names.
  • Backups — see Backup & disaster recovery (scripts/backup.sh dumps SurrealDB + VictoriaMetrics).
  • Logs — docker compose logs -f <service>.
  • Restarts — every long-running service, SurrealDB, NATS and VictoriaMetrics included, runs with restart: unless-stopped, so after a Docker or host restart the stack comes back without a docker compose up -d. A service you stopped with docker compose stop stays stopped. The *-migrate and backup one-shots are restart: "no". render-charts checks both.
  • Database reconnects: every service checks its SurrealDB connection every 5 s. After two checks in a row get no answer within 5 s, it opens a new connection, logging "Database connection presumed dead" and then "Database connection replaced". This covers a connection that died without being closed, as when SurrealDB moves to a new IP, so no service needs a restart after SurrealDB restarts. A call already waiting on the dead connection is not cut short. It ends at its caller's deadline: the gateway's timeout_secs, or 60 s for an agent's inventory request.
  • Gateway time budgets — an optional [http] section in gateway-svc.toml bounds every call the gateway makes to the services, VictoriaMetrics, Loki and the NATS monitor: timeout_secs (default 60, whole request), connect_timeout_secs (default 10) and backup_timeout_secs (default 3600, for a backup download, as the edge allows, including relaying the archive to the browser; a restore no longer goes through the gateway). Each must be at least 1, or the gateway refuses to start. Keep timeout_secs above control-svc's command_timeout_secs (default 20). The bundled Loki gives up on a log query after 30s (query_timeout) and answers 504 "Request timed out…", well before this budget would cut the request; a Loki you run yourself should also time queries out below timeout_secs.
  • Self-monitoring (the "Hub services" dashboard panel, Loki logs) is wired in the dev docker-compose.yml; it's omitted from this minimal hub stack.

Behind a reverse proxy

The edge keys its /api/auth/login rate limit on the client's address and hands that address to the gateway for the audit log. Out of the box the stack is reached directly (ports: 8080:80), the connection's peer is the client, and the edge trusts no forwarded header.

Put a TLS terminator or reverse proxy in front (Caddy, Traefik, nginx on the host, a cloud LB, cloudflared for a Cloudflare Tunnel, tailscale serve / Funnel) and every request arrives from that one hop. Name it in deploy/compose/edge.nginx.conf: uncomment the set_real_ip_from / real_ip_header / real_ip_recursive block at server level and list only the proxy's address, as the edge sees it. Read it off the 503s the edge logs (the first field of each line of docker compose logs --no-log-prefix edge) instead of guessing. Trusting anything wider lets whoever can reach the edge forge an address. Where it comes from depends on how the proxy connects:

  • A proxy on the host going to 127.0.0.1:8080 (Caddy, nginx, tailscale serve, cloudflared as a host service) arrives from the compose network's gateway (e.g. 172.18.0.1). Docker's own port relay hands other connections to the published port that same address: IPv6 clients ([::]:8080) on Linux, and on Docker Desktop (macOS, Windows) every client. So trusting the gateway trusts them too, and they can forge an address straight past the proxy. Publish the edge on loopback only, in a compose.override.yaml next to compose.yaml (!override and !reset need Docker Compose 2.24.4 or newer):

    services:
      edge:
        ports: !override
          - "127.0.0.1:8080:80"
    
  • A proxy container on the compose network (Traefik, Caddy, cloudflared) arrives from its container address, which changes when it is recreated. Name the service instead, set_real_ip_from caddy;: nginx looks the name up once at start. Start the edge after it (depends_on: [caddy] on edge in the override; a name that does not resolve stops the edge from starting) and run docker compose restart edge whenever the proxy container is recreated. Then the edge needs no published port at all: set ports: !reset [] on edge in the override.

  • A proxy on another machine going to the host's LAN address arrives from that machine's address. Firewall port 8080 to that machine.

cloudflared also sends CF-Connecting-IP, which real_ip_header can name instead of X-Forwarded-For. Then:

docker compose restart edge

Check that a page loads through the proxy. If it answers 503 with "Fleeger is behind a proxy the edge does not trust", the address does not match: correct it and restart the edge again.

Until the proxy is trusted, everything through it gets a 503. A request that carries X-Forwarded-For or X-Real-IP came through a proxy; if the edge did not take the client's address from it, the proxy is not covered by set_real_ip_from (not set, or an address that misses it), and the edge answers 503 with what to set. Without that check every client would share the proxy's address: one login bucket for everybody (10 attempts a minute, which an unauthenticated attacker can exhaust to lock every operator out) and the proxy's address in the audit log. This also stops agents connecting through wss://<host>/nats behind that proxy, and auth-svc when its [nats] public_url (config/auth-svc.toml) is that URL: then revoking or offboarding an agent fails, so the agent stays admitted. Requests without either header are served as before. A 503 on login can also be the rate limit; the body says which.

Behind the edge's per-address limit, auth-svc caps the sign-ins it runs at once, since each costs an argon2 hash of about 19 MiB in SurrealDB, whether or not the name exists. A sign-in over the cap waits for a slot, then is refused with 429 and Retry-After: 1. The cap is per auth-svc instance, set in config/auth-svc.toml:

[login]
max_concurrent = 4   # sign-ins in flight at once (default 4)
max_wait_ms = 2000   # how long one waits for a slot before the 429 (default 2000)
  • A trusted proxy that does not send the header real_ip_header names gets the 503 too.
  • A proxy that sets neither header (a plain TCP forwarder, or one that sends only Forwarded or CF-Connecting-IP) is not noticed, and the edge sees only its address. Use an HTTP-level proxy, or configure the PROXY protocol with set_real_ip_from yourself.
  • A client that sends one of these headers itself, for example through a forward proxy in your office, gets the 503 when it reaches the edge directly.
  • If you keep your own copy of edge.nginx.conf, carry over the map … $fleeger_refuse block and the if ($fleeger_refuse) check; without them nothing is checked.

Upgrading

Read the notes for the release you are upgrading to under Upgrading a running Fleeger hub to 1.0.0-rc.5 first; they list every step that needs a hand. Three apply to every compose upgrade from before 1.0.0-rc1:

  • Take compose.yaml and the images from the same release. The *-migrate one-shots no longer receive FLEEGER_CREDENTIAL_KEY, but images built before that change still check for the key before migrating, so a current compose.yaml run with older images (an older TAG) makes alerting-migrate and inventory-migrate exit Invalid credential master key, and those two services never start. Pull or build the images of the release whose compose.yaml you run.
  • Bound the ingestion streams once. A hub that ran a release which created the metrics and logs JetStream streams unbounded keeps them until they are migrated, and fills the NATS account's disk with them. ingestion-svc logs the 'metrics' stream already exists with Limits retention at every start while one is left. After the upgrade, with ingestion-svc running, run
    docker compose exec ingestion-svc /usr/local/bin/service stream migrate
    It loses no message that has not been forwarded; the Helm chart reference's Ingestion streams section covers what it does and when it refuses.
  • Agent revocations are re-asserted. auth-svc checks every 10 seconds that the NATS resolver's SPOKES account carries every revoked agent credential, and pushes it again when it does not. A resolver that lost its store (the nats-data volume) therefore re-admits offboarded agents only until auth-svc's next pass, and for as long as auth-svc is down.

Upgrading an existing (pre-hardening) deployment

A stack deployed before the security hardening needs a few one-time steps beyond docker compose up -d:

  1. Add the two new required secrets to .env: SURREAL_PASS and FLEEGER_INTERNAL_TOKEN (see step 2 above).
  2. Rotate the SurrealDB root password on the existing datastore. SurrealDB sets the root user from SURREAL_USER/SURREAL_PASS only when the datastore is first created; they are then ignored, so a running DB keeps root:root. Set the new password against the live DB before restarting the services:
    docker compose exec surrealdb /surreal sql --endpoint http://localhost:8000 \
      --user root --pass root --auth-level root \
      <<< "DEFINE USER OVERWRITE root ON ROOT PASSWORD '$SURREAL_PASS' ROLES OWNER;"
  3. Regenerate the NATS auth material (../nats/gen-auth.sh) — the per-service creds replace the old single hub.creds. This rotates all credentials, so agents must re-onboard. Then ../nats/push-accounts.sh.
  4. Redeploy and re-onboard every spoke agent. Two things changed for agents: their creds were rotated (step 3), and the per-location inbox subject changed (_INBOX_<loc> → _INBOX.<loc>), which is a wire-protocol change — an old agent binary can't use new creds. Rebuild/pull the new spoke-agent image and re-onboard each location (fresh registration token, or docker compose exec -T auth-svc /usr/local/bin/service mint-creds --location <location-id> > <location-id>.creds, see minting by hand), removing its old credentials first (re-onboarding).
  5. Self-signed Redfish/iDRAC hosts: TLS verification is on by default (M7). A host with a self-signed cert (most iDRACs) needs Allow invalid/self-signed TLS certificates ticked on the resource (the checkbox on the iDRAC / Proxmox resource form — stored as allow_invalid_certs on the resource), or hardware collection fails with a cert error. This is a connection property of the endpoint, not the secret: it lived on the credential in an earlier build, where toggling it did nothing unless the secret was also re-entered.

Stored credentials keep decrypting with your existing FLEEGER_CREDENTIAL_KEY — nothing to migrate there.

Rotating the credential key

Stored secrets are sealed under FLEEGER_CREDENTIAL_KEY. Each ciphertext records which key sealed it, so you can roll the key without downtime:

  1. Generate a new key (head -c 32 /dev/urandom | base64).
  2. In .env, set FLEEGER_CREDENTIAL_KEY to the new key and FLEEGER_CREDENTIAL_KEYS_RETIRED to the old one (comma-separate multiple retired keys). Recreate the services: docker compose up -d. New writes use the new key; existing secrets still decrypt via the retired key.
  3. Re-encrypt everything under the new key, in both services that hold secrets: docker compose exec inventory-svc /usr/local/bin/service rekey (credentials) and docker compose exec alerting-svc /usr/local/bin/service rekey (notification-channel secrets). A secret edited while the rekey runs keeps the edit: the rekey only writes over the ciphertext it read, reports such rows as edited or deleted meanwhile, and re-encrypts one that was edited under the retired key. If a row keeps being edited under the retired key, it stops with an error naming it; run it again. Each run is journalled in the audit log as secrets.rekey (never a key) before it re-encrypts anything; when the audit log cannot take the row it refuses and changes nothing.
  4. Make sure every replica of both services runs with the new key (a replica still on the old one seals what it writes under the retired key). Then run rekey again in each; once a run reports 0 ... re-encrypted, drop FLEEGER_CREDENTIAL_KEYS_RETIRED and recreate. Keep a backup of the old key until the rekey is confirmed — anything not yet re-encrypted still needs it.
  5. Keep the old key, off-box, for as long as you keep any backup taken before both rekeys were confirmed. The rekey re-encrypts the live databases only. Every backup taken before it finished can still hold the old key's ciphertexts, even one taken after step 2 that is sealed under the new key. A UI .fbk from before step 2 is also sealed as a whole under the old key. Without the old key such a .fbk cannot be restored at all, and any other such backup brings back credentials and alert-channel secrets that nothing can decrypt. To restore one later, supply the old key again as FLEEGER_CREDENTIAL_KEYS_RETIRED, restore, and run both rekey commands again; see Restoring a backup from before a key rotation finished.

Production note. This Compose stack is a straightforward way to run Fleeger on one host. For Kubernetes, use either the Helm chart in Helm chart: fleeger-hub or the plain manifests in Fleeger on Kubernetes — plain manifests — see Deploying Fleeger on Kubernetes.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.