Zum Inhalt springen
Produkte

// Betrieb

Backup & disaster recovery

This is the platform's own disaster-recovery path — how the hub's state is dumped off-box and how to bring it back. It is separate from the config checkpoints in Config revisions, checkpoints & rollback: checkpoints roll back a change you made; this restores everything after losing the hub's volume.

Read this before you need it. A backup nobody has restored from is not a backup.

There are two ways to capture a backup, sharing one format:

  • Scheduled, off-box → S3 (scripts/backup.sh) — the operational DR path. Runs by hand, from a Kubernetes CronJob, or a compose --profile backup run.
  • On-demand, encrypted download from the UI (admin Backup page) — a single self-contained .fbk file you keep somewhere safe. Encrypted with the credential master key.

Either is restored offline, with the Fleeger services stopped: a .fbk with inventory-svc backup restore, an S3 set with scripts/restore.sh (see Restoring). A restore replaces each database with the backup's, and the services bring an older backup up to date when they start again.

What is backed up

Source Contents How
SurrealDB auth users, roles, RBAC, registration tokens surreal export / HTTP /export
SurrealDB inventory Locations, Resources, credentials (ciphertext), config revisions + checkpoints, hardware inventory surreal export / HTTP /export
SurrealDB ingestion reachability / uptime state surreal export / HTTP /export
SurrealDB alerting alert rules, notification channels, mutes surreal export / HTTP /export
SurrealDB control control-svc's state (its older audit log, kept until copied) surreal export / HTTP /export
SurrealDB audit the hub-wide audit log: imported only into an empty audit database, never emptied by a restore surreal export / HTTP /export
VictoriaMetrics all metric history snapshot / native export (see below)

The scheduled path uploads everything to an S3-compatible bucket (Hetzner Object Storage in production) under a timestamped prefix, alongside a MANIFEST of every object's hash and its signature (MANIFEST.sig, see Signed backup sets). The UI download packs the same databases into one encrypted archive.

Credential master key — back it up separately. Every dump contains credential ciphertext only. It is useless without FLEEGER_CREDENTIAL_KEY, which is not in any backup (it lives in a Kubernetes Secret / env, and the .fbk archive is encrypted with it). Store that key durably and off-box — losing it means the encrypted credentials and alert-channel secrets cannot be recovered even with a perfect database restore.

After a key rotation, keep the old key too. A backup holds whatever ciphertexts the databases held when it was taken, and the rekey only re-encrypts the live databases. A backup taken after the new key became primary but before both rekeys finished is sealed under the new key yet still holds old-key secrets. Keep the old key for as long as you keep any backup taken before both rekeys were confirmed; see Restoring a backup from before a key rotation finished.

NATS auth material (deploy/nats/auth/, generated by gen-auth.sh) is likewise not in the DB. Keep it with the master key — without it, agents can't reconnect after a rebuild. See Agent credentials & revocation.

Encrypted download from the UI

The admin Backup page (visible to holders of iam.manage) is the simplest path for a whole-platform snapshot you control:

  • Download — re-enter your password, optionally tick Include VictoriaMetrics history (much larger; metrics are regenerable), and download a fleeger-backup-<timestamp>.fbk file. It contains all six service databases (and optionally metrics), framed and encrypted with the credential master key (AES-256-GCM). It is only restorable on a Fleeger instance holding that same key, as its FLEEGER_CREDENTIAL_KEY or as one of its FLEEGER_CREDENTIAL_KEYS_RETIRED.
  • Restore — not on the page any more. A restore replaces every service database and runs with the services stopped, so it is a one-shot command on the hub: inventory-svc backup restore. The page says how. POST /api/backup/restore answers 410 Gone.

The download is recorded in the audit log (backup.download: a rejected row for a wrong password, a proposed row before the archive is asked for, and how it ended). It is fail-closed: when the proposed row cannot be written, no archive is made and the gateway answers 503.

The download is password-reverified at the gateway and admin-only. inventory-svc performs the work (it already holds the master key and SurrealDB's root credentials): the databases are read over SurrealDB's HTTP /export, metrics via VictoriaMetrics' native export. The same archive can be written on the hub without the UI: inventory-svc backup export <file> (with --metrics for the history), run like the restore below. It writes a new file only (one that exists is refused), never stdout, and says on stderr how large the archive is and its SHA-256 (as sha256sum <file> prints it).

The command is recorded in the audit log too, as backup.export on platform by cli:<operating-system user>: a proposed row naming the file, the archive's format, the databases and whether the metrics are in it, before the file is created, then completed with the size and SHA-256, or failed saying how far it got and whether the unfinished file was removed (if it could not be, the row names it for you to delete). It is fail-closed: while the audit database cannot take the row (not migrated yet: run inventory-svc db migrate first), it exits non-zero and writes nothing; a scheduled export fails the same way. No key is ever in a row. The archive holds the audit database as it is when that part is read, so it holds the export's own proposed row but not its outcome (details).

A download or restore has a time budget, set by the optional [backup] section of inventory-svc.toml: timeout_secs (default 3000) for all its SurrealDB and VictoriaMetrics calls together, and connect_timeout_secs (default 10) for each connection. Each must be between 1 and 86400, or inventory-svc refuses to start. A store that stops answering fails the operation at that budget. Keep timeout_secs below the time the gateway and edge allow the download route (an hour), so inventory-svc is the one that gives up and reports; for a very large restore, raise it in the inventory-svc.toml the restore runs with. These calls never go through a proxy set in inventory-svc's environment (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, in either case): the SurrealDB calls carry its root credentials, and both stores are reached directly.

inventory-svc sends the archive as it produces it, rather than building it whole first, and in constant memory (about a MiB) whatever the size of a part. It reads one part at a time (each database dump, then the metrics), and each part as its store sends it: every MiB it has read is encrypted and sent before it reads on, so it never holds a part whole, and a backup with metrics fits the services' standard memory limit. The connection to the store stays open while its part is being sent, and closes when the download is cancelled.

Because the stores are read only as fast as the download proceeds, the [backup] timeout_secs budget covers practically the whole download, not just reading the stores: a download still running when it is spent breaks off. The gateway and the edge give the route an hour anyway; to download large backups (with metrics) over a slow link, raise timeout_secs and the gateway's [http] backup_timeout_secs together, keeping the first below the second.

A store that fails on the first part before its first MiB (or the whole part, if it is smaller) has been read is an error the page shows. One that fails later, also part way through a part, breaks the download, the same way as below, and inventory-svc logs why.

Two limits to know:

  • An archive holds at most 64 GiB (the limit of AES-GCM for one value). A larger backup, in practice only one with a very large metrics history, breaks off at that point, and inventory-svc logs encrypt backup.
  • A store answer that ends cleanly but short (the store stops early without an error) is taken as the whole part: the archive cannot tell it from a complete one. Check a restored platform as in the drill below.

The archive's format changed in 1.0.0-rc.5 (framing version 2: each part in blocks of 1 MiB, not after its whole length). Backups taken before still restore. A backup taken on this version restores only on this version or a newer one; an older Fleeger refuses it as not a Fleeger backup archive, so take a backup before upgrading if you may roll back. An archive from a newer Fleeger than the one restoring it is refused as made by a newer Fleeger; restore it there. A restore reads the archive's framing within the [backup] timeout_secs budget too, and refuses one whose framing does not hold together as corrupt backup archive, with nothing imported.

The gateway relays the downloaded archive to the browser as inventory-svc sends it, rather than holding it whole first, so a large download (one with metrics) costs the gateway no more memory than is in flight. Its [http] backup_timeout_secs budget covers relaying the archive too. A download cut off after it started, inventory-svc's connection lost or out of that budget, ends as a broken download (the connection is aborted, and the page reports an error), never as a shorter file that looks complete. A download inventory-svc refuses or fails before it starts (a store that does not answer, say) is not offered as a file: the gateway passes inventory-svc's error answer on as it is, and the page shows its message.

The page saves the download straight to disk as it arrives, through a small service worker at /backup-download/sw.js: the page sends the request itself, so the session token and the password never reach the worker, and hands the answer to it a piece at a time; the worker serves it to the browser as an ordinary download. The browser shows it in its download list, writes it no faster than the disk takes it (the page reads the archive only as fast), and marks it failed if it breaks off. Cancelling it there, with the page's Cancel button, or by closing the tab stops the export on the server at once; after a closed tab, the browser marks the download failed within about three minutes. If the browser stops the worker mid-download (or a Fleeger update retires it), the download fails and the page says so; start it again. The worker's scope is only /backup-download/, so nothing else the UI loads passes through it; a reverse proxy in front of Fleeger that filters paths must let that path through to the frontend. Browsers without service workers there collect the archive in memory and save it at the end, which may not fit for a large one (with metrics); the page then says so. That is the case on a plain-HTTP address other than localhost (service workers need HTTPS), in some private windows, when /backup-download/sw.js cannot be loaded, and in Safari and every browser on iOS (all WebKit), whose downloads from a service worker are not reliable.

This path is ideal for "give me one file I can store in a vault" and for migrating a whole platform between instances that share a master key. For unattended off-box retention, use the scheduled S3 path below.

Running a scheduled backup

scripts/backup.sh runs identically by hand, from cron, or wrapped in a Kubernetes CronJob. It needs the S3 destination and credentials in the environment:

export S3_BUCKET=s3://fleeger-backups
export AWS_ACCESS_KEY_ID=...  AWS_SECRET_ACCESS_KEY=...
# OPTIONAL — only for non-AWS S3-compatible targets (Hetzner, MinIO, Ceph):
export S3_ENDPOINT_URL=https://fsn1.your-objectstorage.com   # Hetzner region endpoint
scripts/backup.sh

S3_ENDPOINT_URL is optional: leave it empty or unset on real AWS S3 and the CLI derives the endpoint from AWS_DEFAULT_REGION. Set it only when the target is something else (Hetzner Object Storage, MinIO, Ceph). The same is true of scripts/restore.sh.

It writes s3://<bucket>/<timestamp>/ containing surreal-<db>.surql.gz per database, the VictoriaMetrics capture, a MANIFEST and its signature MANIFEST.sig (below), then updates a LATEST pointer and pushes a success metric (see Dead-man's switch below). SurrealDB and VictoriaMetrics connection details default to the local dev stack and are overridable via env (see the header of scripts/backup.sh).

How VictoriaMetrics is captured — VM_BACKUP_MODE

VM has no shell and (in k8s) no reachable filesystem, so capture is pluggable:

Mode What it does Use when
docker-exec (default) snapshot API + docker exec … tar of the snapshot dir compose, where the runner can reach the Docker socket
native HTTP /api/v1/export/native → vm-native.bin.gz anywhere with only HTTP to VM (the compose --profile backup runner)
none skip VM entirely Kubernetes, where a vmbackup sidecar streams VM to S3 separately

VictoriaMetrics' snapshot endpoints require the snapshot key. In docker-exec mode export FLEEGER_VM_SNAPSHOT_KEY (the value from deploy/compose/.env) in the environment backup.sh runs in, a host cron job included; the script hands it to curl on stdin, not on a command line. Without it the snapshot fails with a 401 and the backup stops before its upload step: the SurrealDB dumps already taken are not uploaded either, so the run leaves no backup set at all, and LATEST and the success metric stay where they were. native mode needs no key. The chart's vmbackup sidecar gets the key from the <fullname>-victoriametrics Secret (or victoriametrics.auth.existingSecret); a sidecar you added to the plain manifests needs it too, as deploy/k8s/optional/backup-cronjob.yaml shows.

Scheduling

Kubernetes — turn on backup in charts/fleeger-hub; a CronJob runs the backup (VM_BACKUP_MODE=none) and a vmbackup sidecar in the VM pod incrementally streams VictoriaMetrics to S3 (it shares the storage PVC, which a separate Job can't). Minimal values:

backup:
  enabled: true
  schedule: "0 3 * * *"
  s3Bucket: s3://fleeger-backups
  s3EndpointUrl: https://fsn1.your-objectstorage.com
  existingSecret: fleeger-s3        # keys: access-key-id, secret-access-key
  retentionDays: "30"
  vmbackup:
    dst: s3://fleeger-backups/vm     # a path DISTINCT from the DB sets

Save them as fleeger-backup.yaml and pass it with -f, to the install or to a running release:

helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values \
  -f fleeger-backup.yaml

Without --reset-then-reuse-values that upgrade resets every other value you set at install (see the Helm chart reference's Upgrade section).

compose / bare host — the runner ships as an opt-in profile; wrap one-shot runs in host cron:

cd deploy/compose
# one run (VM_BACKUP_MODE=native — captures VM over HTTP):
S3_BUCKET=s3://fleeger-backups S3_ENDPOINT_URL=https://… \
AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… RETENTION_DAYS=30 \
  docker compose --profile backup run --rm backup

# host crontab — daily 03:00:
0 3 * * * cd /opt/fleeger/deploy/compose && S3_BUCKET=… S3_ENDPOINT_URL=… \
  AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… RETENTION_DAYS=30 \
  docker compose --profile backup run --rm backup >> /var/log/fleeger-backup.log 2>&1

Suggested cadence: daily full backups. Config changes are already journalled as revisions, so sub-daily backups add little.

Retention

Two complementary knobs — use either or both:

  • Script-side prune — set RETENTION_DAYS=N (or backup.retentionDays) and backup.sh deletes S3 sets older than N days. Timestamped prefixes sort lexically, so it only ever removes YYYYMMDDThhmmssZ sets (never LATEST).
  • Bucket lifecycle — an S3/Object-Storage lifecycle rule expiring objects by age. Belt-and-braces if the backup job itself stops running.

The vmbackup sidecar keeps its own incremental history under vmbackup.dst; prune that with a bucket lifecycle rule.

Signed backup sets

Write access to the backup bucket must not make anyone admin of the hub. A restore checks what a dump may define (no namespace users, no REMOVE, …), but the rows it holds — users, roles, credentials, alert channels — are restored as they are, so a set someone put into the bucket would decide who can sign in. So backup.sh signs every set, and scripts/restore.sh restores only signed ones.

What is signed. MANIFEST (manifest format 1) is text, one LF-ended line each:

fleeger-backup-manifest 1
backup_ts=20261025T030000Z
surreal_ns=fleeger
surreal_dbs=auth inventory ingestion alerting control audit
vm_backup_mode=none
vm_snapshot=
key_id=3f2a9c0d81b4e6f7
part <sha256 hex> <size in bytes> surreal-alerting.surql.gz
part <sha256 hex> <size in bytes> surreal-audit.surql.gz
…

with one part line per object of the set. Its bytes, exactly as stored, are signed with Ed25519; MANIFEST.sig holds the 64-byte signature in base64 on one line. key_id is the first 16 hex digits of the SHA-256 of the signing key's public key (DER), which is what both scripts print. The parts are uploaded first, then MANIFEST, then MANIFEST.sig: a set whose upload stopped part way has no signature.

What a restore checks, right after reading the set and before reading anything in it (--dry-run too): MANIFEST.sig verifies under one of the configured public keys; MANIFEST is in the format above; its backup_ts is the timestamp being restored (a set copied to another prefix, to pass an old set off as a new one, is refused; a --from-dir set may sit in a directory of any name); every listed part is there with its size and SHA-256; and the set holds nothing the manifest does not list. Nothing overrides a failure of any of these, --allow-unsigned included. The signed backup_ts is printed with the verification, and the audit log row of the restore records which key verified it.

Unsigned sets — every set written by an older release, a set backup.sh wrote without a key, and the pre-restore sets restore.sh itself writes — are refused unless --allow-unsigned is given, which restores them under a loud warning (and still checks the parts of a manifest in the new format). --allow-unsigned also restores a signed set while no public key is configured, unchecked. Use it only for a set you know where it came from. A set written after signing was enabled that has no MANIFEST.sig had it removed, or the backup ran without its key: that run's log has the ERROR (no backup signing key). A MANIFEST with a non-empty key_id= and no MANIFEST.sig was signed: its signature was removed, and the refusal says so (and does not offer --allow-unsigned); restored with --allow-unsigned anyway, the warning says the signature was removed. The dry-run summary, the confirmation (with --force and RESTORE_CONFIRM=yes too) and the end of the restore each say again whether the set's signature was verified, and with which key.

The keys. The private key is a hub secret that only the backup job mounts. Mounting is not the only way to it, though: on Kubernetes whoever can read Secrets in the hub's namespace can read it, and so can whoever can read the Helm release history there (a chart-made key is in the release's stored manifest, itself a Secret in that namespace); with compose, whoever can read .env or docker inspect the backup container. Hold those as tightly as the key. The public key is not secret, and restores need it:

openssl genpkey -algorithm ed25519 -out backup-signing.pem     # the private key
openssl pkey -in backup-signing.pem -pubout | sed -n 2p        # its public key

A public key is the one base64 line of its PEM (60 characters, starting MCowBQYDK2VwAyEA). restore.sh takes them from FLEEGER_BACKUP_VERIFY_KEYS (space- or comma-separated), from the file FLEEGER_BACKUP_VERIFY_KEYS_FILE (one per line or PEM blocks) and from --verify-key <key>, and refuses anything that is not an Ed25519 public key before it downloads anything. A private key given as one (its base64 starts MC4CAQAwBQYDK2VwBCIE, or MFECAQEw / MFMCAQEw in the v2 forms) is refused as "the private key", without printing any of it: treat it as leaked (below). Nothing of a value that is not a key is printed either. Verifying needs OpenSSL 3 (openssl pkeyutl -rawin); the backup image has it. Before it checks a set, restore.sh runs a known-answer test of the openssl it finds (RFC 8032's test vector must verify, and not for another message): one that cannot check Ed25519 is refused as such, rather than every good set reading as tampered with. A MANIFEST.sig that is not a regular file is refused, with or without --allow-unsigned.

A restore never holds the private key. Whoever can run commands where the signing key is can sign a set of their own, so restores run elsewhere: on Kubernetes in deploy/k8s/optional/restore-s3-pod.yaml, which gets the S3 credentials and the public keys and no signing key (never kubectl exec into the backup CronJob's pod); with compose, the command below empties the key (-e FLEEGER_BACKUP_SIGNING_KEY=). restore.sh itself takes FLEEGER_BACKUP_SIGNING_KEY and FLEEGER_BACKUP_SIGNING_KEY_FILE out of its environment first thing, so no command it starts gets them, refuses to read the key's file (or anything under /etc/fleeger/backup-signing/) as FLEEGER_BACKUP_VERIFY_KEYS_FILE, and warns in its banner when either was set. backup.sh likewise takes them out of its environment before it runs anything, and removes the PEM it writes for a one-line key right after signing.

  • Kubernetes (chart): the chart generates the private key into <fullname>-backup-signing on the first install with backup.enabled (kept across upgrades and helm uninstall; a client-side render refuses to generate it, see the Helm chart reference's GitOps section), or takes backup.signing.existingSecret (key signing-key, PEM). Only the backup CronJob mounts it. Helm cannot derive the public key, so the install notes print the command that does:

    kubectl -n fleeger get secret fleeger-fleeger-hub-backup-signing -o jsonpath='{.data.signing-key}' \
      | base64 -d | openssl pkey -pubout | sed -n 2p

    Put it into backup.signing.publicKeys (checked at render; the CronJob gets them as FLEEGER_BACKUP_VERIFY_KEYS and warns when its own key is not among them) and into the restore pod's FLEEGER_BACKUP_VERIFY_KEYS.

  • compose: FLEEGER_BACKUP_SIGNING_KEY (the one base64 line of the private key's PEM) and FLEEGER_BACKUP_VERIFY_KEYS in deploy/compose/.env; .env.example has the commands. Only the backup service gets them; the restore command below runs in its image with the signing key emptied.

  • By hand / host cron: FLEEGER_BACKUP_SIGNING_KEY_FILE=<pem> (or FLEEGER_BACKUP_SIGNING_KEY) for backup.sh.

Keep a copy of the public key off the hub, with the credential master key: after losing the hub, a restore needs it to accept the sets (and without it, only --allow-unsigned restores them, unchecked). The private key need not be kept: a rebuilt hub can sign new sets with a new key.

Without a key, backup.sh still writes the set, unsigned, and logs an ERROR: a backup never stops for want of a key. That run prunes nothing (RETENTION_DAYS is skipped, and it says so), so signed sets are never replaced by unsigned ones, and pushes no success metric: LATEST moves to the set, but the dead-man's switch fires after its window, so a hub without a key is alerted. Setting the key is therefore a required step, for compose too (FLEEGER_BACKUP_SIGNING_KEY): until then the backup alert fires. A bucket lifecycle rule still expires by age. An unsigned run into a prefix a signed run already wrote (the same BACKUP_TS) removes that run's MANIFEST.sig first. Both FLEEGER_BACKUP_SIGNING_KEY and FLEEGER_BACKUP_SIGNING_KEY_FILE set is a configuration error: the run stops before it dumps anything.

A key that is configured but cannot sign (not Ed25519, not readable, or a key file that is not there: the CronJobs mount the Secret optional, so a missing Secret lands here instead of keeping the Job from starting) also leaves the set unsigned, and then fails the run after the upload: old sets are not pruned and LATEST and the success metric stay where they were, so the dead-man's switch fires. The values backup.sh writes into MANIFEST (BACKUP_TS, SURREAL_NS, the database and snapshot names) are held to what restore.sh reads, and a hash that cannot be taken stops the run before anything is uploaded.

Rotating the key: add the new public key to the list next to the old one, switch the private key (a new backup.signing.existingSecret, or the new value in .env), and drop the old public key once no set signed with it is kept (RETENTION_DAYS, or the bucket's lifecycle rule).

After a leak of the private key (or a suspicion of one), remove its public key from every list at once (backup.signing.publicKeys, .env, the restore pod, your off-hub copy) and switch to a new key: whoever has the leaked key can sign a set that every restore still listing it accepts. The sets signed with the leaked key are then refused like unsigned ones; restore one only with --allow-unsigned, and only one you know where it came from (one written before the leak, compared against a copy you trust).

What the signature does not cover:

  • VictoriaMetrics history from vmbackup (the chart's sidecar, or the one backup-cronjob.yaml describes): it is written to its own S3 path by vmbackup and read back by vmrestore, neither of which knows about MANIFEST. Whoever can write that path can change the metric history a vmrestore brings back (not accounts or settings). Keep it in a bucket or prefix the hub's restores do not write to, with its own credentials. A native or docker-exec set's VictoriaMetrics file is a part of the set, and is checked like the dumps.
  • An old set that really was signed is accepted: a signature says which backup job wrote a set, not that it is the newest. Someone with write access to the bucket can delete the newer sets, or point LATEST at an old one, so that a restore brings back older, genuine data (a rollback, to accounts or roles as they were then). Pick the timestamp to restore yourself, and look at the dry run's backup_ts. A set copied to another timestamp is refused (above).
  • Another hub's sets, if two hubs share a signing key: a set of one would verify on the other. Give every hub its own key (the chart generates one per release), and list only that hub's public keys in its restores.

The .fbk download is not signed: it is sealed (AES-256-GCM) with the credential master key, so only a holder of that key can make one that restores; write access to wherever it is stored is not enough.

Dead-man's switch

A backup that silently stops is worse than none. On every success backup.sh pushes fleeger_backup_last_success_timestamp to VictoriaMetrics, and a built-in alert rule (alert_rule:backup_stale, seeded by alerting migration 0003) fires when it goes stale:

time() - max(last_over_time(fleeger_backup_last_success_timestamp[7d]) or vector(0)) > 129600

last_over_time([7d]) looks back across the whole backup interval (a bare gauge would drop out of instant queries ~5 min after each push and false-fire between daily backups). or vector(0) makes a missing metric read as "infinitely stale" so silence means broken, not all clear. Default threshold 36h — one missed daily cycle plus a buffer. It's an ordinary, always-on threshold rule: attach a notification channel on the Alerts page to be paged, or mute it there. Its threshold and for_secs can be tuned on its card there or with PUT /api/alert-rules/backup_stale; its query, [7d] window included, is fixed. If you back up less often than weekly, disable backup_stale and create your own threshold rule with the same query and a wider window.

Restoring

A restore is a maintenance step: every Fleeger service is stopped, SurrealDB (and VictoriaMetrics and NATS) keep running, and each database in the backup is replaced — emptied, then the backup's dump imported into it, its migration state included. Everything written since the backup was taken is gone. When the services start again, their db migrate brings a backup from an older release up to date, exactly as an upgrade would, then defines again whatever the database defines otherwise than its migrations . That also repairs a hub restored through the Backup page of an older release, which put the backup's older definitions back over the live ones.

Why a restore replaces instead of merging. A SurrealDB export is plain INSERT statements, one per table, each listing the table's rows. Imported over live data, an INSERT that meets a record that already exists fails with already exists, and with it the whole statement — every row of that table, not only the clashing one. Restoring into running services also put back the backup's migration stamps over the newer live ones, so the migrations made since the backup ran again over data that had them already, or never ran for data that lacked them. Emptying each database first, with the services stopped, makes a restore exact, and the services migrate the restored data on start.

The audit log is the exception (The audit log): a restore never empties the audit database. When the live one holds a row it is kept as it is and the backup's is not imported; only an audit database with no row (a new or rebuilt hub) takes the backup's, first, through a transaction in which the database itself refuses to be emptied while it holds a row. Each restore is journalled there: a proposed backup.restore row after the pre-restore backup and before any service database is replaced (when it cannot be written, none is), and a completed or failed row after the last. inventory-svc backup restore also runs the audit database's migrations first; scripts/restore.sh needs it migrated (start inventory-svc of the release once): when the set's audit log is not imported and the live audit database has no audit_event table, it refuses before writing anything ("Is the audit database migrated?"), whatever the flags. After an archive's audit log is imported, both tools define its append_only event again right away, before anything else (the archive's schema and its migration stamps are not trusted to hold it); inventory-svc's db migrate and start define it again too.

Restore a set or .fbk of this release with this release's tools. A scripts/restore.sh or inventory-svc from an older release knows nothing of the audit database: it would empty (remove) the live audit database like any other and import the archive's over it, losing the hub's audit history.

There are two ways, one per kind of backup; both work the same:

Backup Restore with
.fbk file (UI download, backup export) inventory-svc backup restore <file>
S3 set (scripts/backup.sh) scripts/restore.sh <timestamp>

Both, before writing anything:

  1. read the backup's migration stamps of every database and the live database's, and refuse (nothing written) unless the backup fits the hub, by the rules of the migration runner — it takes a database's newest stamp (by applied_at) as its state and runs every migration whose id sorts after it:

    • the live database must exist: one the namespace does not hold (a new, throwaway namespace) is refused, is never read (a read creates it), and cannot be in the pre-restore backup; restoring creates it;
    • the live database must have stamps (a hub that never started is refused: start it once so it migrates, stop the services, re-run);
    • the backup's newest migration must be one the live database has, and not sort after the live newest — a backup from a newer release is refused; restore it on that release or a newer one;
    • every migration in the backup must be known to the live database;
    • no live migration that sorts before the backup's newest may be missing from the backup (the runner would never apply it);
    • no backup migration may sort after the backup's own newest (the runner would apply it a second time);
    • with an auth database in the backup, the live auth database's newest migration must be the one the re-key file was built for (the restore tool must be the hub's own release).

    Each database's line says which migrations run when the services start. --no-version-check restores although a migration check refuses: for a throwaway namespace, and for putting back a pre-restore backup. It does not override the release check: --allow-other-release does, and is meant only for putting back a pre-restore backup after restoring an archive whose auth database is at another release (which leaves the live one there); the restore says when its way back needs it.

  2. ask for confirmation: retype the archive's file name (or the S3 set's timestamp) on a terminal; without a terminal only --force or RESTORE_CONFIRM=yes will do. --dry-run stops after the checks.

  3. write a pre-restore backup of every live database the restore replaces (without the metrics history) — the way back. It needs about as much free space as a backup without metrics. If it cannot be written whole, nothing is replaced. A database that does not exist yet is not in it (the .fbk restore and the set's MANIFEST name them): putting the pre-restore backup back leaves such a database as the restore made it; REMOVE DATABASE <name>; removes it.

Before any of that, with the checks, every dump is read through as it will be imported, and one that holds what a SurrealDB export of a Fleeger database does not write is refused, whatever the flags: each statement must start with OPTION IMPORT, a data statement (INSERT, CREATE, UPSERT, UPDATE, RELATE, DELETE) or DEFINE of a table, field, index, event, function, param, analyzer, access or user, the last two ON DATABASE. Outside strings and quoted identifiers, no statement may hold DEFINE, REMOVE, ALTER or ACCESS followed by a word anywhere but at its start, ON ROOT, ON NAMESPACE, ON NS, or USE NS/USE DB; a comment is only a -- line of its own between statements. The guard does not read code the way SurrealDB's lexer does, so it refuses whatever the two could read differently: outside strings and quoted identifiers only printable ASCII, space and tab (a CR only at the end of a line); in a comment line only printable ASCII and tab (SurrealDB also ends a comment at CR, U+0085, U+2028 and U+2029, and reads VT, FF and Unicode spaces as whitespace); and no / (a regex literal can hold a quote; no export writes a division). The refusal names the line (counted as the dump is imported) and the reason; nothing is written. (A table named root, ns or namespace, or a field named access, would be refused too: none of Fleeger's is.) This guard is a pre-filter, which turns a crafted archive away early and says why; what keeps a restore safe are the two server-side steps below.

Then, database by database in archive (name) order, each is emptied (REMOVE DATABASE, DEFINE DATABASE) and its dump imported without the access key clause and without the rows of the migration lock (schema_lock: a lock held when the backup was taken would otherwise come back held, and db migrate would wait on it). The import runs as a user of that database only: a database OWNER with a random name (fleeger_restore_…) and password, defined right before the import. It is removed again after the import however that went, and also when its own definition was sent but answered with an error (the server may have applied it); inventory-svc's restore also removes it if the task that replaces the database stops part way. The server itself refuses whatever the dump would do outside that database: a user or access on the namespace or root level (DEFINE USER OVERWRITE root ON ROOT … included), or a write to another database (USE DB auth; …). Such an import fails, and the restore stops there. If the user cannot be made at all (restore.sh reads its name and password from /dev/urandom), the database is left emptied and not imported, and the restore ends with the usual PARTIALLY RESTORED summary and the way back.

A removal can still fail, and a restore that is killed (SIGKILL, a node lost) cannot remove its user at all. So:

  • The check after each import (below) removes this run's own user if it is still there, and reports it as a failed clean-up, not as SECURITY. If it cannot remove it either, it says so, with the statement to run, and the restore goes on. Any other database user, one named like a restore user included, is still an extra: every database the check looks at was emptied before its import, so only the import can have defined it.
  • As a restore starts, and again once its last import is done, every service database of the namespace (with restore.sh, every database) is searched for users named fleeger_restore_ followed by 16 hex digits. Each is removed, and reported as a clean-up: left by an earlier restore that was stopped. One that cannot be removed, or a database that cannot be searched, is reported with the statement to run; none of it stops the restore. Such a user is an OWNER of its database only, with a random password no one was given.

After every import that was sent, however it went, every database restored so far is checked again, against the server's own text of its definitions: INFO FOR DB, INFO FOR TABLE for each of its tables (events, fields with their VALUE/DEFAULT/ASSERT, indexes, views), and the accesses and users of the namespace and root, as the server prints what it parsed (keywords in capitals, no comments, nested statements spelled out), not as the dump spelled it. Then:

  • What no migration defines is removed: in the database, an access other than auth's user and every database user (DEFINE USER … ON DATABASE); on the namespace and root levels, every access and user that was not there before the first import. The restore says so loudly (SECURITY: …). A database user an operator defined by hand in a service database is removed too: it is in the backup, but no migration defines it. Define it again after the restore (the SECURITY line names it).
  • A namespace or root access or user that was there before and is not as it was stops the restore, with what it read before and how to define it again (OVERWRITE, with its password or key). The server prints [REDACTED] for a password hash or a signing key, so this sees a changed role or duration, not a changed password: what keeps the import from changing one is that it runs as a database user.
  • A definition whose body would define, remove or change something when it runs (an event, function, field, view, param or access holding DEFINE, REMOVE, ALTER, USE, ACCESS, ROOT, NAMESPACE or NS as a word after its own start, in any case), or one of a kind no Fleeger export holds, stops the restore before anything runs it (not the recount, not the next import). It names each, with the REMOVE statement to run as root in that database; do not start the services until it is removed, or restore an archive you trust. None of Fleeger's own definitions holds such a word; a future migration whose definition did would stop every restore until this check knows it.

If the definitions cannot be listed or removed, the restore stops after that database and says what to check and remove by hand; if they cannot be listed before the first import, or the import user cannot be defined, that database is left emptied and not imported.

Right after that check, for auth, whatever its import's outcome, the auth database's user access is re-defined with a fresh signing key, so everyone is signed out (below); for ingestion, once the check found nothing to stop for, the resource counts per location are recomputed from the restored resources (services/ingestion-svc/migrations/after-restore/recount_resources.surql; the recount writes, which runs the database's events); if that fails, the restore goes on, says what to run, and exits non-zero when it is done. Once the last import is done, every restored database is checked once more, and auth is re-keyed again after that, so the access the restore ends with is this release's, with a key of the server's. A metrics history in the backup is imported last: it only adds samples to what VictoriaMetrics holds, and putting the pre-restore backup back does not take them out.

Whoever can write the backup bucket can already plant data rows (an auth account with a password they know, say), which no check of definitions sees: treat write access to the bucket, and to any .fbk you restore, as admin-equivalent.

The first failure stops the restore. It says which databases were replaced, which one failed (emptied, and maybe partly imported), which were not attempted, and how to put back the pre-restore backup. Ctrl-C, a hangup or SIGTERM stops it before the next database (and before the metrics history), never in the middle of one (the database under way is finished, with its re-key, recount and checks). There is no automatic rollback.

When it is done, start the services; their db migrate runs the migrations named in the checks, and everyone signs in again.

Restoring a .fbk: inventory-svc backup restore

The restore runs in the inventory-svc image of the hub's release, which holds the master key and the SurrealDB credentials. With compose, from deploy/compose, with the archive in ./restore:

docker compose stop gateway-svc auth-svc inventory-svc ingestion-svc \
  alerting-svc control-svc core-agent
mkdir -p restore && cp ~/fleeger-backup-20260817T072200Z.fbk restore/

# Check it against the live databases (writes nothing):
docker compose run --rm --no-deps --entrypoint /usr/local/bin/service \
  --user "$(id -u):$(id -g)" -v "$PWD/restore:/restore" -e TMPDIR=/restore \
  inventory-svc backup restore --dry-run /restore/fleeger-backup-20260817T072200Z.fbk

# Restore it (asks you to retype the file name):
docker compose run --rm --no-deps --entrypoint /usr/local/bin/service \
  --user "$(id -u):$(id -g)" -v "$PWD/restore:/restore" -e TMPDIR=/restore \
  inventory-svc backup restore /restore/fleeger-backup-20260817T072200Z.fbk

docker compose up -d

--no-deps keeps the stopped services stopped; --user lets the command write the pre-restore backup into ./restore as you. -e TMPDIR=/restore puts the decoded temp copy of the archive (below) on that bind mount, next to the archive, so ./restore needs free space for the archive, about three quarters of it again for the copy, and the pre-restore backup. Without it the copy goes to the run container's /tmp, its writable layer, which is on the disk of Docker's data root (docker info -f '{{.DockerRootDir}}'): that disk then needs the free space of about three quarters of the archive instead. On Kubernetes, scale the Fleeger deployments to zero and use the pod in deploy/k8s/optional/restore-pod.yaml, whose header has the steps (kubectl cp the archive in, kubectl exec -it the same command, kubectl cp the pre-restore backup out). As it is, it fits a hub deployed from deploy/k8s (the Secrets fleeger-credential-key and fleeger-db, the ConfigMap fleeger-config, the label app.kubernetes.io/instance: fleeger the network policies select on), once its image is set to the inventory-svc image the hub runs. For a Helm install, change the image to <image.registry>/inventory-svc:<image.tag or the chart's appVersion>, the Secrets to <fullname>-credential-key (or credential.existingSecret) and <fullname>-db (or surrealdb.existingSecret), the ConfigMap to <fullname>-config, and the instance label to the release name. During a credential-key rotation the pod reads FLEEGER_CREDENTIAL_KEYS_RETIRED from the optional Secret fleeger-credential-keys-retired, as the hub's services from deploy/k8s do; for a Helm install, point it at credential.retiredKeysSecret instead, or drop the entry. The pod sets TMPDIR=/restore, so the decoded archive lands on its own claim, fleeger-restore, not in the running inventory-svc pod's /tmp (a restore no longer goes through that pod, which is why the chart and deploy/k8s put no size on its /tmp for it): size the claim for the archive, about three quarters of it again for the decoded copy, and the pre-restore backup.

Options: --dry-run, --force (or RESTORE_CONFIRM=yes), --no-version-check, --allow-other-release, and --pre-restore-file <path> (default: fleeger-pre-restore-<time>.fbk next to the archive; it must not exist yet). The archive is always a local file; restoring straight from a remote source is not supported yet. A Ctrl-C, hangup or SIGTERM while the pre-restore backup is written stops the restore with nothing replaced and the unfinished file removed. The connection to the inventory database (for the secrets check below) is opened only after the pre-restore backup is written, so the checks and a dry run create nothing, also in a namespace that does not exist. The archive is read as it is used, and kept decoded in a temp file under TMPDIR (about three quarters of its size; /tmp when unset; see above for where that is with compose and on Kubernetes); nothing of it is held whole in memory. The temp file is removed when the restore ends, however it ends; a disk that fills while it is written stops the restore with nothing imported. Its authentication tag is checked over the whole file before anything is checked or written; an archive that does not decrypt, a framing that does not hold together, or a part this release does not restore, writes nothing. All of it counts against the [backup] timeout_secs budget above; the re-key, the emptying and the recount of each database have 30 seconds each of their own. An import cut off by the budget may still be under way on the server, so the restore stops there without the re-key and the checks (they would not be final): it says the outcome is unknown, that the services must not start, and the command that runs the restore again once the database server is idle (with a larger timeout_secs), which empties that database again and does them.

The databases come back as they were dumped, so their secrets come back under the key they were sealed with, which can be a retired one (see below). Once the import is over, metrics included, the restore re-encrypts under the current FLEEGER_CREDENTIAL_KEY every credential in inventory-svc's database still under a retired key, the way inventory-svc's service rekey does. It also counts the notification channels still under another key, but does not rewrite those, since they belong to alerting-svc. When any secret is left off the current key, or one of these checks could not run, the restore still succeeds, and prints how many and in which service to run service rekey.

Restoring an S3 set: scripts/restore.sh

scripts/restore.sh <timestamp> downloads a set and restores SurrealDB automatically, and (for native-mode sets) imports the VM history too. A docker-exec set's VM snapshot tarball is saved to RESTORE_DIR (default: the directory you run the script from) for the manual step below; everything else is downloaded to a temp directory that is removed when the script exits.

The script refuses to replace data quietly:

  • it prints the target endpoint, namespace and backup set before anything is downloaded or written;
  • --dry-run downloads the set, runs the checks above, says what it would do and exits, writing nothing;
  • it refuses a set with more than one VictoriaMetrics snapshot tarball rather than restore one of them and drop the rest, and one whose vm-*.tar.gz is not a file (below);
  • it refuses a set whose signature does not verify under the configured public keys, or that is not as its signed MANIFEST lists it, and an unsigned one without --allow-unsigned (Signed backup sets); and it says again, at the confirmation and at the end, whether the signature was verified;
  • on a terminal it makes you retype the backup timestamp (a reflexive y won't do it);
  • with no terminal — a CronJob, CI, kubectl exec without -t — it refuses outright unless you passed --force or RESTORE_CONFIRM=yes. That is what stops the backup image (which already carries the S3 credentials and the production SURREAL_* env) from restoring by accident.

The pre-restore backup is a set like the ones backup.sh writes, but unsigned (the restore holds no signing key; it is the hub's own data, written to the local disk, not the bucket): RESTORE_DIR/pre-restore-<time>/ with a surreal-<db>.surql.gz per database and a MANIFEST. RESTORE_DIR must be an existing, writable directory; it is checked before the set is downloaded, also on a dry run. --from-dir <dir> restores a set from a local directory instead of S3, which is how a pre-restore set is put back (the summary prints the exact command, with --allow-other-release when it is needed). A --from-dir set is read where it is, and a symlink in it counts as the file it points to: the checks read the file through the link, and so does the import, later. Keep the directory (and whatever its links point to) where only you can write until the restore is done, or copy it first (cp -rL).

scripts/restore.sh --from-dir ./pre-restore-20261016T230431Z --no-version-check --allow-unsigned

With compose, run it in the backup image, from deploy/compose, with a directory for the pre-restore set (the image's / is not writable for its user, and the CLIs want a writable home):

mkdir -p restore
docker compose --profile backup run --rm --no-deps \
  --entrypoint /usr/local/bin/restore.sh --user "$(id -u):$(id -g)" \
  -e FLEEGER_BACKUP_SIGNING_KEY= \
  -e HOME=/tmp -e RESTORE_DIR=/restore -w /restore -v "$PWD/restore:/restore" \
  backup --dry-run 20260817T072200Z
# then the same without --dry-run

-e FLEEGER_BACKUP_SIGNING_KEY= keeps the backup service's private key out of the restore's container (Signed backup sets); restore.sh checks the set with the service's FLEEGER_BACKUP_VERIFY_KEYS.

On Kubernetes, restore from deploy/k8s/optional/restore-s3-pod.yaml, a pod of the backup image with the S3 credentials, the SurrealDB password and the public keys, and no signing key; its header has the steps (and what to change for a Helm install). Not from the backup CronJob's pod, which mounts the private key. RESTORE_DIR is its claim; kubectl cp the pre-restore set out before you delete it.

While it replaces the databases, the script stops only between two of them on SIGHUP, SIGINT (Ctrl-C) or SIGTERM, and ignores SIGPIPE. Each surreal command it runs ignores those three signals (it is started from a subshell that ignores them), and runs in a session of its own where setsid is installed; so neither a Ctrl-C at the terminal nor a signal to the whole process group ends an import part way. That the SurrealDB 3.2.4 CLI keeps the signals it inherits ignored was checked on the real CLI, not inferred: an import of 150,000 rows sent SIGINT, SIGHUP and SIGTERM to its whole process group finished all of them and exited 0 when started that way, and died at about 88,000 rows with exit code 130 when not. Their output and errors go to files and are printed after each database (output to stdout, errors to stderr), and the script carries on when writing its own output fails. This is meant for a dropped kubectl exec or SSH session (a hangup, and output that is gone) and a plain kill. SIGKILL cannot be caught: a pod or container killed outright (kill -9, docker kill, a pod deleted past its grace period) can stop it between the auth import and its re-key; if that happens, run the re-key by hand (below).

The re-key file is rekey_access.surql: the script uses FLEEGER_REKEY_FILE if set, else services/auth-svc/migrations/after-restore/rekey_access.surql next to the script in a checkout, else /usr/local/share/fleeger/rekey_access.surql (the backup image). A set with an auth dump is not restored without one. The banner prints which file it uses, the auth migration the file was built for (its -- built-for: line) and its SHA-256. Run the backup image of the hub's own release (backup.image, or a rebuilt fleeger/backup:local), or point FLEEGER_REKEY_FILE at that release's file. The recount file is found the same way (FLEEGER_RECOUNT_FILE, then services/ingestion-svc/migrations/after-restore/recount_resources.surql, then /usr/local/share/fleeger/recount_resources.surql); a set with an ingestion dump is not restored without one.

export S3_BUCKET=s3://fleeger-backups  S3_ENDPOINT_URL=https://fsn1.your-objectstorage.com
export AWS_ACCESS_KEY_ID=...  AWS_SECRET_ACCESS_KEY=...
export FLEEGER_BACKUP_VERIFY_KEYS=MCowBQYDK2VwAyEA...   # the backup signing key's public key

# Always look first — no confirmation needed, nothing is written:
scripts/restore.sh --dry-run 20260817T072200Z

# Then, with the services stopped, restore (prompts on a terminal):
scripts/restore.sh 20260817T072200Z

# Non-interactive (automation that genuinely means it):
RESTORE_CONFIRM=yes scripts/restore.sh 20260817T072200Z

just restore <ts> [options] forwards to the same script with its options (just restore <ts> --dry-run), so it prompts too.

SurrealDB restore details

SurrealDB exports tables alphabetically, which can place a graph-edge (INSERT RELATION) statement before the records it links — e.g. the roles user→role edges before the user rows. A naive import then fails with "record 'user:…' does not exist". Both restore paths rewrite each dump so every INSERT RELATION runs last, after its endpoints exist, and leave out the migration lock's rows, the same way (fleeger_shared::restore and the awk pass in the script are kept identical, and tested to be).

To verify a backup without touching live data, restore it into a throwaway namespace by setting SURREAL_NS. That namespace has no migration state to check against, so pass --no-version-check:

SURREAL_NS=restoretest scripts/restore.sh --no-version-check 20260817T072200Z
# then check row counts match the live namespace, and REMOVE NAMESPACE restoretest

Re-keying after a restore

Both restore paths re-key the auth database on their own. When that step failed (the restore says so), or after restoring the auth database by any other means, re-define the user access by hand before anyone signs in, with the rekey_access.surql of the release the hub runs. This signs everyone out. INFO FOR DB does not show the key, so it cannot tell whether this is needed; when in doubt, run it.

Compose (in deploy/compose; the backup image carries the file):

docker compose --profile backup run --rm --entrypoint sh backup -c \
  'surreal import --endpoint "$SURREAL_ENDPOINT" --username "$SURREAL_USER" \
     --namespace "$SURREAL_NS" --database auth /usr/local/share/fleeger/rekey_access.surql'

Kubernetes, from a checkout of the hub's release (the data namespace is fleeger). The SurrealDB pod has SURREAL_USER and SURREAL_PASS in its environment, and surreal import reads both from there when no --username/--password is given:

kubectl -n <ns> exec -i deploy/<surrealdb> -- /surreal import \
  --endpoint http://localhost:8000 \
  --namespace fleeger --database auth /dev/stdin \
  < services/auth-svc/migrations/after-restore/rekey_access.surql

The gateway's auth cache may keep accepting a token it already checked for up to 5 seconds afterwards. Upgrading to a release with auth migration 0013_rekey_access, and again to one with 0015_drop_signup, re-keys every install once in the same way.

The core agent's revocation after a restore

The auth database holds the revocation ledger, and with it the revocation of the bundled core agent's earlier, wider NATS JWTs that ./gen-auth.sh fix-perms core-agent records. A backup taken before that ran does not hold it: after restoring one, the next push of the SPOKES account (auth-svc's reconcile, within 10 s) leaves it out, and the old core-agent.creds connects again with its old grants. Run ./gen-auth.sh fix-perms core-agent again after such a restore (Agent credentials & revocation); it re-issues the core agent's credential and records the revocation once more. On a hub without the core agent, or one that never ran it, there is nothing to do.

Restoring a backup from before a key rotation finished

A backup holds the secrets as they were stored when it was taken. A key rotation and its rekey runs re-encrypt the live databases only, never a backup already taken. Any backup taken before both rekeys were confirmed can hold old-key secrets, including one taken after the new key became primary: it is sealed under the new key, but the secrets inside are still the old key's until the rekey reaches them. The same goes for alert-channel secrets after a rotation done before alerting-svc had a rekey: they stayed on the old key until you rekeyed or re-entered them.

  • a .fbk is sealed as a whole under the key that was primary when it was taken. backup restore opens it with any key inventory-svc holds, current or retired, so it refuses with could not decrypt the backup — wrong master key when that key is no longer supplied;
  • the databases inside a .fbk or an S3 set are imported as they are, so the stored credentials (inventory) and notification-channel secrets (alerting) come back as the old key's ciphertexts, both on rows the backup restores over and on rows it brings back. backup restore then re-encrypts the credentials under the current key where the old key is supplied, and reports what it left, with the notification channels still under the old key. scripts/restore.sh never decrypts anything, so it succeeds either way and reports nothing; without the old key those secrets then fail at their next use.

To restore such a backup:

  1. Supply the old key again as retired, to both inventory-svc and alerting-svc. With compose, set FLEEGER_CREDENTIAL_KEYS_RETIRED in .env (comma-separated for several, e.g. a backup from two rotations ago) and run docker compose up -d. With Helm, put it in the Secret named by credential.retiredKeysSecret (key retired-keys) and upgrade with --reset-then-reuse-values --set credential.retiredKeysSecret=<name>. With the raw manifests in deploy/k8s/, create the Secret fleeger-credential-keys-retired (key retired-keys) and kubectl -n fleeger rollout restart deploy/inventory-svc deploy/alerting-svc (Secrets the hub needs). The current key stays as FLEEGER_CREDENTIAL_KEY.
  2. Restore the backup (inventory-svc backup restore <file> for a .fbk, or scripts/restore.sh <ts>), with the services stopped, then start them again.
  3. Run service rekey in inventory-svc and in alerting-svc, as in step 3 of the rotation, until a second run of each reports 0 ... re-encrypted. A run that stops with cannot decrypt … (is the retired key present?) names a secret sealed under a key you have not supplied. After backup restore the inventory-svc credentials are usually done already: the restore's warning names the services that still need it.
  4. Only then drop the old key from the retired keys again. Keep your own copy of it while you keep any backup taken before both rekeys were confirmed.

If the old key is gone, a .fbk sealed under it cannot be restored. An S3 set, or a .fbk sealed under the new key, can still be, but every credential and notification channel it restores under the old key must be re-entered. Until all of them are, rekey cannot finish: it aborts on the first secret it cannot decrypt, so the next rotation's rekey stops there too.

VictoriaMetrics restore

native-mode sets (compose runner, or the UI download with metrics): the set contains vm-native.bin.gz, and restore.sh imports it over /api/v1/import/native automatically — no downtime, no data-dir swap.

docker-exec snapshot tarballs: the tarball is a complete data directory. restore.sh saves it as vm-<snapshot>.tar.gz in RESTORE_DIR — an existing, writable directory you choose, by default the one you run the script from — and prints its full path. It saves the tarball before importing anything, so if it cannot (the directory is missing or read-only, as / is in the backup image), it stops with nothing imported and you re-run with a different RESTORE_DIR. The tarball can be several GB, so pick a directory with room for it:

RESTORE_DIR=/srv/restore scripts/restore.sh 20260817T072200Z

backup.sh writes one tarball per set. A set holding more than one (two runs into the same timestamp, files copied in by hand), or a vm-*.tar.gz that is not a file (a directory, a dangling link), is refused before anything runs, also on a dry run and whatever flags are given: which snapshot belongs to the dumps cannot be told from the set. The refusal names the tarballs and the snapshot the set's MANIFEST names; copy the set into a directory, keep the tarball you mean (or none), and restore that with --from-dir.

Then restore it deliberately:

  1. Stop VictoriaMetrics: docker compose stop victoriametrics
  2. Extract and replace the data dir contents, using the path restore.sh printed:
    mkdir -p /tmp/vm-restore
    tar xzf /srv/restore/vm-<snapshot>.tar.gz -C /tmp/vm-restore
    # replace the vm-data volume's contents with /tmp/vm-restore/<snapshot>/
  3. Start it again: docker compose start victoriametrics

Kubernetes vmbackup sidecar (backup.vmbackup): restore with vmrestore from vmbackup.dst into the VM storage path while VM is stopped — see the VictoriaMetrics vmrestore docs.

Metrics history is the least critical tier (config and audit live in SurrealDB); losing the most recent window between backups is acceptable.

Full disaster recovery — from nothing

Rebuilding a hub from a total loss, in order:

  1. Restore the secrets you kept off-box — FLEEGER_CREDENTIAL_KEY (into the Secret / env) and the NATS auth material (deploy/nats/auth/). Without the key the credential ciphertext is dead weight; without the NATS material agents can't reconnect. Have the backup signing key's public key at hand for step 4 (FLEEGER_BACKUP_VERIFY_KEYS, Signed backup sets); the rebuilt hub may sign its new sets with a new private key. If the backup you will restore was taken before a key rotation's rekeys were confirmed, also supply the old key as retired, and rekey after step 5 (above).
  2. Bring up the hub (compose or Helm) with that key in place, so the schema migrations run and the services boot. The restore checks the backup against the migration state this leaves, so use the release the backup was taken on or a newer one.
  3. Stop the Fleeger services (SurrealDB, VictoriaMetrics and NATS keep running; the commands are above).
  4. Restore SurrealDB from the latest set (scripts/restore.sh <ts>), or the .fbk (inventory-svc backup restore <file>). Either signs everyone out (above).
  5. Start the services (docker compose up -d, or scale the deployments back up): their db migrate brings an older backup up to date. Log in again.
  6. Restore VictoriaMetrics per its mode above (optional — history only).
  7. Verify: log in, confirm Locations/Resources/credentials resolve, alert rules are present, the audit log (Audit page; audit_event in the audit database) is intact and holds the restore's backup.restore rows, and agents reconnect and resume reporting.

Restore drill checklist

Do this periodically, not just after a disaster:

  1. Run scripts/restore.sh --no-version-check <ts> with SURREAL_NS=restoretest (or restore a .fbk with backup restore on a scratch instance sharing the master key). The namespace is the only thing keeping this off live data, and --no-version-check (the databases of a throwaway namespace do not exist yet, so there is no migration state to check against; they are not read, and the restore creates them) turns off the migration checks, so read the banner the script prints and check the endpoint and namespace before confirming — --dry-run first if you want to see the set without touching anything. The live services may keep running for this one: nothing of the live namespace is written. Run scripts/restore.sh --dry-run <ts> against the live namespace too: it says whether the set would restore on this release, and whether its signature verifies with the public key you keep off the hub (pass that copy with --verify-key).
  2. Confirm row counts in restoretest match live for user, location, resource, config_revision, checkpoint, alert rules, and audit_event in the audit database.
  3. Confirm the VM history imports (native) or the tarball holds a non-empty snapshot dir (docker-exec).
  4. REMOVE NAMESPACE restoretest.
  5. Confirm the Backup dead-man's switch alert is present and not firing.

A restore that has been drilled is the only kind you can trust at 3am.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.