// Betrieb
Backup & disaster recovery
This is the platform's own disaster-recovery path — how the hub's state is dumped off-box and how to bring it back. It is separate from the config checkpoints in Config revisions, checkpoints & rollback: checkpoints roll back a change you made; this restores everything after losing the hub's volume.
Read this before you need it. A backup nobody has restored from is not a backup.
There are two ways to capture a backup, sharing one format:
- Scheduled, off-box → S3 (
scripts/backup.sh) — the operational DR path. Runs by hand, from a Kubernetes CronJob, or a compose--profile backuprun. - On-demand, encrypted download from the UI (admin Backup page) — a
single self-contained
.fbkfile you keep somewhere safe. Encrypted with the credential master key.
Either is restored offline, with the Fleeger services stopped: a .fbk
with inventory-svc backup restore, an S3 set with scripts/restore.sh (see
Restoring). A restore replaces each database with the backup's,
and the services bring an older backup up to date when they start again.
What is backed up
| Source | Contents | How |
|---|---|---|
SurrealDB auth |
users, roles, RBAC, registration tokens | surreal export / HTTP /export |
SurrealDB inventory |
Locations, Resources, credentials (ciphertext), config revisions + checkpoints, hardware inventory | surreal export / HTTP /export |
SurrealDB ingestion |
reachability / uptime state | surreal export / HTTP /export |
SurrealDB alerting |
alert rules, notification channels, mutes | surreal export / HTTP /export |
SurrealDB control |
control-svc's state (its older audit log, kept until copied) | surreal export / HTTP /export |
SurrealDB audit |
the hub-wide audit log: imported only into an empty audit database, never emptied by a restore | surreal export / HTTP /export |
| VictoriaMetrics | all metric history | snapshot / native export (see below) |
The scheduled path uploads everything to an S3-compatible bucket (Hetzner
Object Storage in production) under a timestamped prefix, alongside a MANIFEST
of every object's hash and its signature (MANIFEST.sig, see
Signed backup sets). The UI download packs the same
databases into one encrypted archive.
Credential master key — back it up separately. Every dump contains credential ciphertext only. It is useless without
FLEEGER_CREDENTIAL_KEY, which is not in any backup (it lives in a Kubernetes Secret / env, and the.fbkarchive is encrypted with it). Store that key durably and off-box — losing it means the encrypted credentials and alert-channel secrets cannot be recovered even with a perfect database restore.After a key rotation, keep the old key too. A backup holds whatever ciphertexts the databases held when it was taken, and the rekey only re-encrypts the live databases. A backup taken after the new key became primary but before both rekeys finished is sealed under the new key yet still holds old-key secrets. Keep the old key for as long as you keep any backup taken before both rekeys were confirmed; see Restoring a backup from before a key rotation finished.
NATS auth material (
deploy/nats/auth/, generated bygen-auth.sh) is likewise not in the DB. Keep it with the master key — without it, agents can't reconnect after a rebuild. See Agent credentials & revocation.
Encrypted download from the UI
The admin Backup page (visible to holders of iam.manage) is the simplest
path for a whole-platform snapshot you control:
- Download — re-enter your password, optionally tick Include VictoriaMetrics
history (much larger; metrics are regenerable), and download a
fleeger-backup-<timestamp>.fbkfile. It contains all six service databases (and optionally metrics), framed and encrypted with the credential master key (AES-256-GCM). It is only restorable on a Fleeger instance holding that same key, as itsFLEEGER_CREDENTIAL_KEYor as one of itsFLEEGER_CREDENTIAL_KEYS_RETIRED. - Restore — not on the page any more. A restore replaces every
service database and runs with the services stopped, so it is a one-shot
command on the hub:
inventory-svc backup restore. The page says how.POST /api/backup/restoreanswers410 Gone.
The download is recorded in the audit log (backup.download:
a rejected row for a wrong password, a proposed row before the archive is
asked for, and how it ended). It is fail-closed: when the proposed row cannot
be written, no archive is made and the gateway answers 503.
The download is password-reverified at the gateway and admin-only. inventory-svc
performs the work (it already holds the master key and SurrealDB's root
credentials): the databases are read over SurrealDB's HTTP /export, metrics
via VictoriaMetrics' native export. The same archive can be written on the hub
without the UI: inventory-svc backup export <file> (with --metrics for the
history), run like the restore below. It writes a new file only (one that
exists is refused), never stdout, and says on stderr how large the archive is
and its SHA-256 (as sha256sum <file> prints it).
The command is recorded in the audit log too, as backup.export on
platform by cli:<operating-system user>: a proposed row naming the file,
the archive's format, the databases and whether the metrics are in it, before
the file is created, then completed with the size and SHA-256, or failed
saying how far it got and whether the unfinished file was removed (if it
could not be, the row names it for you to delete). It is fail-closed:
while the audit database cannot take the row (not migrated yet: run
inventory-svc db migrate first), it exits non-zero and writes nothing; a
scheduled export fails the same way. No key is ever in a row. The archive
holds the audit database as it is when that part is read, so it holds the
export's own proposed row but not its outcome
(details).
A download or restore has a time budget, set by the optional [backup] section
of inventory-svc.toml: timeout_secs (default 3000) for all its SurrealDB
and VictoriaMetrics calls together, and connect_timeout_secs (default 10)
for each connection. Each must be between 1 and 86400, or inventory-svc
refuses to start. A store that stops answering fails the operation at that
budget. Keep timeout_secs below the time the gateway and edge allow the
download route (an hour), so inventory-svc is the one that gives up and
reports; for a very large restore, raise it in the inventory-svc.toml the
restore runs with. These calls never go through a proxy set in inventory-svc's
environment (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, in either case): the
SurrealDB calls carry its root credentials, and both stores are reached
directly.
inventory-svc sends the archive as it produces it, rather than building it whole first, and in constant memory (about a MiB) whatever the size of a part. It reads one part at a time (each database dump, then the metrics), and each part as its store sends it: every MiB it has read is encrypted and sent before it reads on, so it never holds a part whole, and a backup with metrics fits the services' standard memory limit. The connection to the store stays open while its part is being sent, and closes when the download is cancelled.
Because the stores are read only as fast as the download proceeds, the
[backup] timeout_secs budget covers practically the whole download, not just
reading the stores: a download still running when it is spent breaks off.
The gateway and the edge give the route an hour anyway; to download large
backups (with metrics) over a slow link, raise timeout_secs and the
gateway's [http] backup_timeout_secs together, keeping the first below the
second.
A store that fails on the first part before its first MiB (or the whole part, if it is smaller) has been read is an error the page shows. One that fails later, also part way through a part, breaks the download, the same way as below, and inventory-svc logs why.
Two limits to know:
- An archive holds at most 64 GiB (the limit of AES-GCM for one value). A
larger backup, in practice only one with a very large metrics history,
breaks off at that point, and inventory-svc logs
encrypt backup. - A store answer that ends cleanly but short (the store stops early without an error) is taken as the whole part: the archive cannot tell it from a complete one. Check a restored platform as in the drill below.
The archive's format changed in 1.0.0-rc.5 (framing version 2: each part in
blocks of 1 MiB, not after its whole length). Backups taken before still
restore. A backup taken on this version restores only on this version or a
newer one; an older Fleeger refuses it as not a Fleeger backup archive, so
take a backup before upgrading if you may roll back. An archive from a newer
Fleeger than the one restoring it is refused as made by a newer Fleeger;
restore it there. A restore reads the archive's framing within the
[backup] timeout_secs budget too, and refuses one whose framing does not
hold together as corrupt backup archive, with nothing imported.
The gateway relays the downloaded archive to the browser as inventory-svc
sends it, rather than holding it whole first, so a large download (one with
metrics) costs the gateway no more memory than is in flight. Its
[http] backup_timeout_secs budget covers relaying the archive too. A download cut
off after it started, inventory-svc's connection lost or out of that budget,
ends as a broken download (the connection is aborted, and the page reports an
error), never as a shorter file that looks complete. A download inventory-svc
refuses or fails before it starts (a store that does not answer, say) is not
offered as a file: the gateway passes inventory-svc's error answer on as it is,
and the page shows its message.
The page saves the download straight to disk as it arrives, through a small
service worker at /backup-download/sw.js: the page sends the request
itself, so the session token and the password never reach the worker, and
hands the answer to it a piece at a time; the worker serves it to the browser
as an ordinary download. The browser shows it in its download list, writes it
no faster than the disk takes it (the page reads the archive only as fast), and
marks it failed if it breaks off. Cancelling it there, with the page's
Cancel button, or by closing the tab stops the export on the server at
once; after a closed tab, the browser marks the download failed within about
three minutes. If the browser stops the worker mid-download (or a Fleeger
update retires it), the download fails and the page says so; start it again. The worker's scope is
only /backup-download/, so nothing else the UI loads passes through it; a
reverse proxy in front of Fleeger that filters paths must let that path through
to the frontend. Browsers without service workers there collect the archive in
memory and save it at the end, which may not fit for a large one (with
metrics); the page then says so. That is the case on a plain-HTTP address other
than localhost (service workers need HTTPS), in some private windows, when
/backup-download/sw.js cannot be loaded, and in Safari and every browser on
iOS (all WebKit), whose downloads from a service worker are not reliable.
This path is ideal for "give me one file I can store in a vault" and for migrating a whole platform between instances that share a master key. For unattended off-box retention, use the scheduled S3 path below.
Running a scheduled backup
scripts/backup.sh runs identically by hand, from cron, or wrapped in a
Kubernetes CronJob. It needs the S3 destination and credentials in the
environment:
export S3_BUCKET=s3://fleeger-backups
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
# OPTIONAL — only for non-AWS S3-compatible targets (Hetzner, MinIO, Ceph):
export S3_ENDPOINT_URL=https://fsn1.your-objectstorage.com # Hetzner region endpoint
scripts/backup.shS3_ENDPOINT_URL is optional: leave it empty or unset on real AWS S3 and
the CLI derives the endpoint from AWS_DEFAULT_REGION. Set it only when the
target is something else (Hetzner Object Storage, MinIO, Ceph). The same is
true of scripts/restore.sh.
It writes s3://<bucket>/<timestamp>/ containing surreal-<db>.surql.gz per
database, the VictoriaMetrics capture, a MANIFEST and its signature
MANIFEST.sig (below), then updates a LATEST
pointer and pushes a success metric (see Dead-man's switch below). SurrealDB
and VictoriaMetrics connection details default to the local dev stack and are
overridable via env (see the header of scripts/backup.sh).
How VictoriaMetrics is captured — VM_BACKUP_MODE
VM has no shell and (in k8s) no reachable filesystem, so capture is pluggable:
| Mode | What it does | Use when |
|---|---|---|
docker-exec (default) |
snapshot API + docker exec … tar of the snapshot dir |
compose, where the runner can reach the Docker socket |
native |
HTTP /api/v1/export/native → vm-native.bin.gz |
anywhere with only HTTP to VM (the compose --profile backup runner) |
none |
skip VM entirely | Kubernetes, where a vmbackup sidecar streams VM to S3 separately |
VictoriaMetrics' snapshot endpoints require the snapshot key. In
docker-exec mode export FLEEGER_VM_SNAPSHOT_KEY (the value from
deploy/compose/.env) in the environment backup.sh runs in, a host cron job
included; the script hands it to curl on stdin, not on a command line. Without
it the snapshot fails with a 401 and the backup stops before its upload step:
the SurrealDB dumps already taken are not uploaded either, so the run leaves no
backup set at all, and LATEST and the success metric stay where they were.
native mode needs no key. The chart's vmbackup sidecar gets the key from the
<fullname>-victoriametrics Secret (or victoriametrics.auth.existingSecret); a
sidecar you added to the plain manifests needs it too, as
deploy/k8s/optional/backup-cronjob.yaml shows.
Scheduling
Kubernetes — turn on backup in charts/fleeger-hub; a CronJob
runs the backup (VM_BACKUP_MODE=none) and a vmbackup sidecar in the VM pod
incrementally streams VictoriaMetrics to S3 (it shares the storage PVC, which a
separate Job can't). Minimal values:
backup:
enabled: true
schedule: "0 3 * * *"
s3Bucket: s3://fleeger-backups
s3EndpointUrl: https://fsn1.your-objectstorage.com
existingSecret: fleeger-s3 # keys: access-key-id, secret-access-key
retentionDays: "30"
vmbackup:
dst: s3://fleeger-backups/vm # a path DISTINCT from the DB sets
Save them as fleeger-backup.yaml and pass it with -f, to the install or
to a running release:
helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values \
-f fleeger-backup.yamlWithout --reset-then-reuse-values that upgrade resets every other value you
set at install (see the Helm chart reference's
Upgrade section).
compose / bare host — the runner ships as an opt-in profile; wrap one-shot runs in host cron:
cd deploy/compose
# one run (VM_BACKUP_MODE=native — captures VM over HTTP):
S3_BUCKET=s3://fleeger-backups S3_ENDPOINT_URL=https://… \
AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… RETENTION_DAYS=30 \
docker compose --profile backup run --rm backup
# host crontab — daily 03:00:
0 3 * * * cd /opt/fleeger/deploy/compose && S3_BUCKET=… S3_ENDPOINT_URL=… \
AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… RETENTION_DAYS=30 \
docker compose --profile backup run --rm backup >> /var/log/fleeger-backup.log 2>&1Suggested cadence: daily full backups. Config changes are already journalled as revisions, so sub-daily backups add little.
Retention
Two complementary knobs — use either or both:
- Script-side prune — set
RETENTION_DAYS=N(orbackup.retentionDays) andbackup.shdeletes S3 sets older than N days. Timestamped prefixes sort lexically, so it only ever removesYYYYMMDDThhmmssZsets (neverLATEST). - Bucket lifecycle — an S3/Object-Storage lifecycle rule expiring objects by age. Belt-and-braces if the backup job itself stops running.
The vmbackup sidecar keeps its own incremental history under vmbackup.dst;
prune that with a bucket lifecycle rule.
Signed backup sets
Write access to the backup bucket must not make anyone admin of the hub. A restore checks what a dump may define (no namespace users, no
REMOVE, …), but the rows it holds — users, roles, credentials, alert
channels — are restored as they are, so a set someone put into the bucket
would decide who can sign in. So backup.sh signs every set, and
scripts/restore.sh restores only signed ones.
What is signed. MANIFEST (manifest format 1) is text, one LF-ended line
each:
fleeger-backup-manifest 1
backup_ts=20261025T030000Z
surreal_ns=fleeger
surreal_dbs=auth inventory ingestion alerting control audit
vm_backup_mode=none
vm_snapshot=
key_id=3f2a9c0d81b4e6f7
part <sha256 hex> <size in bytes> surreal-alerting.surql.gz
part <sha256 hex> <size in bytes> surreal-audit.surql.gz
…
with one part line per object of the set. Its bytes, exactly
as stored, are signed with Ed25519; MANIFEST.sig holds the 64-byte
signature in base64 on one line. key_id is the first 16 hex digits of the
SHA-256 of the signing key's public key (DER), which is what both scripts
print. The parts are uploaded first, then MANIFEST, then MANIFEST.sig: a
set whose upload stopped part way has no signature.
What a restore checks, right after reading the set and before reading
anything in it (--dry-run too): MANIFEST.sig verifies under one of the
configured public keys; MANIFEST is in the format above; its backup_ts is
the timestamp being restored (a set copied to another prefix, to pass an old
set off as a new one, is refused; a --from-dir set may sit in a directory of
any name); every listed part is there with its size and SHA-256; and the set
holds nothing the manifest does not list. Nothing overrides a failure of any
of these, --allow-unsigned included. The signed backup_ts is printed with
the verification, and the audit log row of the restore records which key
verified it.
Unsigned sets — every set written by an older release, a set backup.sh wrote
without a key, and the pre-restore sets restore.sh itself writes — are
refused unless --allow-unsigned is given, which restores them under a loud
warning (and still checks the parts of a manifest in the new format).
--allow-unsigned also restores a signed set while no public key is
configured, unchecked. Use it only for a set you know where it came from.
A set written after signing was enabled that has no MANIFEST.sig had it
removed, or the backup ran without its key: that run's log has the ERROR
(no backup signing key). A MANIFEST with a non-empty key_id= and no
MANIFEST.sig was signed: its signature was removed, and the refusal says
so (and does not offer --allow-unsigned); restored with --allow-unsigned
anyway, the warning says the signature was removed. The dry-run summary, the
confirmation (with --force and RESTORE_CONFIRM=yes too) and the end of
the restore each say again whether the set's signature was verified, and
with which key.
The keys. The private key is a hub secret that only the backup job
mounts. Mounting is not the only way to it, though: on Kubernetes whoever can
read Secrets in the hub's namespace can read it, and so can whoever can read
the Helm release history there (a chart-made key is in the release's stored
manifest, itself a Secret in that namespace); with compose, whoever can read
.env or docker inspect the backup container. Hold those as tightly as
the key. The public key is not secret, and restores need it:
openssl genpkey -algorithm ed25519 -out backup-signing.pem # the private key
openssl pkey -in backup-signing.pem -pubout | sed -n 2p # its public keyA public key is the one base64 line of its PEM (60 characters, starting
MCowBQYDK2VwAyEA). restore.sh takes them from FLEEGER_BACKUP_VERIFY_KEYS
(space- or comma-separated), from the file FLEEGER_BACKUP_VERIFY_KEYS_FILE
(one per line or PEM blocks) and from --verify-key <key>, and refuses
anything that is not an Ed25519 public key before it downloads anything. A
private key given as one (its base64 starts MC4CAQAwBQYDK2VwBCIE, or MFECAQEw / MFMCAQEw in the v2 forms) is
refused as "the private key", without printing any of it: treat it as leaked
(below). Nothing of a value that is not a key is printed either. Verifying
needs OpenSSL 3 (openssl pkeyutl -rawin); the backup image has it. Before it
checks a set, restore.sh runs a known-answer test of the openssl it finds
(RFC 8032's test vector must verify, and not for another message): one that
cannot check Ed25519 is refused as such, rather than every good set reading
as tampered with. A MANIFEST.sig that is not a regular file is refused,
with or without --allow-unsigned.
A restore never holds the private key. Whoever can run commands where
the signing key is can sign a set of their own, so restores run elsewhere:
on Kubernetes in deploy/k8s/optional/restore-s3-pod.yaml,
which gets the S3 credentials and the public keys and no signing key (never
kubectl exec into the backup CronJob's pod); with compose, the command
below empties the key
(-e FLEEGER_BACKUP_SIGNING_KEY=). restore.sh itself takes
FLEEGER_BACKUP_SIGNING_KEY and FLEEGER_BACKUP_SIGNING_KEY_FILE out of its
environment first thing, so no command it starts gets them, refuses to read
the key's file (or anything under /etc/fleeger/backup-signing/) as
FLEEGER_BACKUP_VERIFY_KEYS_FILE, and warns in its banner when either was
set. backup.sh likewise takes them out of its environment before it runs
anything, and removes the PEM it writes for a one-line key right after
signing.
Kubernetes (chart): the chart generates the private key into
<fullname>-backup-signingon the first install withbackup.enabled(kept across upgrades andhelm uninstall; a client-side render refuses to generate it, see the Helm chart reference's GitOps section), or takesbackup.signing.existingSecret(keysigning-key, PEM). Only the backup CronJob mounts it. Helm cannot derive the public key, so the install notes print the command that does:kubectl -n fleeger get secret fleeger-fleeger-hub-backup-signing -o jsonpath='{.data.signing-key}' \ | base64 -d | openssl pkey -pubout | sed -n 2pPut it into
backup.signing.publicKeys(checked at render; the CronJob gets them asFLEEGER_BACKUP_VERIFY_KEYSand warns when its own key is not among them) and into the restore pod'sFLEEGER_BACKUP_VERIFY_KEYS.compose:
FLEEGER_BACKUP_SIGNING_KEY(the one base64 line of the private key's PEM) andFLEEGER_BACKUP_VERIFY_KEYSindeploy/compose/.env;.env.examplehas the commands. Only thebackupservice gets them; the restore command below runs in its image with the signing key emptied.By hand / host cron:
FLEEGER_BACKUP_SIGNING_KEY_FILE=<pem>(orFLEEGER_BACKUP_SIGNING_KEY) forbackup.sh.
Keep a copy of the public key off the hub, with the credential master key:
after losing the hub, a restore needs it to accept the sets (and without it,
only --allow-unsigned restores them, unchecked). The private key need not be
kept: a rebuilt hub can sign new sets with a new key.
Without a key, backup.sh still writes the set, unsigned, and logs an
ERROR: a backup never stops for want of a key. That run prunes nothing
(RETENTION_DAYS is skipped, and it says so), so signed sets are never
replaced by unsigned ones, and pushes no success metric: LATEST moves
to the set, but the dead-man's switch fires after its
window, so a hub without a key is alerted. Setting the key is therefore a
required step, for compose too (FLEEGER_BACKUP_SIGNING_KEY): until then the
backup alert fires. A bucket lifecycle rule still expires by age. An
unsigned run into a prefix a signed run already wrote (the same
BACKUP_TS) removes that run's MANIFEST.sig first. Both
FLEEGER_BACKUP_SIGNING_KEY and FLEEGER_BACKUP_SIGNING_KEY_FILE set is a
configuration error: the run stops before it dumps anything.
A key that is configured but cannot sign (not Ed25519, not readable, or a
key file that is not there: the CronJobs mount the Secret optional, so a
missing Secret lands here instead of keeping the Job from starting) also
leaves the set unsigned, and then fails the run after the upload: old sets
are not pruned and LATEST and the success metric stay where they were, so
the dead-man's switch fires. The values backup.sh writes into MANIFEST
(BACKUP_TS, SURREAL_NS, the database and snapshot names) are held to
what restore.sh reads, and a hash that cannot be taken stops the run
before anything is uploaded.
Rotating the key: add the new public key to the list next to the old one,
switch the private key (a new backup.signing.existingSecret, or the new value
in .env), and drop the old public key once no set signed with it is kept
(RETENTION_DAYS, or the bucket's lifecycle rule).
After a leak of the private key (or a suspicion of one), remove its public
key from every list at once (backup.signing.publicKeys, .env, the
restore pod, your off-hub copy) and switch to a new key: whoever has the
leaked key can sign a set that every restore still listing it accepts. The
sets signed with the leaked key are then refused like unsigned ones; restore
one only with --allow-unsigned, and only one you know where it came from
(one written before the leak, compared against a copy you trust).
What the signature does not cover:
- VictoriaMetrics history from
vmbackup(the chart's sidecar, or the onebackup-cronjob.yamldescribes): it is written to its own S3 path byvmbackupand read back byvmrestore, neither of which knows aboutMANIFEST. Whoever can write that path can change the metric history avmrestorebrings back (not accounts or settings). Keep it in a bucket or prefix the hub's restores do not write to, with its own credentials. Anativeordocker-execset's VictoriaMetrics file is a part of the set, and is checked like the dumps. - An old set that really was signed is accepted: a signature says which
backup job wrote a set, not that it is the newest. Someone with write
access to the bucket can delete the newer sets, or point
LATESTat an old one, so that a restore brings back older, genuine data (a rollback, to accounts or roles as they were then). Pick the timestamp to restore yourself, and look at the dry run'sbackup_ts. A set copied to another timestamp is refused (above). - Another hub's sets, if two hubs share a signing key: a set of one would verify on the other. Give every hub its own key (the chart generates one per release), and list only that hub's public keys in its restores.
The .fbk download is not signed: it is sealed (AES-256-GCM) with the
credential master key, so only a holder of that key can make one that
restores; write access to wherever it is stored is not enough.
Dead-man's switch
A backup that silently stops is worse than none. On every success backup.sh
pushes fleeger_backup_last_success_timestamp to VictoriaMetrics, and a built-in
alert rule (alert_rule:backup_stale, seeded by alerting migration 0003) fires
when it goes stale:
time() - max(last_over_time(fleeger_backup_last_success_timestamp[7d]) or vector(0)) > 129600
last_over_time([7d]) looks back across the whole backup interval (a bare gauge
would drop out of instant queries ~5 min after each push and false-fire between
daily backups). or vector(0) makes a missing metric read as "infinitely
stale" so silence means broken, not all clear. Default threshold 36h — one
missed daily cycle plus a buffer. It's an ordinary, always-on threshold rule:
attach a notification channel on the Alerts page to be paged, or mute it
there. Its threshold and for_secs can be tuned on its card there or
with PUT /api/alert-rules/backup_stale; its query, [7d] window included, is fixed. If you back up less often than weekly, disable backup_stale and create
your own threshold rule with the same query and a wider window.
Restoring
A restore is a maintenance step: every Fleeger service is stopped,
SurrealDB (and VictoriaMetrics and NATS) keep running, and each database in
the backup is replaced — emptied, then the backup's dump imported into it,
its migration state included. Everything written since the backup was taken is
gone. When the services start again, their db migrate brings a backup from
an older release up to date, exactly as an upgrade would, then defines again
whatever the database defines otherwise than its migrations . That also repairs
a hub restored through the Backup page of an older release, which put the backup's
older definitions back over the live ones.
Why a restore replaces instead of merging. A SurrealDB export is plain
INSERTstatements, one per table, each listing the table's rows. Imported over live data, anINSERTthat meets a record that already exists fails with already exists, and with it the whole statement — every row of that table, not only the clashing one. Restoring into running services also put back the backup's migration stamps over the newer live ones, so the migrations made since the backup ran again over data that had them already, or never ran for data that lacked them. Emptying each database first, with the services stopped, makes a restore exact, and the services migrate the restored data on start.
The audit log is the exception (The audit log): a
restore never empties the audit database. When the live one holds a row it
is kept as it is and the backup's is not imported; only an audit database
with no row (a new or rebuilt hub) takes the backup's, first, through a
transaction in which the database itself refuses to be emptied while it holds
a row. Each restore is journalled there: a proposed backup.restore row
after the pre-restore backup and before any service database is replaced
(when it cannot be written, none is), and a completed or failed row after
the last. inventory-svc backup restore also runs the audit database's
migrations first; scripts/restore.sh needs it migrated (start inventory-svc
of the release once): when the set's audit log is not imported and the live
audit database has no audit_event table, it refuses before writing
anything ("Is the audit database migrated?"), whatever the flags. After an
archive's audit log is imported, both tools define its append_only event
again right away, before anything else (the archive's schema and its
migration stamps are not trusted to hold it); inventory-svc's db migrate
and start define it again too.
Restore a set or .fbk of this release with this release's tools. A
scripts/restore.sh or inventory-svc from an older release knows nothing of the audit database: it would empty (remove) the live audit database like any
other and import the archive's over it, losing the hub's audit history.
There are two ways, one per kind of backup; both work the same:
| Backup | Restore with |
|---|---|
.fbk file (UI download, backup export) |
inventory-svc backup restore <file> |
S3 set (scripts/backup.sh) |
scripts/restore.sh <timestamp> |
Both, before writing anything:
read the backup's migration stamps of every database and the live database's, and refuse (nothing written) unless the backup fits the hub, by the rules of the migration runner — it takes a database's newest stamp (by
applied_at) as its state and runs every migration whose id sorts after it:- the live database must exist: one the namespace does not hold (a new, throwaway namespace) is refused, is never read (a read creates it), and cannot be in the pre-restore backup; restoring creates it;
- the live database must have stamps (a hub that never started is refused: start it once so it migrates, stop the services, re-run);
- the backup's newest migration must be one the live database has, and not sort after the live newest — a backup from a newer release is refused; restore it on that release or a newer one;
- every migration in the backup must be known to the live database;
- no live migration that sorts before the backup's newest may be missing from the backup (the runner would never apply it);
- no backup migration may sort after the backup's own newest (the runner would apply it a second time);
- with an auth database in the backup, the live auth database's newest migration must be the one the re-key file was built for (the restore tool must be the hub's own release).
Each database's line says which migrations run when the services start.
--no-version-checkrestores although a migration check refuses: for a throwaway namespace, and for putting back a pre-restore backup. It does not override the release check:--allow-other-releasedoes, and is meant only for putting back a pre-restore backup after restoring an archive whose auth database is at another release (which leaves the live one there); the restore says when its way back needs it.ask for confirmation: retype the archive's file name (or the S3 set's timestamp) on a terminal; without a terminal only
--forceorRESTORE_CONFIRM=yeswill do.--dry-runstops after the checks.write a pre-restore backup of every live database the restore replaces (without the metrics history) — the way back. It needs about as much free space as a backup without metrics. If it cannot be written whole, nothing is replaced. A database that does not exist yet is not in it (the
.fbkrestore and the set'sMANIFESTname them): putting the pre-restore backup back leaves such a database as the restore made it;REMOVE DATABASE <name>;removes it.
Before any of that, with the checks, every dump is read through as it will be
imported, and one that holds what a SurrealDB export of a Fleeger database does
not write is refused, whatever the flags: each statement must start with
OPTION IMPORT, a data statement (INSERT, CREATE, UPSERT, UPDATE,
RELATE, DELETE) or DEFINE of a table, field, index, event, function,
param, analyzer, access or user, the last two ON DATABASE. Outside strings
and quoted identifiers, no statement may hold DEFINE, REMOVE, ALTER or
ACCESS followed by a word anywhere but at its start, ON ROOT,
ON NAMESPACE, ON NS, or USE NS/USE DB; a comment is only a -- line
of its own between statements. The guard does not read code the way
SurrealDB's lexer does, so it refuses whatever the two could read differently:
outside strings and quoted identifiers only printable ASCII, space and tab (a
CR only at the end of a line); in a comment line only printable ASCII and tab
(SurrealDB also ends a comment at CR, U+0085, U+2028 and U+2029, and reads VT,
FF and Unicode spaces as whitespace); and no / (a regex literal can hold a
quote; no export writes a division). The refusal names the line (counted as
the dump is imported) and the reason; nothing is written. (A table named
root, ns or namespace, or a field named access, would be refused too:
none of Fleeger's is.) This guard is a pre-filter, which turns a crafted
archive away early and says why; what keeps a restore safe are the two
server-side steps below.
Then, database by database in archive (name) order, each is emptied
(REMOVE DATABASE, DEFINE DATABASE) and its dump imported without the
access key clause and without the rows of the migration lock
(schema_lock: a lock held when the backup was taken would otherwise come back
held, and db migrate would wait on it). The import runs as a user of that
database only: a database OWNER with a random name (fleeger_restore_…)
and password, defined right before the import. It is removed again after the
import however that went, and also when its own definition was sent but
answered with an error (the server may have applied it); inventory-svc's
restore also removes it if the task that replaces the database stops part way. The server itself refuses whatever
the dump would do outside that database: a user or access on the namespace or
root level (DEFINE USER OVERWRITE root ON ROOT … included), or a write to
another database (USE DB auth; …). Such an import fails, and the restore
stops there. If the user cannot be made at all (restore.sh reads its name
and password from /dev/urandom), the database is left emptied and not
imported, and the restore ends with the usual PARTIALLY RESTORED summary and
the way back.
A removal can still fail, and a restore that is killed (SIGKILL, a node
lost) cannot remove its user at all. So:
- The check after each import (below) removes this run's own user if it is
still there, and reports it as a failed clean-up, not as
SECURITY. If it cannot remove it either, it says so, with the statement to run, and the restore goes on. Any other database user, one named like a restore user included, is still an extra: every database the check looks at was emptied before its import, so only the import can have defined it. - As a restore starts, and again once its last import is done, every service
database of the namespace (with
restore.sh, every database) is searched for users namedfleeger_restore_followed by 16 hex digits. Each is removed, and reported as a clean-up: left by an earlier restore that was stopped. One that cannot be removed, or a database that cannot be searched, is reported with the statement to run; none of it stops the restore. Such a user is anOWNERof its database only, with a random password no one was given.
After every import that was sent, however it went, every database restored
so far is checked again, against the server's own text of its definitions:
INFO FOR DB, INFO FOR TABLE for each of its tables (events, fields with
their VALUE/DEFAULT/ASSERT, indexes, views), and the accesses and users
of the namespace and root, as the server prints what it parsed (keywords in
capitals, no comments, nested statements spelled out), not as the dump spelled
it. Then:
- What no migration defines is removed: in the database, an access other
than auth's
userand every database user (DEFINE USER … ON DATABASE); on the namespace and root levels, every access and user that was not there before the first import. The restore says so loudly (SECURITY: …). A database user an operator defined by hand in a service database is removed too: it is in the backup, but no migration defines it. Define it again after the restore (theSECURITYline names it). - A namespace or root access or user that was there before and is not as it
was stops the restore, with what it read before and how to define it again
(
OVERWRITE, with its password or key). The server prints[REDACTED]for a password hash or a signing key, so this sees a changed role or duration, not a changed password: what keeps the import from changing one is that it runs as a database user. - A definition whose body would define, remove or change something when it
runs (an event, function, field, view, param or access holding
DEFINE,REMOVE,ALTER,USE,ACCESS,ROOT,NAMESPACEorNSas a word after its own start, in any case), or one of a kind no Fleeger export holds, stops the restore before anything runs it (not the recount, not the next import). It names each, with theREMOVEstatement to run as root in that database; do not start the services until it is removed, or restore an archive you trust. None of Fleeger's own definitions holds such a word; a future migration whose definition did would stop every restore until this check knows it.
If the definitions cannot be listed or removed, the restore stops after that database and says what to check and remove by hand; if they cannot be listed before the first import, or the import user cannot be defined, that database is left emptied and not imported.
Right after that check, for auth, whatever its import's outcome, the auth
database's user access is re-defined with a fresh signing key, so everyone
is signed out (below); for ingestion, once the
check found nothing to stop for, the resource counts per location are
recomputed from the restored resources
(services/ingestion-svc/migrations/after-restore/recount_resources.surql; the
recount writes, which runs the database's events); if that fails, the restore
goes on, says what to run, and exits non-zero when it is done. Once the last
import is done, every restored database is checked once more, and auth is
re-keyed again after that, so the access the restore ends with is this
release's, with a key of the server's. A metrics history in the backup is
imported last: it only adds samples to what VictoriaMetrics holds, and putting
the pre-restore backup back does not take them out.
Whoever can write the backup bucket can already plant data rows (an auth
account with a password they know, say), which no check of definitions sees:
treat write access to the bucket, and to any .fbk you restore, as
admin-equivalent.
The first failure stops the restore. It says which databases were replaced,
which one failed (emptied, and maybe partly imported), which were not
attempted, and how to put back the pre-restore backup. Ctrl-C, a hangup or
SIGTERM stops it before the next database (and before the metrics history),
never in the middle of one (the database under way is finished, with its
re-key, recount and checks). There is no automatic rollback.
When it is done, start the services; their db migrate runs the migrations
named in the checks, and everyone signs in again.
Restoring a .fbk: inventory-svc backup restore
The restore runs in the inventory-svc image of the hub's release, which holds
the master key and the SurrealDB credentials. With compose, from
deploy/compose, with the archive in ./restore:
docker compose stop gateway-svc auth-svc inventory-svc ingestion-svc \
alerting-svc control-svc core-agent
mkdir -p restore && cp ~/fleeger-backup-20260817T072200Z.fbk restore/
# Check it against the live databases (writes nothing):
docker compose run --rm --no-deps --entrypoint /usr/local/bin/service \
--user "$(id -u):$(id -g)" -v "$PWD/restore:/restore" -e TMPDIR=/restore \
inventory-svc backup restore --dry-run /restore/fleeger-backup-20260817T072200Z.fbk
# Restore it (asks you to retype the file name):
docker compose run --rm --no-deps --entrypoint /usr/local/bin/service \
--user "$(id -u):$(id -g)" -v "$PWD/restore:/restore" -e TMPDIR=/restore \
inventory-svc backup restore /restore/fleeger-backup-20260817T072200Z.fbk
docker compose up -d--no-deps keeps the stopped services stopped; --user lets the command
write the pre-restore backup into ./restore as you. -e TMPDIR=/restore
puts the decoded temp copy of the archive (below) on that bind mount, next
to the archive, so ./restore needs free space for the archive, about three
quarters of it again for the copy, and the pre-restore backup. Without it the
copy goes to the run container's /tmp, its writable layer, which is on the
disk of Docker's data root (docker info -f '{{.DockerRootDir}}'): that
disk then needs the free space of about three quarters of the archive
instead. On Kubernetes, scale the
Fleeger deployments to zero and use the pod in
deploy/k8s/optional/restore-pod.yaml,
whose header has the steps (kubectl cp the archive in, kubectl exec -it the
same command, kubectl cp the pre-restore backup out). As it is, it fits a hub
deployed from deploy/k8s (the Secrets fleeger-credential-key and
fleeger-db, the ConfigMap fleeger-config, the label
app.kubernetes.io/instance: fleeger the network policies select on), once its
image is set to the inventory-svc image the hub runs. For a Helm install,
change the image to <image.registry>/inventory-svc:<image.tag or the chart's appVersion>, the Secrets to <fullname>-credential-key (or
credential.existingSecret) and <fullname>-db (or
surrealdb.existingSecret), the ConfigMap to <fullname>-config, and the
instance label to the release name. During a credential-key rotation the pod
reads FLEEGER_CREDENTIAL_KEYS_RETIRED from the optional Secret
fleeger-credential-keys-retired, as the hub's services from deploy/k8s do; for a Helm install, point it at credential.retiredKeysSecret
instead, or drop the entry. The pod sets TMPDIR=/restore, so the decoded archive
lands on its own claim, fleeger-restore, not in the running inventory-svc
pod's /tmp (a restore no longer goes through that pod, which is why the chart and deploy/k8s put no size on its /tmp for it): size the
claim for the archive, about three quarters of it again for the decoded copy,
and the pre-restore backup.
Options: --dry-run, --force (or RESTORE_CONFIRM=yes), --no-version-check,
--allow-other-release, and --pre-restore-file <path> (default:
fleeger-pre-restore-<time>.fbk next to the archive; it must not exist yet).
The archive is always a local file; restoring straight from a remote source is not supported yet.
A Ctrl-C, hangup or SIGTERM while the pre-restore backup is written stops
the restore with nothing replaced and the unfinished file removed. The
connection to the inventory database (for the secrets check below) is opened
only after the pre-restore backup is written, so the checks and a dry run
create nothing, also in a namespace that does not exist. The archive is read as it is used, and
kept decoded in a temp file under TMPDIR (about three quarters of its size;
/tmp when unset; see above for where that is with compose and on
Kubernetes); nothing of it is held whole in memory. The temp file is removed
when the restore ends, however it ends; a disk that fills while it is written
stops the restore with nothing imported. Its authentication tag is checked over
the whole file before anything is checked or written; an archive that does not
decrypt, a framing that does not hold together, or a part this release does
not restore, writes nothing. All of it counts against the [backup] timeout_secs budget above; the re-key, the emptying and the recount of each
database have 30 seconds each of their own. An import cut off by the budget
may still be under way on the server, so the restore stops there without the
re-key and the checks (they would not be final): it says the outcome is
unknown, that the services must not start, and the command that runs the
restore again once the database server is idle (with a larger
timeout_secs), which empties that database again and does them.
The databases come back as they were dumped, so their secrets come back under
the key they were sealed with, which can be a retired one (see
below). Once the
import is over, metrics included, the restore re-encrypts under the current
FLEEGER_CREDENTIAL_KEY every credential in inventory-svc's database still
under a retired key, the way inventory-svc's service rekey does. It also
counts the notification channels still under another key, but does not
rewrite those, since they belong to alerting-svc. When any secret is left off
the current key, or one of these checks could not run, the restore still
succeeds, and prints how many and in which service to run service rekey.
Restoring an S3 set: scripts/restore.sh
scripts/restore.sh <timestamp> downloads a set and restores SurrealDB
automatically, and (for native-mode sets) imports the VM history too. A
docker-exec set's VM snapshot tarball is saved to RESTORE_DIR (default: the
directory you run the script from) for the manual step
below; everything else is downloaded to a temp
directory that is removed when the script exits.
The script refuses to replace data quietly:
- it prints the target endpoint, namespace and backup set before anything is downloaded or written;
--dry-rundownloads the set, runs the checks above, says what it would do and exits, writing nothing;- it refuses a set with more than one VictoriaMetrics snapshot tarball rather
than restore one of them and drop the rest, and one whose
vm-*.tar.gzis not a file (below); - it refuses a set whose signature does not verify under the configured public
keys, or that is not as its signed
MANIFESTlists it, and an unsigned one without--allow-unsigned(Signed backup sets); and it says again, at the confirmation and at the end, whether the signature was verified; - on a terminal it makes you retype the backup timestamp (a reflexive
ywon't do it); - with no terminal — a CronJob, CI,
kubectl execwithout-t— it refuses outright unless you passed--forceorRESTORE_CONFIRM=yes. That is what stops the backup image (which already carries the S3 credentials and the productionSURREAL_*env) from restoring by accident.
The pre-restore backup is a set like the ones backup.sh writes, but
unsigned (the restore holds no signing key; it is the hub's own data, written
to the local disk, not the bucket):
RESTORE_DIR/pre-restore-<time>/ with a surreal-<db>.surql.gz per database
and a MANIFEST. RESTORE_DIR must be an existing, writable directory; it is
checked before the set is downloaded, also on a dry run. --from-dir <dir>
restores a set from a local directory instead of S3, which is how a
pre-restore set is put back (the summary prints the exact command, with
--allow-other-release when it is needed). A --from-dir set is read where
it is, and a symlink in it counts as the file it points to: the checks read
the file through the link, and so does the import, later. Keep the directory
(and whatever its links point to) where only you can write until the restore
is done, or copy it first (cp -rL).
scripts/restore.sh --from-dir ./pre-restore-20261016T230431Z --no-version-check --allow-unsignedWith compose, run it in the backup image, from deploy/compose, with a
directory for the pre-restore set (the image's / is not writable for its
user, and the CLIs want a writable home):
mkdir -p restore
docker compose --profile backup run --rm --no-deps \
--entrypoint /usr/local/bin/restore.sh --user "$(id -u):$(id -g)" \
-e FLEEGER_BACKUP_SIGNING_KEY= \
-e HOME=/tmp -e RESTORE_DIR=/restore -w /restore -v "$PWD/restore:/restore" \
backup --dry-run 20260817T072200Z
# then the same without --dry-run-e FLEEGER_BACKUP_SIGNING_KEY= keeps the backup service's private key out
of the restore's container (Signed backup sets);
restore.sh checks the set with the service's FLEEGER_BACKUP_VERIFY_KEYS.
On Kubernetes, restore from
deploy/k8s/optional/restore-s3-pod.yaml,
a pod of the backup image with the S3 credentials, the SurrealDB password and
the public keys, and no signing key; its header has the steps (and what to
change for a Helm install). Not from the backup CronJob's pod, which mounts the
private key. RESTORE_DIR is its claim; kubectl cp the pre-restore set out
before you delete it.
While it replaces the databases, the script stops only between two of them on
SIGHUP, SIGINT (Ctrl-C) or SIGTERM, and ignores SIGPIPE. Each
surreal command it runs ignores those three signals (it is started from a
subshell that ignores them), and runs in a session of its own where setsid
is installed; so neither a Ctrl-C at the terminal nor a signal to the whole
process group ends an import part way. That the SurrealDB 3.2.4 CLI keeps the
signals it inherits ignored was checked on the real CLI, not inferred: an
import of 150,000 rows sent SIGINT, SIGHUP and SIGTERM to its whole
process group finished all of them and exited 0 when started that way, and
died at about 88,000 rows with exit code 130 when not.
Their output and errors go to files and are
printed after each database (output to stdout, errors to stderr), and the
script carries on when writing its own output fails. This is meant for a
dropped kubectl exec or SSH session (a hangup, and output that is gone) and
a plain kill. SIGKILL cannot be caught: a pod or container killed outright
(kill -9, docker kill, a pod deleted past its grace period) can stop it
between the auth import and its re-key; if that happens, run the re-key by hand
(below).
The re-key file is rekey_access.surql: the script uses
FLEEGER_REKEY_FILE if set, else
services/auth-svc/migrations/after-restore/rekey_access.surql next to the
script in a checkout, else /usr/local/share/fleeger/rekey_access.surql (the
backup image). A set with an auth dump is not restored without one. The
banner prints which file it uses, the auth migration the file was built for
(its -- built-for: line) and its SHA-256. Run the backup image of the hub's
own release (backup.image, or a rebuilt fleeger/backup:local), or point
FLEEGER_REKEY_FILE at that release's file. The recount file is found the
same way (FLEEGER_RECOUNT_FILE, then
services/ingestion-svc/migrations/after-restore/recount_resources.surql, then
/usr/local/share/fleeger/recount_resources.surql); a set with an ingestion
dump is not restored without one.
export S3_BUCKET=s3://fleeger-backups S3_ENDPOINT_URL=https://fsn1.your-objectstorage.com
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
export FLEEGER_BACKUP_VERIFY_KEYS=MCowBQYDK2VwAyEA... # the backup signing key's public key
# Always look first — no confirmation needed, nothing is written:
scripts/restore.sh --dry-run 20260817T072200Z
# Then, with the services stopped, restore (prompts on a terminal):
scripts/restore.sh 20260817T072200Z
# Non-interactive (automation that genuinely means it):
RESTORE_CONFIRM=yes scripts/restore.sh 20260817T072200Zjust restore <ts> [options] forwards to the same script with its options
(just restore <ts> --dry-run), so it prompts too.
SurrealDB restore details
SurrealDB exports tables alphabetically, which can place a graph-edge
(INSERT RELATION) statement before the records it links — e.g. the roles
user→role edges before the user rows. A naive import then fails with
"record 'user:…' does not exist". Both restore paths rewrite each dump so
every INSERT RELATION runs last, after its endpoints exist, and leave
out the migration lock's rows, the same way (fleeger_shared::restore and the
awk pass in the script are kept identical, and tested to be).
To verify a backup without touching live data, restore it into a throwaway
namespace by setting SURREAL_NS. That namespace has no migration state to
check against, so pass --no-version-check:
SURREAL_NS=restoretest scripts/restore.sh --no-version-check 20260817T072200Z
# then check row counts match the live namespace, and REMOVE NAMESPACE restoretestRe-keying after a restore
Both restore paths re-key the auth database on their own. When that step
failed (the restore says so), or after restoring the auth database by any
other means, re-define the user access by hand before anyone signs in, with
the rekey_access.surql of the release the hub runs. This signs everyone
out. INFO FOR DB does not show the key, so it cannot tell whether this is
needed; when in doubt, run it.
Compose (in deploy/compose; the backup image carries the file):
docker compose --profile backup run --rm --entrypoint sh backup -c \
'surreal import --endpoint "$SURREAL_ENDPOINT" --username "$SURREAL_USER" \
--namespace "$SURREAL_NS" --database auth /usr/local/share/fleeger/rekey_access.surql'Kubernetes, from a checkout of the hub's release (the data namespace is
fleeger). The SurrealDB pod has SURREAL_USER and SURREAL_PASS in its
environment, and surreal import reads both from there when no
--username/--password is given:
kubectl -n <ns> exec -i deploy/<surrealdb> -- /surreal import \
--endpoint http://localhost:8000 \
--namespace fleeger --database auth /dev/stdin \
< services/auth-svc/migrations/after-restore/rekey_access.surqlThe gateway's auth cache may keep accepting a token it already checked for up
to 5 seconds afterwards. Upgrading to a release with auth migration
0013_rekey_access, and again to one with 0015_drop_signup,
re-keys every install once in the same way.
The core agent's revocation after a restore
The auth database holds the revocation ledger, and with it the revocation of
the bundled core agent's earlier, wider NATS JWTs that ./gen-auth.sh fix-perms core-agent records. A backup taken before that ran does not
hold it: after restoring one, the next push of the SPOKES account (auth-svc's
reconcile, within 10 s) leaves it out, and the old core-agent.creds connects
again with its old grants. Run ./gen-auth.sh fix-perms core-agent again
after such a restore (Agent credentials & revocation); it re-issues the
core agent's credential and records the revocation once more. On a hub
without the core agent, or one that never ran it, there is nothing to do.
Restoring a backup from before a key rotation finished
A backup holds the secrets as they were stored when it was taken. A
key rotation and its rekey runs
re-encrypt the live databases only, never a backup already taken. Any backup
taken before both rekeys were confirmed can hold old-key secrets, including one
taken after the new key became primary: it is sealed under the new key, but
the secrets inside are still the old key's until the rekey reaches them. The
same goes for alert-channel secrets after a rotation done before alerting-svc
had a rekey: they stayed on the old key until you rekeyed or re-entered
them.
- a
.fbkis sealed as a whole under the key that was primary when it was taken.backup restoreopens it with any key inventory-svc holds, current or retired, so it refuses with could not decrypt the backup — wrong master key when that key is no longer supplied; - the databases inside a
.fbkor an S3 set are imported as they are, so the stored credentials (inventory) and notification-channel secrets (alerting) come back as the old key's ciphertexts, both on rows the backup restores over and on rows it brings back.backup restorethen re-encrypts the credentials under the current key where the old key is supplied, and reports what it left, with the notification channels still under the old key.scripts/restore.shnever decrypts anything, so it succeeds either way and reports nothing; without the old key those secrets then fail at their next use.
To restore such a backup:
- Supply the old key again as retired, to both inventory-svc and
alerting-svc. With compose, set
FLEEGER_CREDENTIAL_KEYS_RETIREDin.env(comma-separated for several, e.g. a backup from two rotations ago) and rundocker compose up -d. With Helm, put it in the Secret named bycredential.retiredKeysSecret(keyretired-keys) and upgrade with--reset-then-reuse-values --set credential.retiredKeysSecret=<name>. With the raw manifests indeploy/k8s/, create the Secretfleeger-credential-keys-retired(keyretired-keys) andkubectl -n fleeger rollout restart deploy/inventory-svc deploy/alerting-svc(Secrets the hub needs). The current key stays asFLEEGER_CREDENTIAL_KEY. - Restore the backup (
inventory-svc backup restore <file>for a.fbk, orscripts/restore.sh <ts>), with the services stopped, then start them again. - Run
service rekeyin inventory-svc and in alerting-svc, as in step 3 of the rotation, until a second run of each reports0 ... re-encrypted. A run that stops with cannot decrypt … (is the retired key present?) names a secret sealed under a key you have not supplied. Afterbackup restorethe inventory-svc credentials are usually done already: the restore's warning names the services that still need it. - Only then drop the old key from the retired keys again. Keep your own copy of it while you keep any backup taken before both rekeys were confirmed.
If the old key is gone, a .fbk sealed under it cannot be restored. An S3 set,
or a .fbk sealed under the new key, can still be, but every credential and
notification channel it restores under the old key must be re-entered. Until
all of them are, rekey cannot finish: it aborts on the first secret it cannot
decrypt, so the next rotation's rekey stops there too.
VictoriaMetrics restore
native-mode sets (compose runner, or the UI download with metrics): the set
contains vm-native.bin.gz, and restore.sh imports it over
/api/v1/import/native automatically — no downtime, no data-dir swap.
docker-exec snapshot tarballs: the tarball is a complete data directory.
restore.sh saves it as vm-<snapshot>.tar.gz in RESTORE_DIR — an existing,
writable directory you choose, by default the one you run the script from — and
prints its full path. It saves the tarball before importing anything, so if it
cannot (the directory is missing or read-only, as / is in the backup image),
it stops with nothing imported and you re-run with a different RESTORE_DIR.
The tarball can be several GB, so pick a directory with room for it:
RESTORE_DIR=/srv/restore scripts/restore.sh 20260817T072200Zbackup.sh writes one tarball per set. A set holding more than one (two runs
into the same timestamp, files copied in by hand), or a vm-*.tar.gz that is
not a file (a directory, a dangling link), is refused before anything runs,
also on a dry run and whatever flags are given: which snapshot belongs to the
dumps cannot be told from the set. The refusal names the tarballs and the
snapshot the set's MANIFEST names; copy the set into a directory, keep the
tarball you mean (or none), and restore that with --from-dir.
Then restore it deliberately:
- Stop VictoriaMetrics:
docker compose stop victoriametrics - Extract and replace the data dir contents, using the path
restore.shprinted:mkdir -p /tmp/vm-restore tar xzf /srv/restore/vm-<snapshot>.tar.gz -C /tmp/vm-restore # replace the vm-data volume's contents with /tmp/vm-restore/<snapshot>/ - Start it again:
docker compose start victoriametrics
Kubernetes vmbackup sidecar (backup.vmbackup): restore with vmrestore
from vmbackup.dst into the VM storage path while VM is stopped — see the
VictoriaMetrics vmrestore docs.
Metrics history is the least critical tier (config and audit live in SurrealDB); losing the most recent window between backups is acceptable.
Full disaster recovery — from nothing
Rebuilding a hub from a total loss, in order:
- Restore the secrets you kept off-box —
FLEEGER_CREDENTIAL_KEY(into the Secret / env) and the NATS auth material (deploy/nats/auth/). Without the key the credential ciphertext is dead weight; without the NATS material agents can't reconnect. Have the backup signing key's public key at hand for step 4 (FLEEGER_BACKUP_VERIFY_KEYS, Signed backup sets); the rebuilt hub may sign its new sets with a new private key. If the backup you will restore was taken before a key rotation's rekeys were confirmed, also supply the old key as retired, and rekey after step 5 (above). - Bring up the hub (compose or Helm) with that key in place, so the schema migrations run and the services boot. The restore checks the backup against the migration state this leaves, so use the release the backup was taken on or a newer one.
- Stop the Fleeger services (SurrealDB, VictoriaMetrics and NATS keep running; the commands are above).
- Restore SurrealDB from the latest set (
scripts/restore.sh <ts>), or the.fbk(inventory-svc backup restore <file>). Either signs everyone out (above). - Start the services (
docker compose up -d, or scale the deployments back up): theirdb migratebrings an older backup up to date. Log in again. - Restore VictoriaMetrics per its mode above (optional — history only).
- Verify: log in, confirm Locations/Resources/credentials resolve, alert
rules are present, the audit log (Audit page;
audit_eventin theauditdatabase) is intact and holds the restore'sbackup.restorerows, and agents reconnect and resume reporting.
Restore drill checklist
Do this periodically, not just after a disaster:
- Run
scripts/restore.sh --no-version-check <ts>withSURREAL_NS=restoretest(or restore a.fbkwithbackup restoreon a scratch instance sharing the master key). The namespace is the only thing keeping this off live data, and--no-version-check(the databases of a throwaway namespace do not exist yet, so there is no migration state to check against; they are not read, and the restore creates them) turns off the migration checks, so read the banner the script prints and check the endpoint and namespace before confirming —--dry-runfirst if you want to see the set without touching anything. The live services may keep running for this one: nothing of the live namespace is written. Runscripts/restore.sh --dry-run <ts>against the live namespace too: it says whether the set would restore on this release, and whether its signature verifies with the public key you keep off the hub (pass that copy with--verify-key). - Confirm row counts in
restoretestmatch live foruser,location,resource,config_revision,checkpoint, alert rules, andaudit_eventin theauditdatabase. - Confirm the VM history imports (native) or the tarball holds a non-empty snapshot dir (docker-exec).
REMOVE NAMESPACE restoretest.- Confirm the Backup dead-man's switch alert is present and not firing.
A restore that has been drilled is the only kind you can trust at 3am.