// Deployment-Referenz
Secrets the hub needs
The base manifests deliberately contain no secrets. Create these five in the
fleeger namespace before (or alongside) applying the base. Nothing here is
generated for you the way Helm does it — that trade is the price of plain
manifests, and it keeps credentials out of git.
| Secret | Keys | What it is | How to create |
|---|---|---|---|
fleeger-nats-auth |
*.creds, *.nk, resolver.conf, SPOKES.jwt |
NATS decentralized-auth material | kubectl create secret … --from-file (below) |
fleeger-credential-key |
credential-key |
Master key for stored credentials | secrets.example.yaml or kubectl create secret |
fleeger-db |
db-password |
SurrealDB root password | secrets.example.yaml or kubectl create secret |
fleeger-internal |
internal-token |
Inter-service shared secret | secrets.example.yaml or kubectl create secret |
fleeger-victoriametrics |
delete-key, snapshot-key |
VictoriaMetrics' admin-API keys | secrets.example.yaml or kubectl create secret |
1. NATS auth material (fleeger-nats-auth) — required
This is the one Fleeger cannot generate on the cluster (it needs nsc). Run
the generator once, locally:
deploy/nats/gen-auth.sh # writes gitignored files under deploy/nats/auth/Regenerating rotates all NATS credentials — every agent must re-onboard. See Agent credentials & revocation.
Then load every piece into one Secret. Include all the *.creds — a service
whose cred key is missing from the Secret will fail to start (the volume
projection references it by key):
kubectl -n fleeger create secret generic fleeger-nats-auth \
--from-file=ingestion.creds=deploy/nats/auth/ingestion.creds \
--from-file=gateway.creds=deploy/nats/auth/gateway.creds \
--from-file=alerting.creds=deploy/nats/auth/alerting.creds \
--from-file=inventory.creds=deploy/nats/auth/inventory.creds \
--from-file=control.creds=deploy/nats/auth/control.creds \
--from-file=core-agent.creds=deploy/nats/auth/core-agent.creds \
--from-file=sys.creds=deploy/nats/auth/sys.creds \
--from-file=operator.nk=deploy/nats/auth/operator.nk \
--from-file=spokes-account.nk=deploy/nats/auth/spokes-account.nk \
--from-file=resolver.conf=deploy/nats/auth/resolver.conf \
--from-file=SPOKES.jwt="$(ls deploy/nats/auth/nsc-data/nats/nsc/stores/fleeger/accounts/SPOKES/SPOKES.jwt)"core-agent.creds is only needed if you enable the bundled cluster-monitoring
agent (Monitoring the hub's own cluster (bundled core-agent)); it's
harmless to include always, so the command above does.
2. The other four — required
Two ways; pick one.
a) From the example file (good for GitOps once encrypted):
cp deploy/k8s/secrets/secrets.example.yaml deploy/k8s/secrets/secrets.yaml
# edit secrets.yaml — replace every REPLACE-ME with a generated value
kubectl -n fleeger apply -f deploy/k8s/secrets/secrets.yamlsecrets.yaml is gitignored. Do not commit plaintext secrets; if you keep them
in git, seal them first (e.g. Sealed Secrets, SOPS, External Secrets).
b) Imperatively (nothing written to disk):
head -c 32 /dev/urandom | base64 | tr -d '\n' | \
kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | \
kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | \
kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=/dev/stdin
d=$(mktemp -d) && openssl rand -hex 32 | tr -d '\n' >"$d/delete-key" &&
openssl rand -hex 32 | tr -d '\n' >"$d/snapshot-key" &&
kubectl -n fleeger create secret generic fleeger-victoriametrics --from-file="$d"
rm -rf "$d"fleeger-victoriametrics holds two different hex keys. VictoriaMetrics
requires delete-key for every series delete, and only ingestion-svc holds it;
without it, anyone who can reach VictoriaMetrics deletes every series with one
request. snapshot-key guards the snapshot and other maintenance endpoints;
only a vmbackup sidecar holds it, if you add one
(optional/backup-cronjob.yaml shows how).
Hex, because the snapshot key travels in the sidecar's snapshot URLs. No
surrounding whitespace or trailing newline (hence tr -d '\n'): VictoriaMetrics
does not trim its keys, so a padded one never matches.
Back up
fleeger-credential-key. It encrypts every stored infra credential and alert-channel config; lose it and those become unrecoverable.
3. Rotating the credential key (optional)
inventory-svc and alerting-svc (and the optional
restore pod) read
FLEEGER_CREDENTIAL_KEYS_RETIRED from a sixth Secret,
fleeger-credential-keys-retired (key retired-keys): the previous key(s),
decrypt-only, comma-separated if there are several. The reference is
optional: true, so the Secret exists only while a rotation is under way;
without it the variable is unset and nothing changes. This is the manifest
form of the compose walkthrough in
Deploying Fleeger
and the chart's credential.retiredKeysSecret.
Rotate the same way you created the Secrets in section 2. If you keep them
declaratively (path a: secrets.yaml, or its sealed / SOPS / GitOps source),
make every change below in that source and apply it from there; do not mix
in the imperative kubectl commands. A key changed only in the cluster is
reverted by the next apply or sync of the source, and once the retired-keys
Secret is gone, everything already re-encrypted under the new key is
unreadable.
The services do not restart on their own when a Secret changes (checksum annotations cover only NATS and the edge), so every step that
changes a Secret ends with a rollout restart.
Copy the current key into the retired-keys Secret, before you replace it. Nothing on this path refuses a key change the way the chart's
credential.allowKeyChangedoes, so this order is all that keeps the old key reachable.Declaratively: uncomment the
fleeger-credential-keys-retiredblock at the end ofsecrets.yaml, setretired-keysto the currentcredential-key(appended with a comma if the block already lists keys), and apply. The list is comma-separated with no spaces: the check below matches each entry exactly, sok0, k1does not containk1. Imperatively:cur=$(kubectl -n fleeger get secret fleeger-credential-key -o jsonpath='{.data.credential-key}' | base64 -d) [ -n "$cur" ] && printf '%s' "$cur" | \ kubectl -n fleeger create secret generic fleeger-credential-keys-retired --from-file=retired-keys=/dev/stdinIf that fails with
AlreadyExists(an earlier rotation's Secret is still there), add the key to its comma-separated list instead of creating a new one:prev=$(kubectl -n fleeger get secret fleeger-credential-keys-retired -o jsonpath='{.data.retired-keys}' | base64 -d) [ -n "$cur" ] && [ -n "$prev" ] && printf '%s,%s' "$prev" "$cur" | \ kubectl -n fleeger create secret generic fleeger-credential-keys-retired --from-file=retired-keys=/dev/stdin \ --dry-run=client -o yaml | kubectl -n fleeger apply -f -Either way, check that the current key really is in the retired Secret before you go on; do not replace the key unless this prints
OK:cur=$(kubectl -n fleeger get secret fleeger-credential-key -o jsonpath='{.data.credential-key}' | base64 -d) ret=$(kubectl -n fleeger get secret fleeger-credential-keys-retired -o jsonpath='{.data.retired-keys}' | base64 -d) if [ -n "$cur" ] && case ",$ret," in *",$cur,"*) true ;; *) false ;; esac then echo OK; else echo 'MISSING: the current key is not in retired-keys'; fiKeep a copy of the old key off-box as well (step 5).
Generate the new key into a file in a fresh temporary directory, readable only by you:
d=$(mktemp -d) && (umask 077 && head -c 32 /dev/urandom | base64 | tr -d '\n' >"$d/new-credential-key")Back it up off-box now, before it reaches the cluster: once step 4 has re-encrypted everything under it and the old key is retired, losing it makes every stored credential and alert-channel config unrecoverable. Then put it in place. Declaratively: set
credential-keyinsecrets.yaml(or its sealed source) to the new key and apply. Imperatively:kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key="$d/new-credential-key" \ --dry-run=client -o yaml | kubectl -n fleeger apply -f -Once the key is backed up (and in your source), remove the directory:
rm -rf "$d"Restart both services so they read the new key and the retired one, and wait for the rollouts:
kubectl -n fleeger rollout restart deploy/inventory-svc deploy/alerting-svc kubectl -n fleeger rollout status deploy/inventory-svc kubectl -n fleeger rollout status deploy/alerting-svcThen check that the pods actually see the retired key(s). The reference is optional, so a misspelt Secret name or key leaves the variable unset without any error, and nothing would show it until
rekeyfails:kubectl -n fleeger exec deploy/inventory-svc -- sh -c 'test -n "$FLEEGER_CREDENTIAL_KEYS_RETIRED"' && echo inventory-svc OK || echo inventory-svc MISSING kubectl -n fleeger exec deploy/alerting-svc -- sh -c 'test -n "$FLEEGER_CREDENTIAL_KEYS_RETIRED"' && echo alerting-svc OK || echo alerting-svc MISSINGRe-encrypt everything under the new key, in both services:
kubectl -n fleeger exec deploy/inventory-svc -- /usr/local/bin/service rekey kubectl -n fleeger exec deploy/alerting-svc -- /usr/local/bin/service rekeyEach run is journalled in the hub-wide audit log (
secrets.rekey, never a key) before it re-encrypts anything, and refuses, changing nothing, when the audit log cannot take the row.Then run both again to confirm. Step 3 waited for the rollouts, so no pod still writes under the old key; the second run is the proof that nothing is left under it, and re-encrypts what the first one did not finish (it stops on a row that keeps being edited meanwhile). Repeat until each reports
0 ... re-encrypted. Only then delete the retired-keys Secret (declaratively: remove the block fromsecrets.yamlor your source too, or the next apply or sync brings it back) and restart again:kubectl -n fleeger delete secret fleeger-credential-keys-retired kubectl -n fleeger rollout restart deploy/inventory-svc deploy/alerting-svcKeep the old key, off-box, for as long as you keep any backup taken before both rekeys were confirmed: restoring one needs it back in
fleeger-credential-keys-retired, a restart, and bothrekeyruns again (Backup & disaster recovery).