Zum Inhalt springen
Produkte

// Deployment-Referenz

Secrets the hub needs

The base manifests deliberately contain no secrets. Create these five in the fleeger namespace before (or alongside) applying the base. Nothing here is generated for you the way Helm does it — that trade is the price of plain manifests, and it keeps credentials out of git.

Secret Keys What it is How to create
fleeger-nats-auth *.creds, *.nk, resolver.conf, SPOKES.jwt NATS decentralized-auth material kubectl create secret … --from-file (below)
fleeger-credential-key credential-key Master key for stored credentials secrets.example.yaml or kubectl create secret
fleeger-db db-password SurrealDB root password secrets.example.yaml or kubectl create secret
fleeger-internal internal-token Inter-service shared secret secrets.example.yaml or kubectl create secret
fleeger-victoriametrics delete-key, snapshot-key VictoriaMetrics' admin-API keys secrets.example.yaml or kubectl create secret

1. NATS auth material (fleeger-nats-auth) — required

This is the one Fleeger cannot generate on the cluster (it needs nsc). Run the generator once, locally:

deploy/nats/gen-auth.sh          # writes gitignored files under deploy/nats/auth/

Regenerating rotates all NATS credentials — every agent must re-onboard. See Agent credentials & revocation.

Then load every piece into one Secret. Include all the *.creds — a service whose cred key is missing from the Secret will fail to start (the volume projection references it by key):

kubectl -n fleeger create secret generic fleeger-nats-auth \
  --from-file=ingestion.creds=deploy/nats/auth/ingestion.creds \
  --from-file=gateway.creds=deploy/nats/auth/gateway.creds \
  --from-file=alerting.creds=deploy/nats/auth/alerting.creds \
  --from-file=inventory.creds=deploy/nats/auth/inventory.creds \
  --from-file=control.creds=deploy/nats/auth/control.creds \
  --from-file=core-agent.creds=deploy/nats/auth/core-agent.creds \
  --from-file=sys.creds=deploy/nats/auth/sys.creds \
  --from-file=operator.nk=deploy/nats/auth/operator.nk \
  --from-file=spokes-account.nk=deploy/nats/auth/spokes-account.nk \
  --from-file=resolver.conf=deploy/nats/auth/resolver.conf \
  --from-file=SPOKES.jwt="$(ls deploy/nats/auth/nsc-data/nats/nsc/stores/fleeger/accounts/SPOKES/SPOKES.jwt)"

core-agent.creds is only needed if you enable the bundled cluster-monitoring agent (Monitoring the hub's own cluster (bundled core-agent)); it's harmless to include always, so the command above does.

2. The other four — required

Two ways; pick one.

a) From the example file (good for GitOps once encrypted):

cp deploy/k8s/secrets/secrets.example.yaml deploy/k8s/secrets/secrets.yaml
# edit secrets.yaml — replace every REPLACE-ME with a generated value
kubectl -n fleeger apply -f deploy/k8s/secrets/secrets.yaml

secrets.yaml is gitignored. Do not commit plaintext secrets; if you keep them in git, seal them first (e.g. Sealed Secrets, SOPS, External Secrets).

b) Imperatively (nothing written to disk):

head -c 32 /dev/urandom | base64 | tr -d '\n' | \
  kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin

head -c 24 /dev/urandom | base64 | tr -d '\n' | \
  kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=/dev/stdin

head -c 24 /dev/urandom | base64 | tr -d '\n' | \
  kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=/dev/stdin

d=$(mktemp -d) && openssl rand -hex 32 | tr -d '\n' >"$d/delete-key" &&
  openssl rand -hex 32 | tr -d '\n' >"$d/snapshot-key" &&
  kubectl -n fleeger create secret generic fleeger-victoriametrics --from-file="$d"
rm -rf "$d"

fleeger-victoriametrics holds two different hex keys. VictoriaMetrics requires delete-key for every series delete, and only ingestion-svc holds it; without it, anyone who can reach VictoriaMetrics deletes every series with one request. snapshot-key guards the snapshot and other maintenance endpoints; only a vmbackup sidecar holds it, if you add one (optional/backup-cronjob.yaml shows how). Hex, because the snapshot key travels in the sidecar's snapshot URLs. No surrounding whitespace or trailing newline (hence tr -d '\n'): VictoriaMetrics does not trim its keys, so a padded one never matches.

Back up fleeger-credential-key. It encrypts every stored infra credential and alert-channel config; lose it and those become unrecoverable.

3. Rotating the credential key (optional)

inventory-svc and alerting-svc (and the optional restore pod) read FLEEGER_CREDENTIAL_KEYS_RETIRED from a sixth Secret, fleeger-credential-keys-retired (key retired-keys): the previous key(s), decrypt-only, comma-separated if there are several. The reference is optional: true, so the Secret exists only while a rotation is under way; without it the variable is unset and nothing changes. This is the manifest form of the compose walkthrough in Deploying Fleeger and the chart's credential.retiredKeysSecret.

Rotate the same way you created the Secrets in section 2. If you keep them declaratively (path a: secrets.yaml, or its sealed / SOPS / GitOps source), make every change below in that source and apply it from there; do not mix in the imperative kubectl commands. A key changed only in the cluster is reverted by the next apply or sync of the source, and once the retired-keys Secret is gone, everything already re-encrypted under the new key is unreadable.

The services do not restart on their own when a Secret changes (checksum annotations cover only NATS and the edge), so every step that changes a Secret ends with a rollout restart.

  1. Copy the current key into the retired-keys Secret, before you replace it. Nothing on this path refuses a key change the way the chart's credential.allowKeyChange does, so this order is all that keeps the old key reachable.

    Declaratively: uncomment the fleeger-credential-keys-retired block at the end of secrets.yaml, set retired-keys to the current credential-key (appended with a comma if the block already lists keys), and apply. The list is comma-separated with no spaces: the check below matches each entry exactly, so k0, k1 does not contain k1. Imperatively:

    cur=$(kubectl -n fleeger get secret fleeger-credential-key -o jsonpath='{.data.credential-key}' | base64 -d)
    [ -n "$cur" ] && printf '%s' "$cur" | \
      kubectl -n fleeger create secret generic fleeger-credential-keys-retired --from-file=retired-keys=/dev/stdin

    If that fails with AlreadyExists (an earlier rotation's Secret is still there), add the key to its comma-separated list instead of creating a new one:

    prev=$(kubectl -n fleeger get secret fleeger-credential-keys-retired -o jsonpath='{.data.retired-keys}' | base64 -d)
    [ -n "$cur" ] && [ -n "$prev" ] && printf '%s,%s' "$prev" "$cur" | \
      kubectl -n fleeger create secret generic fleeger-credential-keys-retired --from-file=retired-keys=/dev/stdin \
        --dry-run=client -o yaml | kubectl -n fleeger apply -f -

    Either way, check that the current key really is in the retired Secret before you go on; do not replace the key unless this prints OK:

    cur=$(kubectl -n fleeger get secret fleeger-credential-key -o jsonpath='{.data.credential-key}' | base64 -d)
    ret=$(kubectl -n fleeger get secret fleeger-credential-keys-retired -o jsonpath='{.data.retired-keys}' | base64 -d)
    if [ -n "$cur" ] && case ",$ret," in *",$cur,"*) true ;; *) false ;; esac
    then echo OK; else echo 'MISSING: the current key is not in retired-keys'; fi

    Keep a copy of the old key off-box as well (step 5).

  2. Generate the new key into a file in a fresh temporary directory, readable only by you:

    d=$(mktemp -d) && (umask 077 && head -c 32 /dev/urandom | base64 | tr -d '\n' >"$d/new-credential-key")

    Back it up off-box now, before it reaches the cluster: once step 4 has re-encrypted everything under it and the old key is retired, losing it makes every stored credential and alert-channel config unrecoverable. Then put it in place. Declaratively: set credential-key in secrets.yaml (or its sealed source) to the new key and apply. Imperatively:

    kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key="$d/new-credential-key" \
      --dry-run=client -o yaml | kubectl -n fleeger apply -f -

    Once the key is backed up (and in your source), remove the directory:

    rm -rf "$d"
  3. Restart both services so they read the new key and the retired one, and wait for the rollouts:

    kubectl -n fleeger rollout restart deploy/inventory-svc deploy/alerting-svc
    kubectl -n fleeger rollout status deploy/inventory-svc
    kubectl -n fleeger rollout status deploy/alerting-svc

    Then check that the pods actually see the retired key(s). The reference is optional, so a misspelt Secret name or key leaves the variable unset without any error, and nothing would show it until rekey fails:

    kubectl -n fleeger exec deploy/inventory-svc -- sh -c 'test -n "$FLEEGER_CREDENTIAL_KEYS_RETIRED"' && echo inventory-svc OK || echo inventory-svc MISSING
    kubectl -n fleeger exec deploy/alerting-svc -- sh -c 'test -n "$FLEEGER_CREDENTIAL_KEYS_RETIRED"' && echo alerting-svc OK || echo alerting-svc MISSING
  4. Re-encrypt everything under the new key, in both services:

    kubectl -n fleeger exec deploy/inventory-svc -- /usr/local/bin/service rekey
    kubectl -n fleeger exec deploy/alerting-svc -- /usr/local/bin/service rekey

    Each run is journalled in the hub-wide audit log (secrets.rekey, never a key) before it re-encrypts anything, and refuses, changing nothing, when the audit log cannot take the row.

    Then run both again to confirm. Step 3 waited for the rollouts, so no pod still writes under the old key; the second run is the proof that nothing is left under it, and re-encrypts what the first one did not finish (it stops on a row that keeps being edited meanwhile). Repeat until each reports 0 ... re-encrypted. Only then delete the retired-keys Secret (declaratively: remove the block from secrets.yaml or your source too, or the next apply or sync brings it back) and restart again:

    kubectl -n fleeger delete secret fleeger-credential-keys-retired
    kubectl -n fleeger rollout restart deploy/inventory-svc deploy/alerting-svc
  5. Keep the old key, off-box, for as long as you keep any backup taken before both rekeys were confirmed: restoring one needs it back in fleeger-credential-keys-retired, a restart, and both rekey runs again (Backup & disaster recovery).

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.