Zum Inhalt springen
Produkte

// Deployment

Deploying Fleeger on Kubernetes

There are two ways to run the Fleeger hub on Kubernetes — pick one, don't mix them for the same install:

Both describe the same system and expect the same Secrets. Choose Helm if you want configuration knobs and lifecycle management; choose manifests if you want to see and own every object, run under GitOps, or simply avoid installing Helm.

For Docker Compose on a single host see Deploying Fleeger; for running an agent see Onboarding a new location.

What runs

The chart brings up, in one namespace:

  • Dependencies — SurrealDB, NATS/JetStream, VictoriaMetrics (each a Deployment + PVC + Service).
  • Services — auth-svc, inventory-svc, gateway-svc, ingestion-svc, alerting-svc. The DB-owning ones run their migrations in an init container (retried until SurrealDB is up); gateway-svc waits for auth-svc.
  • UI — the frontend SPA behind an edge (nginx) that serves it and proxies /api to the gateway. Optionally a Kubernetes Ingress instead.
  • A post-install Job that pushes the SPOKES account into the NATS resolver (the in-cluster equivalent of deploy/nats/push-accounts.sh).

Service names are fixed (so the services find each other by DNS) — install one hub release per namespace. This is true of both the Helm and the manifest path.

Deploying with the Helm chart

The steps below install the hub with Helm. For the kubectl-only path, skip to Deploying with plain manifests.

Prerequisites

  • A cluster + kubectl context, and Helm 3.14 or later (3.8 added OCI; the upgrade commands below use --reset-then-reuse-values, which 3.14 added).
  • The images reachable from the cluster: pull from a registry (set image.registry/image.tag) or load your locally built fleeger/*:local images into the nodes and pass --set image.tag=local to the install in step 3. A chart from a checkout must be told the tag; a chart from the OCI registry should not be (see Installing from the OCI registry).
  • nsc is not needed on the cluster — auth material is generated once, locally, and shipped as a Secret.

Steps

1. Generate the NATS auth material (once, locally)

deploy/nats/gen-auth.sh

This writes gitignored material under deploy/nats/auth/. Regenerating rotates all NATS credentials (every agent must re-onboard). See Agent credentials & revocation.

2. Namespace + secrets

The chart references Secrets it deliberately does not contain:

kubectl create namespace fleeger

# NATS auth material (flat keys the services + resolver expect). Each service
# gets only its own least-privilege creds; only auth-svc gets the seeds.
# Include EVERY key below: a service whose cred is missing references it by key
# in its volume projection, so the pod never starts (CrashLoopBackOff, or stuck
# ContainerCreating for core-agent). The chart preflights this at install time.
kubectl -n fleeger create secret generic fleeger-nats-auth \
  --from-file=ingestion.creds=deploy/nats/auth/ingestion.creds \
  --from-file=gateway.creds=deploy/nats/auth/gateway.creds \
  --from-file=alerting.creds=deploy/nats/auth/alerting.creds \
  --from-file=inventory.creds=deploy/nats/auth/inventory.creds \
  --from-file=control.creds=deploy/nats/auth/control.creds \
  --from-file=core-agent.creds=deploy/nats/auth/core-agent.creds \
  --from-file=sys.creds=deploy/nats/auth/sys.creds \
  --from-file=operator.nk=deploy/nats/auth/operator.nk \
  --from-file=spokes-account.nk=deploy/nats/auth/spokes-account.nk \
  --from-file=resolver.conf=deploy/nats/auth/resolver.conf \
  --from-file=SPOKES.jwt=deploy/nats/auth/nsc-data/nats/nsc/stores/fleeger/accounts/SPOKES/SPOKES.jwt

core-agent.creds is only read when coreAgent.enabled=true, but it is harmless to include always and skipping it is the usual reason a later --set coreAgent.enabled=true leaves a pod stuck in ContainerCreating.

# Credential master key — BACK IT UP (losing it makes stored secrets unrecoverable):
head -c 32 /dev/urandom | base64 | tr -d '\n' | \
  kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin

The SurrealDB root password and the internal service token (the shared secret the gateway attaches so the backend services reject anyone else on the pod network) are both handled for you: on helm install the chart generates strong values into fleeger-db / fleeger-internal Secrets and keeps them across upgrades (they never appear in a manifest or ConfigMap). That needs Helm to read the cluster while it renders: under Argo CD, helm template or another client-side render the chart refuses to generate them and names the existingSecret values to set instead (recipes in the Helm chart reference, section GitOps). To pin or bring your own, create the Secrets yourself as for the credential key above (keys db-password and internal-token) and point surrealdb.existingSecret / internal.existingSecret at them. --set surrealdb.password=… and --set internal.token=… work too, but put the value on the helm command line, readable by every local user with ps, and in the release's stored values. A pinned DB password takes effect only when the datastore is first created.

VictoriaMetrics' admin-API keys are generated the same way (and, like them, not in a client-side render), into <fullname>-victoriametrics: delete-key, which VictoriaMetrics requires for every series delete and only ingestion-svc holds, and snapshot-key, for the snapshot and other maintenance endpoints, which only the vmbackup sidecar holds. To bring your own, create a Secret with both keys (two different openssl rand -hex 32; hex, as the snapshot key rides in the sidecar's URLs) and set victoriametrics.auth.existingSecret. Write each key without surrounding whitespace or a trailing newline (tr -d '\n'): VictoriaMetrics does not trim its keys, so a padded one never matches.

3. Install the hub

helm install fleeger charts/fleeger-hub \
  --namespace fleeger \
  --set image.tag=local \
  --set credential.existingSecret=fleeger-credential-key

image.tag=local runs the fleeger/*:local images you built and loaded. Left empty, the tag is the chart's appVersion, which in a checkout is not the version you built. Pulling from a registry instead of local images? Replace the image.tag line with these, <version> being the published version whose images match your checkout (e.g. 1.0.0-rc.3 for that tag's checkout):

  --set image.registry=registry.gitlab.example.com/you/fleeger \
  --set 'image.tag=<version>' \
  --set 'imagePullSecrets[0].name=<your-pull-secret>'

Name a version, not a moving tag such as latest: the tag is stored in the release and reused on every upgrade, so with latest each node runs whatever the tag pointed to when it last pulled, and an upgrade does not change the tag, so it does not roll the pods. Bump it when you upgrade the checkout.

Prefer no Helm at all? Skip this whole section and use the plain manifests — see Deploying with plain manifests.

4. Watch it settle

kubectl -n fleeger get pods -w

The fleeger-nats-push Job runs once NATS is up; the services finish authenticating shortly after and go Ready.

5. Create the first admin

kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service \
  user add -u admin -e admin@example.com -r admin   # prompts for the password

The command is journalled in the audit log before the user is created, straight into inventory-svc's audit database, so run it once inventory-svc is up (its init container migrates that database); before that it refuses and creates nothing.

6. Reach the UI

The flags below go on the helm install in step 3. The hub is already installed by now, so to add them to it, put them on helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values instead: an upgrade that passes values without that flag resets everything else you set at install (see the Helm chart reference's Upgrade section).

  • Ingress (needs a controller): add these flags to the helm install in step 3 (ending its last line with \), then open the host.

      --set ingress.enabled=true \
      --set ingress.host=fleeger.example.com \
      --set 'edge.trustedProxies[0]=<ingress-controller-pod-CIDR>'

    <ingress-controller-pod-CIDR> is a placeholder. Replace it with the address range of the one hop directly in front of the edge:

    • For an in-cluster controller (ingress-nginx, Traefik, HAProxy) that is your cluster's pod CIDR — the cluster CIDR it was created with (k3s and RKE2 default to 10.42.0.0/16, but check yours; managed clusters show it in their network settings). Confirm it covers the controller: kubectl get pods -A -o wide shows the controller's pods (named e.g. ingress-nginx-controller-…, or traefik-… on k3s) with their IPs, and every one must fall inside the range. A controller running with hostNetwork: true shows node addresses there instead, and the pod CIDR is then not necessarily the right range.
    • If the hop in front of the edge is something else, such as a cloud load balancer, name its source range.

    Trust only that hop. A wider range lets anyone who can reach the edge forge their source address, dodge the login rate limit and write what they like into the audit log. An unreplaced placeholder, or any entry that is not an IPv4/IPv6 address or CIDR, stops the install before anything is applied, with an error naming the entry; the chart cannot tell whether a valid range is the right one, so that part is yours. Leave edge.trustedProxies out entirely and the install stops the same way, with an error saying edge.trustedProxies is empty. The Helm chart reference's Upgrade section explains why, and when the edge.preflightTrustedProxies=false waiver is appropriate instead (only when the ingress is not actually in the request path).

  • NodePort (--set edge.service.type=NodePort --set edge.service.nodePort=30080): http://<node-ip>:30080.

  • Quick check: kubectl -n fleeger port-forward svc/edge 8080:80 → open http://localhost:8080.

7. Enable TLS (do this before exposing it)

The login sends credentials and agents connect over WebSocket, so terminate TLS at the ingress for anything past a trusted LAN. With cert-manager it's a values block:

ingress:
  enabled: true
  className: nginx
  host: fleeger.example.com
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  tls:
    - hosts: [fleeger.example.com]
      secretName: fleeger-tls
edge:
  trustedProxies:
    - "<ingress-controller-pod-CIDR>"   # required with an ingress — see step 6
nats:
  publicUrl: wss://fleeger.example.com/nats   # agents get this at bootstrap

Save it as a file and pass it with -f — to the install, or to a running release with helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values -f fleeger-tls.yaml.

Then onboard agents against the HTTPS host (--set hub.url=https://fleeger.example.com on the agent chart). The chart prints a warning at install time if the ingress is enabled without tls, or if nats.publicUrl is a plaintext ws:// URL.

8. Network segmentation (optional, defense-in-depth)

The services already authenticate each other with the internal token, but you can also lock down the pod network so nothing but the intended caller can even open a connection:

helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values \
  --set networkPolicy.enabled=true

This adds a default-deny-ingress policy plus per-service allow rules (e.g. inventory-svc only accepts from gateway-svc and alerting-svc). It's off by default and unvalidated against any specific CNI — health/readiness probes come from the node's kubelet, which a pod selector can't match. Most CNIs (Cilium, Calico) allow kubelet probes automatically; if your pods go unready after enabling this, pass your node CIDR(s) so probes are allowed through:

helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values \
  --set 'networkPolicy.probeCIDRs={10.0.0.0/8}'

Installing from the OCI registry

The Steps above build the chart from a checkout. Once a release is published you can skip the checkout entirely and helm install the packaged chart straight from the project's OCI registry (Helm 3.8+). The Secrets in Step 1–Step 2 (the NATS auth material and the credential master key) are still required — the charts deliberately don't ship them — so create the namespace and those Secrets first, then install:

# The hub (services + dependencies + UI):
helm install fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
  --version 1.0.0-rc1 \
  --namespace fleeger \
  --set image.registry=registry.example.com/fleeger \
  --set 'imagePullSecrets[0].name=<your-pull-secret>' \
  --set credential.existingSecret=fleeger-credential-key

Point image.registry at the same registry the release published its per-service images to (registry.example.com/fleeger/<service>:1.0.0-rc1), and add an imagePullSecrets entry if that registry is private. Leave image.tag unset: the chart's appVersion is the tag its release pipeline pushed, so each chart upgrade brings the matching images. A tag you set is stored in the release and carried into every later upgrade, including --reset-then-reuse-values ones, which leaves the old images running under the new chart; a release installed with one clears it once with helm upgrade … --reset-then-reuse-values --set image.tag=. Everything else — reaching the UI, TLS, network segmentation — is identical to the checkout install.

A remote agent installs the same way from its own published chart, with the one-time registration token from Onboard (see onboarding a location) in a Secret. Paste it at the prompt; it is not echoed and stays off every command line:

printf 'Registration token: '; stty -echo; IFS= read -r FLEEGER_TOKEN; stty echo; echo
printf '%s' "$FLEEGER_TOKEN" | kubectl create secret generic fleeger-agent-token \
  --from-file=registration-token=/dev/stdin
unset FLEEGER_TOKEN
helm install fleeger-agent oci://registry.example.com/fleeger/charts/fleeger-agent \
  --version 1.0.0-rc1 \
  --set image.repository=registry.example.com/fleeger/spoke-agent \
  --set hub.url=https://fleeger.example.com \
  --set existingSecret=fleeger-agent-token

Bring your own identity provider

Every core service can be turned off with --set services.<name>.enabled=false when you supply that capability yourself. The common case is authentication: front the gateway with your own OIDC provider and disable the bundled auth-svc with this flag on the helm install in step 3:

  --set services.auth-svc.enabled=false

On a running hub, put it on helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values instead.

Release process

The charts and images are cut by GitLab CI on tag pipelines, not by hand:

  • Images — CI builds every per-service and UI image with Kaniko and pushes them to the project's GitLab Container Registry as ${CI_REGISTRY_IMAGE}/<service>:<tag> (services: auth-svc, inventory-svc, gateway-svc, ingestion-svc, alerting-svc, control-svc, spoke-agent, frontend, and the backup runner). See container images & GitLab CI.
  • Charts — the same tag pipeline helm packages both fleeger-hub and fleeger-agent and helm pushes them to the OCI registry under charts/, so oci://registry.example.com/fleeger/charts/<chart> resolves the chart at --version <chart version> (e.g. 1.0.0-rc1).

So a single tag produces the images and both charts; the oci://… installs above just pull what that pipeline published.

The version comes from the tag

CI derives every published version from the git tag, not from the checked-in files: in a tag pipeline FLEEGER_VERSION is the tag itself, and the image tags, the version compiled into the binaries and both charts' version and appVersion (helm package --version --app-version) are all set from it. The version/appVersion in Chart.yaml and the workspace Cargo.toml are not used for anything published, so there is nothing to bump by hand before tagging. Because image.tag defaults to the chart's appVersion, a published chart asks for exactly the image tags its own pipeline pushed, and the charts job waits for the image jobs, so a chart is not published when one of them failed.

  1. Tag release candidates on rc as X.Y.Z-rc.N (no v prefix: the tag is used verbatim as the image tag and chart version). The release itself is tagged X.Y.Z by the create-release job on master.
  2. Update the example versions that quote it: the oci:// install snippets above, installing from scratch, onboarding a location, and the two Helm chart references.

Two separate checks, and neither substitutes for the other:

  • The versions — after the tag pipeline, confirm the published chart renders the tag it was published under in every image::

    helm template oci://registry.example.com/fleeger/charts/fleeger-agent \
      --version X.Y.Z --set image.repository=registry.example.com/fleeger/spoke-agent | grep 'image:'

    (The hub chart needs a key to render: add --set credential.key=$(head -c 32 /dev/urandom | base64), or --set credential.existingSecret=x.)

  • The templates — just render-charts lints and renders both charts across the value combinations CI covers. It does not look at versions, and it prints no manifests. Install kubeconform first or it skips the Kubernetes schema half; the summary line says which mode it ran in.

Deploying with plain manifests (no Helm)

Everything above uses Helm. If you'd rather not — you want to own every object, run under GitOps, or just avoid installing Helm — deploy the manifests in Fleeger on Kubernetes — plain manifests with kubectl instead. They're hand-maintained (not a helm template dump), split so the minimal core is obvious and everything else is opt-in:

deploy/k8s/
├── base/        the minimal hub — apply this and it runs
├── secrets/     the 5 Secrets you create (never committed)
├── optional/    ingress+TLS, cluster monitoring, network policies, backups, NodePort
└── agent/       a spoke-agent for one location

What the minimal setup needs

The smallest working hub is the base + five Secrets + reachable images:

  1. kubectl and a cluster — the only tool required (no Helm, no cluster-side nsc).
  2. Images reachable from the cluster — locally-loaded fleeger/*:local, or a registry you repoint the images at (one place: base/kustomization.yaml, or edit the manifests — see Fleeger on Kubernetes — plain manifests).
  3. Five Secrets (Secrets the hub needs): fleeger-nats-auth (from deploy/nats/gen-auth.sh), fleeger-credential-key, fleeger-db, fleeger-internal, fleeger-victoriametrics (VictoriaMetrics' delete-key and snapshot-key). Unlike Helm, none are generated for you — that's the trade for plain manifests, and it keeps secrets out of git.
  4. The base — dependencies, the core services, the UI + edge, and the one-shot fleeger-nats-push Job.

Required components are SurrealDB, NATS, VictoriaMetrics, auth-svc, gateway-svc, inventory-svc, ingestion-svc, the frontend + edge, and the NATS-push Job. alerting-svc (alerting) and control-svc (control actions) are optional — delete their Service + Deployment from base/40-app-services.yaml if you don't use those features. auth-svc can go only if you front the gateway with your own OIDC (see Bring your own identity provider); on the manifest path that means deleting auth-svc and dropping gateway-svc's wait-for-auth init container plus pointing its config at the tokens your provider issues — not a drop-in change.

Steps

# 1. Namespace
kubectl create namespace fleeger

# 2. Secrets — full commands in "Secrets the hub needs"
deploy/nats/gen-auth.sh
kubectl -n fleeger create secret generic fleeger-nats-auth --from-file=...   # see "secrets"
head -c 32 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-db            --from-file=db-password=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-internal      --from-file=internal-token=/dev/stdin
d=$(mktemp -d) && openssl rand -hex 32 | tr -d '\n' >"$d/delete-key" && openssl rand -hex 32 | tr -d '\n' >"$d/snapshot-key" &&
  kubectl -n fleeger create secret generic fleeger-victoriametrics --from-file="$d"; rm -rf "$d"

# 3. The hub
kubectl -n fleeger apply -f deploy/k8s/base/      # or: kubectl apply -k deploy/k8s/base

# 4. Watch it settle
kubectl -n fleeger get pods -w

# 5. First admin user (same as the Helm path)
kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service \
  user add -u admin -e admin@example.com -r admin   # prompts for the password

# 6. Reach the UI (quick check)
kubectl -n fleeger port-forward svc/edge 8080:80  # http://localhost:8080

The fleeger-nats-auth Secret must include all the *.creds files (ingestion, gateway, alerting, inventory, control, core-agent) — a service whose cred is missing won't start, since its volume references the key by name. The exact --from-file list is in Secrets the hub needs.

The options (everything past the minimum)

Each is a file (or folder) in Optional add-ons (manifests) you apply à la carte; read the note at the top of each before applying.

  • Ingress + TLS (optional/ingress.yaml) — the recommended way to expose Fleeger. Do this before exposing it past a trusted LAN: login and agent traffic must be HTTPS. Then set [nats] public_url = "wss://<host>/nats" in auth-svc.toml (base/10-config.yaml) so agents get the external endpoint at bootstrap. The edge answers everything coming through the Ingress with a 503 until set_real_ip_from in the fleeger-edge ConfigMap names the ingress controller: the raw-manifest counterpart of the chart's edge.trustedProxies check (see Fleeger on Kubernetes — plain manifests).
  • NodePort access (optional/edge-nodeport.yaml) — quick LAN/homelab access with no ingress controller (plaintext HTTP; trusted networks only).
  • Monitoring the hub's own cluster (optional/cluster-monitoring/) — this is the service account to monitor the cluster from inside it. It's the bundled core-agent plus a ServiceAccount + read-only ClusterRole/Binding; a k8s_cluster resource with an empty credential then collects the cluster via that ServiceAccount (no kubeconfig). Covered in more depth under Monitoring the hub's own cluster below.
  • Network segmentation (optional/network-policies.yaml) — default-deny ingress + per-service allow rules. CNI-specific and untested; validate first.
  • Scheduled backups (optional/backup-cronjob.yaml) — nightly SurrealDB dumps to S3 + the backup-success metric. See Backup & disaster recovery.
  • A remote agent (Running a spoke-agent with plain manifests) — a spoke-agent for a location, with optional cluster RBAC if it monitors the cluster it runs in.

Operating the manifest install

The Operating notes below apply, with the Helm verbs swapped for kubectl: change a value in a ConfigMap and kubectl -n fleeger rollout restart deploy/<service> (the service Deployments have no config-checksum auto-roll); upgrade by editing the image tags (or base/kustomization.yaml) and re-applying. NATS and the edge do roll on their own when a release changes their config (nats.conf, or the edge's default.conf and fleeger-upstreams.envsh): their Deployments carry a checksum/nats-conf or checksum/edge-conf annotation over it, as the chart's do, and CI keeps it current. An edit of your own to those ConfigMaps, in place or through a kustomize overlay patch, does not change the annotation, so restart deploy/nats or deploy/edge yourself, or update the annotation with the command in the comment beside it (POSIX shell; on macOS use shasum -a 256 for sha256sum; copy only the hex digest into the quoted value). The nats-push Job is immutable, so kubectl -n fleeger delete job fleeger-nats-push before re-running it.

Rotating the credential key is the exception: the chart's credential.retiredKeysSecret and credential.allowKeyChange have no manifest equivalent. The manifests read the retired key(s) from an optional Secret, fleeger-credential-keys-retired, instead, and a changed Secret restarts nothing, so follow the manifest steps in Secrets the hub needs rather than the Helm bullet below.

inventory-svc runs as one pod

inventory-svc has replicas: 1 and strategy: Recreate, in the chart and in deploy/k8s. A rollout stops the old pod first; the new pod's init container then runs db migrate (it merges duplicate discovered resources, applies the migrations, checks the discovery key and defines again what differs from the migrations; that check replays every migration into a scratch database it creates and removes, a few seconds on each start of every service's pod), and only then does the new pod start. This keeps two rules:

  • No migration runs beside a writing inventory-svc. Migration 0040 builds a UNIQUE index over the committed data, and a discovery report committing while it builds can be missing from the index for good, which lets the next report of that object create a second row.
  • A rollout never has two inventory-svc pods writing. Its per-location lanes, which keep one location's reports in order, hold within one process only.

Recreate covers rollouts only. A pod deleted by hand or evicted, or one on a node gone NotReady, can still run beside its replacement for a while, so restart inventory-svc with kubectl -n fleeger rollout restart deploy/inventory-svc rather than by deleting its pod. Such an overlap cannot duplicate a discovered resource (the UNIQUE discovery key refuses the second row); it only runs a location's reports in two processes for that time.

The price is a gap on every rollout of inventory-svc, typically seconds, one that only changes its configuration included (the chart's config checksum annotation, or with the manifests a kubectl rollout restart, restarts the pod). During it:

  • the inventory API in the UI answers 502/503;
  • every gateway request to /api/resources/{id}/… or /api/credentials/{id} whose location the gateway has not cached fails with 502/503, as the gateway asks inventory-svc for it (services/gateway-svc/src/resource_scope.rs);
  • control actions fail, as control-svc looks the resource up in inventory-svc;
  • that cycle's agent reports (discovery, hardware, collect status, the resource list) are dropped, not queued, and show as a collect error.

If db migrate fails, the gap lasts until it succeeds (the init container retries it, then the pod crash-loops). The built-in workload_degraded rule ships disabled, and sees the hub's own Deployments only if the hub's cluster is monitored by an agent; enabled, it fires after its 10-minute window, so a normal rollout does not trip it. Do not patch the Deployment back to RollingUpdate or scale it up: a second writer (a scaled-up Deployment, or one patched to RollingUpdate) brings the hazards above back. A GitOps tool or server-side apply that keeps the old rollingUpdate block fails with "rollingUpdate may not be specified when strategy type is Recreate": patch it to rollingUpdate: null.

Reaching agents

Agents run at your locations, not in this cluster, and reach the hub over a single endpoint: they bootstrap over HTTPS, then do metrics and inventory over the NATS WebSocket (wss://<hub>/nats). Nothing else — raw NATS, inventory-svc — is ever exposed. See Agent credentials & revocation for the trust model.

To make that endpoint reachable from outside the cluster:

  • Front the edge with an Ingress, WebSocket-capable (the common controllers are). With Helm that is the three flags in step 6, including edge.trustedProxies, which the chart will not render without when an ingress is enabled. The edge already routes /api → gateway and /nats → the NATS WebSocket, so one host serves everything.
  • Tell agents that external URL: --set nats.publicUrl=wss://fleeger.example.com/nats (what they receive at bootstrap; on a running hub, with helm upgrade … --reset-then-reuse-values). Then onboard with --set hub.url=https://fleeger.example.com on the agent chart.

An agent that runs inside the same cluster (monitoring that cluster) can use the in-cluster default and, with --set rbac.create=true on the agent chart, its ServiceAccount for read-only node/pod metrics.

Monitoring the hub's own cluster

The hub can watch the very cluster it runs in with the bundled core agent (--set coreAgent.enabled=true) — no separate agent chart. Two things make that work:

  • Leave the k8s_cluster resource credential empty. With no kubeconfig set, the collector falls back to in-cluster access via its ServiceAccount (kube::Client::try_default()). Do not paste the external admin kubeconfig: its endpoint is a control-plane public IP that a pod on a firewalled cluster can't reach, and it's far more privileged than the collector needs.
  • Let the chart create the RBAC. coreAgent.rbac.create (and coreAgent.serviceAccount.create) are on by default when coreAgent.enabled=true and create the ServiceAccount + a read-only ClusterRole + binding. The role covers the full set the k8s collector discovers — nodes/pods/namespaces/services/PVCs, deployments/statefulsets/daemonsets, jobs/cronjobs, ingresses, HPAs, metrics, and cert-manager CRDs — not just node/pod metrics.

With plain manifests this is the optional/cluster-monitoring/ add-on: apply core-agent-rbac.yaml (the ServiceAccount + read-only ClusterRole/Binding) then core-agent.yaml (the core-agent Deployment). Make sure core-agent.creds is in the fleeger-nats-auth Secret. See Monitoring the hub's own cluster (bundled core-agent).

Operating

These notes are written for the Helm install. For the manifest install the same ideas apply with kubectl verbs — see Operating the manifest install.

  • Config is templated from Helm values into a ConfigMap; change values with helm upgrade … --reset-then-reuse-values --set … (without that flag, an upgrade that passes values resets every value it does not repeat — see the Helm chart reference's Upgrade section). Deployments roll on config changes (checksum annotation).
  • Upgrades — the Helm chart reference's Upgrade commands, which pass no values and so keep the release's own. Migrations re-run idempotently in the init containers; the nats-push Job re-runs.
  • inventory-svc runs as one pod, recreated on a rollout (see inventory-svc runs as one pod): every rollout of it has a short gap.
  • Backups — Backup & disaster recovery covers SurrealDB
    • VictoriaMetrics; run it against the PVCs (e.g. from a Job or kubectl exec).
  • Logs — kubectl -n fleeger logs deploy/<service>.
  • Rotating the credential key — set credential.key/existingSecret to the new key, create a Secret with a retired-keys entry (the old key) and point --set credential.retiredKeysSecret at it, and upgrade with helm upgrade … --reset-then-reuse-values (adding --set credential.allowKeyChange=true if you changed credential.key), then re-encrypt in both services that hold secrets: kubectl -n fleeger exec deploy/inventory-svc -- /usr/local/bin/service rekey (credentials) and kubectl -n fleeger exec deploy/alerting-svc -- /usr/local/bin/service rekey (notification-channel secrets). A secret edited while either runs keeps the edit (see the compose walkthrough). Each run is journalled in the audit log (secrets.rekey, never a key) before it re-encrypts anything, and refuses when the log cannot take the row. Once every replica of both runs with the new key (kubectl -n fleeger rollout status deploy/inventory-svc, then the same for deploy/alerting-svc; a replica still on the old key seals what it writes under the retired one), run rekey again in each. Once that run reports everything already current, clear the retired key and re-arm the key-change refusal: helm upgrade … --reset-then-reuse-values --set credential.retiredKeysSecret= --set credential.allowKeyChange=false. Keep the old key, off-box, for as long as you keep any backup taken before both rekeys were confirmed: restoring one needs it back in credential.retiredKeysSecret, and both rekey runs again afterwards (Backup & disaster recovery). (Same flow as the compose walkthrough in Deploying Fleeger; for the manifest install, see Secrets the hub needs.)

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.