// Deployment
Deploying Fleeger on Kubernetes
There are two ways to run the Fleeger hub on Kubernetes — pick one, don't mix them for the same install:
- Helm chart — Helm chart: fleeger-hub. Templated values, generated secrets, one-command upgrades. The rest of this document is the Helm walkthrough.
- Plain manifests — Fleeger on Kubernetes — plain manifests.
kubectland nothing else: a minimal base plus opt-in add-ons you apply à la carte. Jump to Deploying with plain manifests, or read Fleeger on Kubernetes — plain manifests for the file-by-file tour.
Both describe the same system and expect the same Secrets. Choose Helm if you want configuration knobs and lifecycle management; choose manifests if you want to see and own every object, run under GitOps, or simply avoid installing Helm.
For Docker Compose on a single host see Deploying Fleeger; for running an agent see Onboarding a new location.
What runs
The chart brings up, in one namespace:
- Dependencies — SurrealDB, NATS/JetStream, VictoriaMetrics (each a Deployment + PVC + Service).
- Services —
auth-svc,inventory-svc,gateway-svc,ingestion-svc,alerting-svc. The DB-owning ones run their migrations in an init container (retried until SurrealDB is up);gateway-svcwaits forauth-svc. - UI — the
frontendSPA behind an edge (nginx) that serves it and proxies/apito the gateway. Optionally a Kubernetes Ingress instead. - A post-install Job that pushes the SPOKES account into the NATS resolver
(the in-cluster equivalent of
deploy/nats/push-accounts.sh).
Service names are fixed (so the services find each other by DNS) — install one hub release per namespace. This is true of both the Helm and the manifest path.
Deploying with the Helm chart
The steps below install the hub with Helm. For the kubectl-only path, skip to
Deploying with plain manifests.
Prerequisites
- A cluster +
kubectlcontext, and Helm 3.14 or later (3.8 added OCI; the upgrade commands below use--reset-then-reuse-values, which 3.14 added). - The images reachable from the cluster: pull from a registry (set
image.registry/image.tag) or load your locally builtfleeger/*:localimages into the nodes and pass--set image.tag=localto the install in step 3. A chart from a checkout must be told the tag; a chart from the OCI registry should not be (see Installing from the OCI registry). nscis not needed on the cluster — auth material is generated once, locally, and shipped as a Secret.
Steps
1. Generate the NATS auth material (once, locally)
deploy/nats/gen-auth.shThis writes gitignored material under deploy/nats/auth/. Regenerating rotates
all NATS credentials (every agent must
re-onboard). See
Agent credentials & revocation.
2. Namespace + secrets
The chart references Secrets it deliberately does not contain:
kubectl create namespace fleeger
# NATS auth material (flat keys the services + resolver expect). Each service
# gets only its own least-privilege creds; only auth-svc gets the seeds.
# Include EVERY key below: a service whose cred is missing references it by key
# in its volume projection, so the pod never starts (CrashLoopBackOff, or stuck
# ContainerCreating for core-agent). The chart preflights this at install time.
kubectl -n fleeger create secret generic fleeger-nats-auth \
--from-file=ingestion.creds=deploy/nats/auth/ingestion.creds \
--from-file=gateway.creds=deploy/nats/auth/gateway.creds \
--from-file=alerting.creds=deploy/nats/auth/alerting.creds \
--from-file=inventory.creds=deploy/nats/auth/inventory.creds \
--from-file=control.creds=deploy/nats/auth/control.creds \
--from-file=core-agent.creds=deploy/nats/auth/core-agent.creds \
--from-file=sys.creds=deploy/nats/auth/sys.creds \
--from-file=operator.nk=deploy/nats/auth/operator.nk \
--from-file=spokes-account.nk=deploy/nats/auth/spokes-account.nk \
--from-file=resolver.conf=deploy/nats/auth/resolver.conf \
--from-file=SPOKES.jwt=deploy/nats/auth/nsc-data/nats/nsc/stores/fleeger/accounts/SPOKES/SPOKES.jwtcore-agent.creds is only read when coreAgent.enabled=true, but it is
harmless to include always and skipping it is the usual reason a later
--set coreAgent.enabled=true leaves a pod stuck in ContainerCreating.
# Credential master key — BACK IT UP (losing it makes stored secrets unrecoverable):
head -c 32 /dev/urandom | base64 | tr -d '\n' | \
kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdinThe SurrealDB root password and the internal service token (the shared
secret the gateway attaches so the backend services reject anyone else on the pod
network) are both handled for you: on helm install the chart generates strong
values into fleeger-db / fleeger-internal Secrets and keeps them across
upgrades (they never appear in a manifest or ConfigMap). That needs Helm to read
the cluster while it renders: under Argo CD, helm template or another
client-side render the chart refuses to generate them and names the
existingSecret values to set instead (recipes in the Helm chart reference,
section GitOps). To pin or bring your
own, create the Secrets yourself as for the credential key above (keys
db-password and internal-token) and point surrealdb.existingSecret /
internal.existingSecret at them. --set surrealdb.password=… and
--set internal.token=… work too, but put the value on the helm command line,
readable by every local user with ps, and in the release's stored values. A pinned DB password takes effect only when the datastore is first
created.
VictoriaMetrics' admin-API keys are generated the same way (and,
like them, not in a client-side render), into
<fullname>-victoriametrics: delete-key, which VictoriaMetrics requires for
every series delete and only ingestion-svc holds, and snapshot-key, for the
snapshot and other maintenance endpoints, which only the vmbackup sidecar holds.
To bring your own, create a Secret with both keys (two different
openssl rand -hex 32; hex, as the snapshot key rides in the sidecar's URLs)
and set victoriametrics.auth.existingSecret. Write each key without
surrounding whitespace or a trailing newline (tr -d '\n'): VictoriaMetrics
does not trim its keys, so a padded one never matches.
3. Install the hub
helm install fleeger charts/fleeger-hub \
--namespace fleeger \
--set image.tag=local \
--set credential.existingSecret=fleeger-credential-keyimage.tag=local runs the fleeger/*:local images you built and loaded. Left
empty, the tag is the chart's appVersion, which in a checkout is not the
version you built. Pulling from a registry instead of local images? Replace the
image.tag line with these, <version> being the published version whose
images match your checkout (e.g. 1.0.0-rc.3 for that tag's checkout):
--set image.registry=registry.gitlab.example.com/you/fleeger \
--set 'image.tag=<version>' \
--set 'imagePullSecrets[0].name=<your-pull-secret>'Name a version, not a moving tag such as latest: the tag is stored in the
release and reused on every upgrade, so with latest each node runs whatever
the tag pointed to when it last pulled, and an upgrade does not change the tag,
so it does not roll the pods. Bump it when you upgrade the checkout.
Prefer no Helm at all? Skip this whole section and use the plain manifests — see Deploying with plain manifests.
4. Watch it settle
kubectl -n fleeger get pods -wThe fleeger-nats-push Job runs once NATS is up; the services finish
authenticating shortly after and go Ready.
5. Create the first admin
kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service \
user add -u admin -e admin@example.com -r admin # prompts for the passwordThe command is journalled in the audit log before the user is created, straight into inventory-svc's audit database, so run it once inventory-svc is up (its init container migrates that database); before that it refuses and creates nothing.
6. Reach the UI
The flags below go on the helm install in step 3. The
hub is already installed by now, so to add them to it, put them on
helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values
instead: an upgrade that passes values without that flag resets everything
else you set at install (see the Helm chart reference's
Upgrade section).
Ingress (needs a controller): add these flags to the
helm installin step 3 (ending its last line with\), then open the host.--set ingress.enabled=true \ --set ingress.host=fleeger.example.com \ --set 'edge.trustedProxies[0]=<ingress-controller-pod-CIDR>'<ingress-controller-pod-CIDR>is a placeholder. Replace it with the address range of the one hop directly in front of the edge:- For an in-cluster controller (ingress-nginx, Traefik, HAProxy) that is your
cluster's pod CIDR — the cluster CIDR it was created with (k3s and RKE2
default to
10.42.0.0/16, but check yours; managed clusters show it in their network settings). Confirm it covers the controller:kubectl get pods -A -o wideshows the controller's pods (named e.g.ingress-nginx-controller-…, ortraefik-…on k3s) with their IPs, and every one must fall inside the range. A controller running withhostNetwork: trueshows node addresses there instead, and the pod CIDR is then not necessarily the right range. - If the hop in front of the edge is something else, such as a cloud load balancer, name its source range.
Trust only that hop. A wider range lets anyone who can reach the edge forge their source address, dodge the login rate limit and write what they like into the audit log. An unreplaced placeholder, or any entry that is not an IPv4/IPv6 address or CIDR, stops the install before anything is applied, with an error naming the entry; the chart cannot tell whether a valid range is the right one, so that part is yours. Leave
edge.trustedProxiesout entirely and the install stops the same way, with an error sayingedge.trustedProxiesis empty. The Helm chart reference's Upgrade section explains why, and when theedge.preflightTrustedProxies=falsewaiver is appropriate instead (only when the ingress is not actually in the request path).- For an in-cluster controller (ingress-nginx, Traefik, HAProxy) that is your
cluster's pod CIDR — the cluster CIDR it was created with (k3s and RKE2
default to
NodePort (
--set edge.service.type=NodePort --set edge.service.nodePort=30080):http://<node-ip>:30080.Quick check:
kubectl -n fleeger port-forward svc/edge 8080:80→ open http://localhost:8080.
7. Enable TLS (do this before exposing it)
The login sends credentials and agents connect over WebSocket, so terminate TLS at the ingress for anything past a trusted LAN. With cert-manager it's a values block:
ingress:
enabled: true
className: nginx
host: fleeger.example.com
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
tls:
- hosts: [fleeger.example.com]
secretName: fleeger-tls
edge:
trustedProxies:
- "<ingress-controller-pod-CIDR>" # required with an ingress — see step 6
nats:
publicUrl: wss://fleeger.example.com/nats # agents get this at bootstrap
Save it as a file and pass it with -f — to the install, or to a running
release with
helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values -f fleeger-tls.yaml.
Then onboard agents against the HTTPS host (--set hub.url=https://fleeger.example.com
on the agent chart). The chart prints a warning at install time if the ingress is
enabled without tls, or if nats.publicUrl is a plaintext ws:// URL.
8. Network segmentation (optional, defense-in-depth)
The services already authenticate each other with the internal token, but you can also lock down the pod network so nothing but the intended caller can even open a connection:
helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values \
--set networkPolicy.enabled=trueThis adds a default-deny-ingress policy plus per-service allow rules (e.g. inventory-svc only accepts from gateway-svc and alerting-svc). It's off by default and unvalidated against any specific CNI — health/readiness probes come from the node's kubelet, which a pod selector can't match. Most CNIs (Cilium, Calico) allow kubelet probes automatically; if your pods go unready after enabling this, pass your node CIDR(s) so probes are allowed through:
helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values \
--set 'networkPolicy.probeCIDRs={10.0.0.0/8}'Installing from the OCI registry
The Steps above build the chart from a checkout. Once a release is published you
can skip the checkout entirely and helm install the packaged chart straight
from the project's OCI registry (Helm 3.8+). The Secrets in
Step 1–Step 2
(the NATS auth material and the credential master key) are still required — the
charts deliberately don't ship them — so create the namespace and those Secrets
first, then install:
# The hub (services + dependencies + UI):
helm install fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
--version 1.0.0-rc1 \
--namespace fleeger \
--set image.registry=registry.example.com/fleeger \
--set 'imagePullSecrets[0].name=<your-pull-secret>' \
--set credential.existingSecret=fleeger-credential-keyPoint image.registry at the same registry the release published its
per-service images to (registry.example.com/fleeger/<service>:1.0.0-rc1), and add an
imagePullSecrets entry if that registry is private. Leave image.tag unset:
the chart's appVersion is the tag its release pipeline pushed, so each chart
upgrade brings the matching images. A tag you set is stored in the release and
carried into every later upgrade, including --reset-then-reuse-values ones,
which leaves the old images running under the new chart; a release installed
with one clears it once with
helm upgrade … --reset-then-reuse-values --set image.tag=. Everything else — reaching
the UI, TLS, network segmentation — is identical to the checkout install.
A remote agent installs the same way from its own published chart, with the one-time registration token from Onboard (see onboarding a location) in a Secret. Paste it at the prompt; it is not echoed and stays off every command line:
printf 'Registration token: '; stty -echo; IFS= read -r FLEEGER_TOKEN; stty echo; echo
printf '%s' "$FLEEGER_TOKEN" | kubectl create secret generic fleeger-agent-token \
--from-file=registration-token=/dev/stdin
unset FLEEGER_TOKEN
helm install fleeger-agent oci://registry.example.com/fleeger/charts/fleeger-agent \
--version 1.0.0-rc1 \
--set image.repository=registry.example.com/fleeger/spoke-agent \
--set hub.url=https://fleeger.example.com \
--set existingSecret=fleeger-agent-tokenBring your own identity provider
Every core service can be turned off with --set services.<name>.enabled=false
when you supply that capability yourself. The common case is authentication:
front the gateway with your own OIDC provider and disable the bundled auth-svc
with this flag on the helm install in step 3:
--set services.auth-svc.enabled=falseOn a running hub, put it on
helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values
instead.
Release process
The charts and images are cut by GitLab CI on tag pipelines, not by hand:
- Images — CI builds every per-service and UI image with Kaniko and pushes
them to the project's GitLab Container Registry as
${CI_REGISTRY_IMAGE}/<service>:<tag>(services:auth-svc,inventory-svc,gateway-svc,ingestion-svc,alerting-svc,control-svc,spoke-agent,frontend, and thebackuprunner). See container images & GitLab CI. - Charts — the same tag pipeline
helm packages bothfleeger-hubandfleeger-agentandhelm pushes them to the OCI registry undercharts/, sooci://registry.example.com/fleeger/charts/<chart>resolves the chart at--version <chart version>(e.g.1.0.0-rc1).
So a single tag produces the images and both charts; the oci://… installs
above just pull what that pipeline published.
The version comes from the tag
CI derives every published version from the git tag, not from the checked-in
files: in a tag pipeline FLEEGER_VERSION is the tag itself, and the image
tags, the version compiled into the binaries and both charts' version and
appVersion (helm package --version --app-version) are all set from it. The
version/appVersion in Chart.yaml and the workspace Cargo.toml are not
used for anything published, so there is nothing to bump by hand before
tagging. Because image.tag defaults to the chart's appVersion, a published
chart asks for exactly the image tags its own pipeline pushed, and the charts
job waits for the image jobs, so a chart is not published when one of them
failed.
- Tag release candidates on
rcasX.Y.Z-rc.N(novprefix: the tag is used verbatim as the image tag and chart version). The release itself is taggedX.Y.Zby thecreate-releasejob onmaster. - Update the example versions that quote it: the
oci://install snippets above, installing from scratch, onboarding a location, and the two Helm chart references.
Two separate checks, and neither substitutes for the other:
The versions — after the tag pipeline, confirm the published chart renders the tag it was published under in every
image::helm template oci://registry.example.com/fleeger/charts/fleeger-agent \ --version X.Y.Z --set image.repository=registry.example.com/fleeger/spoke-agent | grep 'image:'(The hub chart needs a key to render: add
--set credential.key=$(head -c 32 /dev/urandom | base64), or--set credential.existingSecret=x.)The templates —
just render-chartslints and renders both charts across the value combinations CI covers. It does not look at versions, and it prints no manifests. Installkubeconformfirst or it skips the Kubernetes schema half; the summary line says which mode it ran in.
Deploying with plain manifests (no Helm)
Everything above uses Helm. If you'd rather not — you want to own every object,
run under GitOps, or just avoid installing Helm — deploy the manifests in
Fleeger on Kubernetes — plain manifests with kubectl instead. They're hand-maintained
(not a helm template dump), split so the minimal core is obvious and everything
else is opt-in:
deploy/k8s/
├── base/ the minimal hub — apply this and it runs
├── secrets/ the 5 Secrets you create (never committed)
├── optional/ ingress+TLS, cluster monitoring, network policies, backups, NodePort
└── agent/ a spoke-agent for one location
What the minimal setup needs
The smallest working hub is the base + five Secrets + reachable images:
kubectland a cluster — the only tool required (no Helm, no cluster-sidensc).- Images reachable from the cluster — locally-loaded
fleeger/*:local, or a registry you repoint the images at (one place:base/kustomization.yaml, or edit the manifests — see Fleeger on Kubernetes — plain manifests). - Five Secrets (Secrets the hub needs):
fleeger-nats-auth(fromdeploy/nats/gen-auth.sh),fleeger-credential-key,fleeger-db,fleeger-internal,fleeger-victoriametrics(VictoriaMetrics'delete-keyandsnapshot-key). Unlike Helm, none are generated for you — that's the trade for plain manifests, and it keeps secrets out of git. - The base — dependencies, the core services, the UI + edge, and the one-shot
fleeger-nats-pushJob.
Required components are SurrealDB, NATS, VictoriaMetrics, auth-svc,
gateway-svc, inventory-svc, ingestion-svc, the frontend + edge, and the
NATS-push Job. alerting-svc (alerting) and control-svc (control actions) are
optional — delete their Service + Deployment from base/40-app-services.yaml if
you don't use those features. auth-svc can go only if you front the gateway with
your own OIDC (see Bring your own identity provider);
on the manifest path that means deleting auth-svc and dropping gateway-svc's
wait-for-auth init container plus pointing its config at the tokens your provider
issues — not a drop-in change.
Steps
# 1. Namespace
kubectl create namespace fleeger
# 2. Secrets — full commands in "Secrets the hub needs"
deploy/nats/gen-auth.sh
kubectl -n fleeger create secret generic fleeger-nats-auth --from-file=... # see "secrets"
head -c 32 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=/dev/stdin
d=$(mktemp -d) && openssl rand -hex 32 | tr -d '\n' >"$d/delete-key" && openssl rand -hex 32 | tr -d '\n' >"$d/snapshot-key" &&
kubectl -n fleeger create secret generic fleeger-victoriametrics --from-file="$d"; rm -rf "$d"
# 3. The hub
kubectl -n fleeger apply -f deploy/k8s/base/ # or: kubectl apply -k deploy/k8s/base
# 4. Watch it settle
kubectl -n fleeger get pods -w
# 5. First admin user (same as the Helm path)
kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service \
user add -u admin -e admin@example.com -r admin # prompts for the password
# 6. Reach the UI (quick check)
kubectl -n fleeger port-forward svc/edge 8080:80 # http://localhost:8080The fleeger-nats-auth Secret must include all the *.creds files
(ingestion, gateway, alerting, inventory, control, core-agent) — a
service whose cred is missing won't start, since its volume references the key by
name. The exact --from-file list is in
Secrets the hub needs.
The options (everything past the minimum)
Each is a file (or folder) in Optional add-ons (manifests) you apply à la carte; read the note at the top of each before applying.
- Ingress + TLS (
optional/ingress.yaml) — the recommended way to expose Fleeger. Do this before exposing it past a trusted LAN: login and agent traffic must be HTTPS. Then set[nats] public_url = "wss://<host>/nats"inauth-svc.toml(base/10-config.yaml) so agents get the external endpoint at bootstrap. The edge answers everything coming through the Ingress with a 503 untilset_real_ip_fromin thefleeger-edgeConfigMap names the ingress controller: the raw-manifest counterpart of the chart'sedge.trustedProxiescheck (see Fleeger on Kubernetes — plain manifests). - NodePort access (
optional/edge-nodeport.yaml) — quick LAN/homelab access with no ingress controller (plaintext HTTP; trusted networks only). - Monitoring the hub's own cluster (
optional/cluster-monitoring/) — this is the service account to monitor the cluster from inside it. It's the bundled core-agent plus a ServiceAccount + read-only ClusterRole/Binding; ak8s_clusterresource with an empty credential then collects the cluster via that ServiceAccount (no kubeconfig). Covered in more depth under Monitoring the hub's own cluster below. - Network segmentation (
optional/network-policies.yaml) — default-deny ingress + per-service allow rules. CNI-specific and untested; validate first. - Scheduled backups (
optional/backup-cronjob.yaml) — nightly SurrealDB dumps to S3 + the backup-success metric. See Backup & disaster recovery. - A remote agent (Running a spoke-agent with plain manifests) — a spoke-agent for a location, with optional cluster RBAC if it monitors the cluster it runs in.
Operating the manifest install
The Operating notes below apply, with the Helm verbs swapped for
kubectl: change a value in a ConfigMap and kubectl -n fleeger rollout restart deploy/<service> (the service Deployments have no config-checksum auto-roll);
upgrade by editing the image tags (or base/kustomization.yaml) and
re-applying. NATS and the edge do roll on their own when a release changes
their config (nats.conf, or the edge's default.conf and
fleeger-upstreams.envsh): their Deployments carry a checksum/nats-conf or
checksum/edge-conf annotation over it, as the chart's do, and CI keeps it
current. An edit of your own to those ConfigMaps, in place or through
a kustomize overlay patch, does not change the annotation, so restart
deploy/nats or deploy/edge yourself, or update the annotation with the
command in the comment beside it (POSIX shell; on macOS use shasum -a 256
for sha256sum; copy only the hex digest into the quoted value). The nats-push Job is
immutable, so kubectl -n fleeger delete job fleeger-nats-push before re-running
it.
Rotating the credential key is the exception: the chart's
credential.retiredKeysSecret and credential.allowKeyChange have no
manifest equivalent. The manifests read the retired key(s) from an optional
Secret, fleeger-credential-keys-retired, instead, and a changed Secret
restarts nothing, so follow the manifest steps in
Secrets the hub needs
rather than the Helm bullet below.
inventory-svc runs as one pod
inventory-svc has replicas: 1 and strategy: Recreate, in the chart and in
deploy/k8s. A rollout stops the old pod first; the new pod's init
container then runs db migrate (it merges duplicate discovered resources,
applies the migrations, checks the discovery key and defines again what
differs from the migrations; that check
replays every migration into a scratch database it creates and removes, a
few seconds on each start of every service's pod), and only then does the
new pod start. This keeps two rules:
- No migration runs beside a writing inventory-svc. Migration 0040 builds a
UNIQUEindex over the committed data, and a discovery report committing while it builds can be missing from the index for good, which lets the next report of that object create a second row. - A rollout never has two inventory-svc pods writing. Its per-location lanes, which keep one location's reports in order, hold within one process only.
Recreate covers rollouts only. A pod deleted by hand or evicted, or one on a
node gone NotReady, can still run beside its replacement for a while, so
restart inventory-svc with kubectl -n fleeger rollout restart deploy/inventory-svc rather than by deleting its pod. Such an overlap cannot
duplicate a discovered resource (the UNIQUE discovery key refuses the second
row); it only runs a location's reports in two processes for that time.
The price is a gap on every rollout of inventory-svc, typically seconds, one
that only changes its configuration included (the chart's config checksum
annotation, or with the manifests a kubectl rollout restart, restarts the
pod). During it:
- the inventory API in the UI answers 502/503;
- every gateway request to
/api/resources/{id}/…or/api/credentials/{id}whose location the gateway has not cached fails with 502/503, as the gateway asks inventory-svc for it (services/gateway-svc/src/resource_scope.rs); - control actions fail, as control-svc looks the resource up in inventory-svc;
- that cycle's agent reports (discovery, hardware, collect status, the resource list) are dropped, not queued, and show as a collect error.
If db migrate fails, the gap lasts until it succeeds (the init container
retries it, then the pod crash-loops). The built-in workload_degraded rule
ships disabled, and sees the hub's own Deployments only if the hub's cluster is
monitored by an agent; enabled, it fires after its 10-minute window, so a
normal rollout does not trip it. Do not patch the Deployment back to
RollingUpdate or scale it up: a second writer (a scaled-up Deployment, or one
patched to RollingUpdate) brings the hazards above back. A GitOps tool or
server-side apply that keeps the old rollingUpdate block fails with
"rollingUpdate may not be specified when strategy type is Recreate": patch it
to rollingUpdate: null.
Reaching agents
Agents run at your locations, not in this cluster, and reach the hub over a
single endpoint: they bootstrap over HTTPS, then do metrics and inventory
over the NATS WebSocket (wss://<hub>/nats). Nothing else — raw NATS,
inventory-svc — is ever exposed. See
Agent credentials & revocation for the trust
model.
To make that endpoint reachable from outside the cluster:
- Front the edge with an Ingress, WebSocket-capable (the common controllers
are). With Helm that is the three flags in step 6,
including
edge.trustedProxies, which the chart will not render without when an ingress is enabled. The edge already routes/api→ gateway and/nats→ the NATS WebSocket, so one host serves everything. - Tell agents that external URL:
--set nats.publicUrl=wss://fleeger.example.com/nats(what they receive at bootstrap; on a running hub, withhelm upgrade … --reset-then-reuse-values). Then onboard with--set hub.url=https://fleeger.example.comon the agent chart.
An agent that runs inside the same cluster (monitoring that cluster) can use
the in-cluster default and, with --set rbac.create=true on the agent chart, its
ServiceAccount for read-only node/pod metrics.
Monitoring the hub's own cluster
The hub can watch the very cluster it runs in with the bundled core agent
(--set coreAgent.enabled=true) — no separate agent chart. Two things make that
work:
- Leave the
k8s_clusterresource credential empty. With no kubeconfig set, the collector falls back to in-cluster access via its ServiceAccount (kube::Client::try_default()). Do not paste the external admin kubeconfig: its endpoint is a control-plane public IP that a pod on a firewalled cluster can't reach, and it's far more privileged than the collector needs. - Let the chart create the RBAC.
coreAgent.rbac.create(andcoreAgent.serviceAccount.create) are on by default whencoreAgent.enabled=trueand create the ServiceAccount + a read-only ClusterRole + binding. The role covers the full set the k8s collector discovers — nodes/pods/namespaces/services/PVCs, deployments/statefulsets/daemonsets, jobs/cronjobs, ingresses, HPAs, metrics, and cert-manager CRDs — not just node/pod metrics.
With plain manifests this is the optional/cluster-monitoring/ add-on: apply
core-agent-rbac.yaml (the ServiceAccount + read-only ClusterRole/Binding) then
core-agent.yaml (the core-agent Deployment). Make sure core-agent.creds is in
the fleeger-nats-auth Secret. See
Monitoring the hub's own cluster (bundled core-agent).
Operating
These notes are written for the Helm install. For the manifest install the same ideas apply with
kubectlverbs — see Operating the manifest install.
- Config is templated from Helm values into a ConfigMap; change values with
helm upgrade … --reset-then-reuse-values --set …(without that flag, an upgrade that passes values resets every value it does not repeat — see the Helm chart reference's Upgrade section). Deployments roll on config changes (checksum annotation). - Upgrades — the Helm chart reference's Upgrade commands, which pass no values and so keep the release's own. Migrations re-run idempotently in the init containers; the nats-push Job re-runs.
- inventory-svc runs as one pod, recreated on a rollout (see inventory-svc runs as one pod): every rollout of it has a short gap.
- Backups — Backup & disaster recovery covers SurrealDB
- VictoriaMetrics; run it against the PVCs (e.g. from a Job or
kubectl exec).
- VictoriaMetrics; run it against the PVCs (e.g. from a Job or
- Logs —
kubectl -n fleeger logs deploy/<service>. - Rotating the credential key — set
credential.key/existingSecretto the new key, create a Secret with aretired-keysentry (the old key) and point--set credential.retiredKeysSecretat it, and upgrade withhelm upgrade … --reset-then-reuse-values(adding--set credential.allowKeyChange=trueif you changedcredential.key), then re-encrypt in both services that hold secrets:kubectl -n fleeger exec deploy/inventory-svc -- /usr/local/bin/service rekey(credentials) andkubectl -n fleeger exec deploy/alerting-svc -- /usr/local/bin/service rekey(notification-channel secrets). A secret edited while either runs keeps the edit (see the compose walkthrough). Each run is journalled in the audit log (secrets.rekey, never a key) before it re-encrypts anything, and refuses when the log cannot take the row. Once every replica of both runs with the new key (kubectl -n fleeger rollout status deploy/inventory-svc, then the same fordeploy/alerting-svc; a replica still on the old key seals what it writes under the retired one), runrekeyagain in each. Once that run reports everything already current, clear the retired key and re-arm the key-change refusal:helm upgrade … --reset-then-reuse-values --set credential.retiredKeysSecret= --set credential.allowKeyChange=false. Keep the old key, off-box, for as long as you keep any backup taken before both rekeys were confirmed: restoring one needs it back incredential.retiredKeysSecret, and bothrekeyruns again afterwards (Backup & disaster recovery). (Same flow as the compose walkthrough in Deploying Fleeger; for the manifest install, see Secrets the hub needs.)