// Deployment-Referenz
Fleeger on Kubernetes — plain manifests
Deploy Fleeger with kubectl and nothing else. This is the Helm-free path:
hand-maintained manifests split into a minimal base plus opt-in add-ons, so you
can see exactly what each piece does and add only what you need.
Prefer Helm (templated values, one-command upgrades)? Use the charts in
charts/ instead — see
Deploying Fleeger on Kubernetes.
The charts and these manifests describe the same system; pick one, don't mix them
for the same install.
deploy/k8s/
├── base/ the minimal hub — apply this and it runs
├── secrets/ the 5 Secrets you must create (never committed)
├── optional/ ingress+TLS, cluster monitoring, network policies, backups, NodePort
└── agent/ a spoke-agent for one location
What the minimal setup needs
A working hub is: the base manifests + the five Secrets + a container registry (or locally-loaded images). Concretely:
- A cluster and
kubectlpointed at it. That's the only tool required — no Helm, nonscon the cluster. - Images reachable from the cluster. Either build locally and load
fleeger/*:localonto the nodes, or push to a registry and repoint the images (one place:base/kustomization.yaml, or edit the manifests — see below). - The five Secrets (see Secrets the hub needs). Four are quick to
generate; the fifth, the NATS auth material, is produced once locally by
deploy/nats/gen-auth.sh(it needsnsc, which the cluster doesn't). A sixth,fleeger-credential-keys-retired, exists only while you rotate the credential key; the services read it as optional (see Secrets the hub needs). - The base manifests — dependencies (SurrealDB, NATS, VictoriaMetrics), the core services, the UI + edge, and a one-shot Job that registers the NATS account.
Everything else — exposing it over TLS, monitoring the cluster itself, network segmentation, backups, remote agents — is optional, in Optional add-ons (manifests) and Running a spoke-agent with plain manifests.
Which services are actually required?
| Component | Required? |
|---|---|
| SurrealDB, NATS, VictoriaMetrics | yes — the backing stores + transport |
| auth-svc | yes (unless you front the gateway with your own OIDC) |
| gateway-svc, inventory-svc, ingestion-svc | yes — API, inventory, metric intake |
| frontend + edge | yes — the UI and the single entry point |
fleeger-nats-push Job |
yes — one-shot, registers the NATS account |
| alerting-svc | optional — only if you use alerting |
| control-svc | optional — only if you use control actions (Proxmox/iDRAC/Kubernetes) |
To drop an optional service, delete its Service + Deployment from
base/40-app-services.yaml (its config block in base/10-config.yaml is
harmless to leave). See "Bring your own identity provider" in the
deployment doc for dropping
auth-svc.
Quick start
# 1. Namespace
kubectl create namespace fleeger
# 2. Secrets (see "Secrets the hub needs" for the full commands)
deploy/nats/gen-auth.sh
kubectl -n fleeger create secret generic fleeger-nats-auth --from-file=... # see "Secrets the hub needs"
head -c 32 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=/dev/stdin
d=$(mktemp -d) && openssl rand -hex 32 | tr -d '\n' >"$d/delete-key" &&
openssl rand -hex 32 | tr -d '\n' >"$d/snapshot-key" &&
kubectl -n fleeger create secret generic fleeger-victoriametrics --from-file="$d"
rm -rf "$d"
# 3. The hub
kubectl -n fleeger apply -f deploy/k8s/base/ # or: kubectl apply -k deploy/k8s/base
# 4. Watch it settle
kubectl -n fleeger get pods -w
# 5. First admin user (journalled in the audit log first: once
# inventory-svc is up, so its audit database is migrated)
kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service \
user add -u admin -e admin@example.com -r admin # prompts for the password
# 6. Reach the UI (quick check)
kubectl -n fleeger port-forward svc/edge 8080:80 # http://localhost:8080The full walkthrough — including TLS, exposing to agents, and day-2 operations — is in Deploying Fleeger on Kubernetes.
Telling the edge who the client is (set_real_ip_from)
The edge keys its /api/auth/login rate limit on the connection's peer address,
and hands that address to the gateway as X-Real-IP (recorded in the control
audit log) and X-Forwarded-For. Out of the box it trusts no forwarded header: a request carrying
X-Forwarded-For or X-Real-IP gets a 503 (see below), so nobody can pick their
own source address or what the audit trail records about them.
That default is right only while clients reach the edge directly —
optional/edge-nodeport.yaml or kubectl port-forward on a trusted network. Put
an Ingress or a cloud LB in front (optional/ingress.yaml does) and every request
arrives from that one hop, so:
- the login limit becomes one global bucket — 10 attempts a minute for the entire deployment, which an unauthenticated attacker can exhaust to lock every operator out; and
- the audit log records the proxy's address for every action.
The fix is to name the hop. The Helm chart exposes it as edge.trustedProxies;
here you edit the nginx config directly — in the fleeger-edge ConfigMap in
base/10-config.yaml, uncomment the set_real_ip_from /
real_ip_header / real_ip_recursive block and list the CIDRs of only that
hop (your ingress controller's pod CIDR, or the LB's source range — trusting
anything wider lets whoever can reach the edge forge an address). The block is
written out with examples where it belongs in the file.
A ConfigMap edit does not restart anything, so finish with a roll:
kubectl -n fleeger apply -f deploy/k8s/base/10-config.yaml
kubectl -n fleeger rollout restart deploy/edgeUntil the hop is trusted, everything through it gets a 503. A request that
carries X-Forwarded-For or X-Real-IP came through a proxy; if the edge did
not take the client's address from it, the proxy is not covered by
set_real_ip_from (not set, or a CIDR that misses it), and the edge answers 503
with what to set. So an Ingress applied without it, or with a wrong CIDR, cannot
silently turn the login limit into one bucket; the Helm chart refuses the same
setup when it renders. This also stops remote agents connecting through
wss://<host>/nats, and auth-svc when its [nats] public_url is that URL (then
revoking or offboarding an agent fails). Requests without either header, such as
kubectl port-forward, a NodePort, in-cluster clients and most load-balancer
health checks, are served as before.
Things to know:
- Keep the directives at server level, where they are in the file (http
level works too). Inside a
locationthey don't reach the check, and every proxied request gets the 503. So does a trusted proxy that does not send the headerreal_ip_headernames. - A proxy that sets neither header (a plain TCP load balancer, or one that sends
only
ForwardedorCF-Connecting-IP) is not noticed, and the edge sees only its address, as before. Use an HTTP-level balancer, or configure the PROXY protocol withset_real_ip_fromyourself. - A client that sends one of these headers itself, for example through a forward proxy in your office, gets the 503 when it reaches the edge directly.
- If kube-proxy rewrites client addresses before the ingress controller sees
them (
externalTrafficPolicy: Clusteron its Service), the controller only knows node addresses. The check then passes, but every node is one login bucket. Set the controller's Service toexternalTrafficPolicy: Local. - A service mesh (Istio, Linkerd) with a sidecar in the edge pod hides the
ingress controller: the edge's peer is the sidecar's loopback address
(
127.0.0.6with Istio,127.0.0.1with Linkerd), so a trusted controller CIDR still gets the 503. Do not trust the loopback address to get past it; that trusts every caller the sidecar accepts. Keep the edge pod out of the mesh instead, by addingsidecar.istio.io/inject: "false"to the labels, orlinkerd.io/inject: disabledto the annotations, of the edge Deployment's pod template inbase/50-frontend-edge.yaml. An edge outside the mesh reaches gateway, auth-svc and NATS without mesh mTLS, so if the mesh enforces it (IstioSTRICTPeerAuthentication, Linkerd'sall-authenticatedpolicy) those services must accept plaintext from the edge. Alternatively keep the sidecar and bypass it only for the edge's inbound port 80 (annotationtraffic.sidecar.istio.io/excludeInboundPorts: "80"orconfig.linkerd.io/skip-inbound-ports: "80"); with Istio, a meshed ingress controller then also needs mTLS turned off towards that port. - A Cloudflare Tunnel with
cloudflaredrunning in the cluster and its ingress rule pointing at the edge Service makescloudflaredthe hop: trust its pods' addresses (the pod CIDR, or a narrower range they are pinned to) and setreal_ip_header CF-Connecting-IP. If the rule points at an ingress controller instead, that controller is the edge's hop as usual, and it is the controller that has to trustcloudflared. Cloudflare's published IP ranges are the hop only when Cloudflare's proxy connects to the edge itself, through a load balancer that keeps the source address; with a tunnel, trusting them leaves the 503 in place.
Pointing images at your registry
The manifests default to fleeger/<service>:local. To use a registry:
- With kustomize: edit the
images:block inbase/kustomization.yaml(one place for all of them), thenkubectl apply -k deploy/k8s/base. - Plain
kubectl apply -f: rewrite the image references, e.g.If the registry is private, add ansed -i 's#fleeger/#registry.example.com/fleeger/#; s#:local#:0.5.0#' \ deploy/k8s/base/*.yaml deploy/k8s/optional/**/*.yaml deploy/k8s/agent/*.yamlimagePullSecrets:entry to each pod spec (or to the namespace's default ServiceAccount).
The frontend and edge pods run nginx as uid/gid 101 with no capabilities,
writing only to emptyDirs at /var/cache/nginx and /run. The frontend image
must therefore be built from a Dockerfile.frontend that makes its nginx config
world-readable (this tree's does). A frontend image built by an older one from a
group-only checkout fails with
open() "/etc/nginx/conf.d/default.conf" failed (13: Permission denied): rebuild
it.
Upgrading
Read the notes for the release you are upgrading to under Upgrading a running Fleeger hub to 1.0.0-rc.5 first; they give each step in order. Two apply to every upgrade from before 1.0.0-rc1:
- The NATS volume grew to 3Gi (
nats-datainbase/20-storage.yaml).kubectl applygrows the claim in place on a StorageClass withallowVolumeExpansion: true. On one without it, it refuses that claim but applies the rest, and NATS restarts at once with the larger JetStream store on the old claim, which turns JetStream off for the agents' account. Migrate the claim before applying; the Helm chart reference's NATS volume section shows how a claim is migrated. - Bound the ingestion streams once. After the upgrade, with
ingestion-svcrunning:The Helm chart reference's Ingestion streams section covers what it does and when it refuses.kubectl -n fleeger exec deploy/ingestion-svc -- /usr/local/bin/service stream migrate
auth-svc re-asserts agent revocations at the NATS resolver every 10 seconds, so a resolver that lost its store re-admits offboarded agents only until its next pass, and for as long as auth-svc is down.
A note on namespaces
The manifests don't hard-code a namespace: on each object — apply them with
-n fleeger (or kubectl apply -k, which sets it). If you use a different
namespace, update the ClusterRoleBinding subject namespace in
optional/cluster-monitoring/core-agent-rbac.yaml to match.