Zum Inhalt springen
Produkte

// Deployment-Referenz

Fleeger on Kubernetes — plain manifests

Deploy Fleeger with kubectl and nothing else. This is the Helm-free path: hand-maintained manifests split into a minimal base plus opt-in add-ons, so you can see exactly what each piece does and add only what you need.

Prefer Helm (templated values, one-command upgrades)? Use the charts in charts/ instead — see Deploying Fleeger on Kubernetes. The charts and these manifests describe the same system; pick one, don't mix them for the same install.

deploy/k8s/
├── base/        the minimal hub — apply this and it runs
├── secrets/     the 5 Secrets you must create (never committed)
├── optional/    ingress+TLS, cluster monitoring, network policies, backups, NodePort
└── agent/       a spoke-agent for one location

What the minimal setup needs

A working hub is: the base manifests + the five Secrets + a container registry (or locally-loaded images). Concretely:

  1. A cluster and kubectl pointed at it. That's the only tool required — no Helm, no nsc on the cluster.
  2. Images reachable from the cluster. Either build locally and load fleeger/*:local onto the nodes, or push to a registry and repoint the images (one place: base/kustomization.yaml, or edit the manifests — see below).
  3. The five Secrets (see Secrets the hub needs). Four are quick to generate; the fifth, the NATS auth material, is produced once locally by deploy/nats/gen-auth.sh (it needs nsc, which the cluster doesn't). A sixth, fleeger-credential-keys-retired, exists only while you rotate the credential key; the services read it as optional (see Secrets the hub needs).
  4. The base manifests — dependencies (SurrealDB, NATS, VictoriaMetrics), the core services, the UI + edge, and a one-shot Job that registers the NATS account.

Everything else — exposing it over TLS, monitoring the cluster itself, network segmentation, backups, remote agents — is optional, in Optional add-ons (manifests) and Running a spoke-agent with plain manifests.

Which services are actually required?

Component Required?
SurrealDB, NATS, VictoriaMetrics yes — the backing stores + transport
auth-svc yes (unless you front the gateway with your own OIDC)
gateway-svc, inventory-svc, ingestion-svc yes — API, inventory, metric intake
frontend + edge yes — the UI and the single entry point
fleeger-nats-push Job yes — one-shot, registers the NATS account
alerting-svc optional — only if you use alerting
control-svc optional — only if you use control actions (Proxmox/iDRAC/Kubernetes)

To drop an optional service, delete its Service + Deployment from base/40-app-services.yaml (its config block in base/10-config.yaml is harmless to leave). See "Bring your own identity provider" in the deployment doc for dropping auth-svc.

Quick start

# 1. Namespace
kubectl create namespace fleeger

# 2. Secrets (see "Secrets the hub needs" for the full commands)
deploy/nats/gen-auth.sh
kubectl -n fleeger create secret generic fleeger-nats-auth --from-file=... # see "Secrets the hub needs"
head -c 32 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n' | kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=/dev/stdin
d=$(mktemp -d) && openssl rand -hex 32 | tr -d '\n' >"$d/delete-key" &&
  openssl rand -hex 32 | tr -d '\n' >"$d/snapshot-key" &&
  kubectl -n fleeger create secret generic fleeger-victoriametrics --from-file="$d"
rm -rf "$d"

# 3. The hub
kubectl -n fleeger apply -f deploy/k8s/base/     # or: kubectl apply -k deploy/k8s/base

# 4. Watch it settle
kubectl -n fleeger get pods -w

# 5. First admin user (journalled in the audit log first: once
#    inventory-svc is up, so its audit database is migrated)
kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service \
  user add -u admin -e admin@example.com -r admin   # prompts for the password

# 6. Reach the UI (quick check)
kubectl -n fleeger port-forward svc/edge 8080:80   # http://localhost:8080

The full walkthrough — including TLS, exposing to agents, and day-2 operations — is in Deploying Fleeger on Kubernetes.

Telling the edge who the client is (set_real_ip_from)

The edge keys its /api/auth/login rate limit on the connection's peer address, and hands that address to the gateway as X-Real-IP (recorded in the control audit log) and X-Forwarded-For. Out of the box it trusts no forwarded header: a request carrying X-Forwarded-For or X-Real-IP gets a 503 (see below), so nobody can pick their own source address or what the audit trail records about them.

That default is right only while clients reach the edge directly — optional/edge-nodeport.yaml or kubectl port-forward on a trusted network. Put an Ingress or a cloud LB in front (optional/ingress.yaml does) and every request arrives from that one hop, so:

  • the login limit becomes one global bucket — 10 attempts a minute for the entire deployment, which an unauthenticated attacker can exhaust to lock every operator out; and
  • the audit log records the proxy's address for every action.

The fix is to name the hop. The Helm chart exposes it as edge.trustedProxies; here you edit the nginx config directly — in the fleeger-edge ConfigMap in base/10-config.yaml, uncomment the set_real_ip_from / real_ip_header / real_ip_recursive block and list the CIDRs of only that hop (your ingress controller's pod CIDR, or the LB's source range — trusting anything wider lets whoever can reach the edge forge an address). The block is written out with examples where it belongs in the file.

A ConfigMap edit does not restart anything, so finish with a roll:

kubectl -n fleeger apply -f deploy/k8s/base/10-config.yaml
kubectl -n fleeger rollout restart deploy/edge

Until the hop is trusted, everything through it gets a 503. A request that carries X-Forwarded-For or X-Real-IP came through a proxy; if the edge did not take the client's address from it, the proxy is not covered by set_real_ip_from (not set, or a CIDR that misses it), and the edge answers 503 with what to set. So an Ingress applied without it, or with a wrong CIDR, cannot silently turn the login limit into one bucket; the Helm chart refuses the same setup when it renders. This also stops remote agents connecting through wss://<host>/nats, and auth-svc when its [nats] public_url is that URL (then revoking or offboarding an agent fails). Requests without either header, such as kubectl port-forward, a NodePort, in-cluster clients and most load-balancer health checks, are served as before.

Things to know:

  • Keep the directives at server level, where they are in the file (http level works too). Inside a location they don't reach the check, and every proxied request gets the 503. So does a trusted proxy that does not send the header real_ip_header names.
  • A proxy that sets neither header (a plain TCP load balancer, or one that sends only Forwarded or CF-Connecting-IP) is not noticed, and the edge sees only its address, as before. Use an HTTP-level balancer, or configure the PROXY protocol with set_real_ip_from yourself.
  • A client that sends one of these headers itself, for example through a forward proxy in your office, gets the 503 when it reaches the edge directly.
  • If kube-proxy rewrites client addresses before the ingress controller sees them (externalTrafficPolicy: Cluster on its Service), the controller only knows node addresses. The check then passes, but every node is one login bucket. Set the controller's Service to externalTrafficPolicy: Local.
  • A service mesh (Istio, Linkerd) with a sidecar in the edge pod hides the ingress controller: the edge's peer is the sidecar's loopback address (127.0.0.6 with Istio, 127.0.0.1 with Linkerd), so a trusted controller CIDR still gets the 503. Do not trust the loopback address to get past it; that trusts every caller the sidecar accepts. Keep the edge pod out of the mesh instead, by adding sidecar.istio.io/inject: "false" to the labels, or linkerd.io/inject: disabled to the annotations, of the edge Deployment's pod template in base/50-frontend-edge.yaml. An edge outside the mesh reaches gateway, auth-svc and NATS without mesh mTLS, so if the mesh enforces it (Istio STRICT PeerAuthentication, Linkerd's all-authenticated policy) those services must accept plaintext from the edge. Alternatively keep the sidecar and bypass it only for the edge's inbound port 80 (annotation traffic.sidecar.istio.io/excludeInboundPorts: "80" or config.linkerd.io/skip-inbound-ports: "80"); with Istio, a meshed ingress controller then also needs mTLS turned off towards that port.
  • A Cloudflare Tunnel with cloudflared running in the cluster and its ingress rule pointing at the edge Service makes cloudflared the hop: trust its pods' addresses (the pod CIDR, or a narrower range they are pinned to) and set real_ip_header CF-Connecting-IP. If the rule points at an ingress controller instead, that controller is the edge's hop as usual, and it is the controller that has to trust cloudflared. Cloudflare's published IP ranges are the hop only when Cloudflare's proxy connects to the edge itself, through a load balancer that keeps the source address; with a tunnel, trusting them leaves the 503 in place.

Pointing images at your registry

The manifests default to fleeger/<service>:local. To use a registry:

  • With kustomize: edit the images: block in base/kustomization.yaml (one place for all of them), then kubectl apply -k deploy/k8s/base.
  • Plain kubectl apply -f: rewrite the image references, e.g.
    sed -i 's#fleeger/#registry.example.com/fleeger/#; s#:local#:0.5.0#' \
      deploy/k8s/base/*.yaml deploy/k8s/optional/**/*.yaml deploy/k8s/agent/*.yaml
    If the registry is private, add an imagePullSecrets: entry to each pod spec (or to the namespace's default ServiceAccount).

The frontend and edge pods run nginx as uid/gid 101 with no capabilities, writing only to emptyDirs at /var/cache/nginx and /run. The frontend image must therefore be built from a Dockerfile.frontend that makes its nginx config world-readable (this tree's does). A frontend image built by an older one from a group-only checkout fails with open() "/etc/nginx/conf.d/default.conf" failed (13: Permission denied): rebuild it.

Upgrading

Read the notes for the release you are upgrading to under Upgrading a running Fleeger hub to 1.0.0-rc.5 first; they give each step in order. Two apply to every upgrade from before 1.0.0-rc1:

  • The NATS volume grew to 3Gi (nats-data in base/20-storage.yaml). kubectl apply grows the claim in place on a StorageClass with allowVolumeExpansion: true. On one without it, it refuses that claim but applies the rest, and NATS restarts at once with the larger JetStream store on the old claim, which turns JetStream off for the agents' account. Migrate the claim before applying; the Helm chart reference's NATS volume section shows how a claim is migrated.
  • Bound the ingestion streams once. After the upgrade, with ingestion-svc running:
    kubectl -n fleeger exec deploy/ingestion-svc -- /usr/local/bin/service stream migrate
    The Helm chart reference's Ingestion streams section covers what it does and when it refuses.

auth-svc re-asserts agent revocations at the NATS resolver every 10 seconds, so a resolver that lost its store re-admits offboarded agents only until its next pass, and for as long as auth-svc is down.

A note on namespaces

The manifests don't hard-code a namespace: on each object — apply them with -n fleeger (or kubectl apply -k, which sets it). If you use a different namespace, update the ClusterRoleBinding subject namespace in optional/cluster-monitoring/core-agent-rbac.yaml to match.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.