Zum Inhalt springen
Produkte

// Deployment-Referenz

Helm chart: fleeger-hub

Helm chart for the Fleeger hub — the control plane and admin UI. Chart version 1.0.0-rc1 (appVersion 1.0.0-rc1).

Install one hub release per namespace: service names are fixed (surrealdb, nats, auth-svc, …) so the services resolve each other by in-namespace DNS.

What it deploys

  • Core services — auth-svc, inventory-svc, ingestion-svc, alerting-svc, control-svc, gateway-svc. Each can be toggled off individually (see Toggling services).
  • Frontend + edge — the admin UI and an nginx edge that serves it and proxies /api to the gateway (and exposes ws://edge/nats).
  • Bundled dependencies — SurrealDB (v3.3.4), NATS/JetStream (2.15), VictoriaMetrics (v1.153.0).
  • Optional — a bundled "core" agent (coreAgent), scheduled backups + a VictoriaMetrics vmbackup sidecar (backup), an ingress, and NetworkPolicies.

VictoriaMetrics is the metrics store. Log aggregation is off by default — this chart deploys no Loki and no log shipper (only the compose stack does). Point loki.url at a Loki you run yourself to enable per-stream dead-log purging, pod-log ingestion and the Observability Explorer's log tab; leave it empty and those features stay cleanly disabled (/api/logs/* returns 404).

Prerequisites

A Kubernetes cluster with a default StorageClass (or set *.persistence.storageClass), Helm 3.14 or later (the upgrade recipes below use --reset-then-reuse-values, which 3.14 added), and the following Secrets in the target namespace:

Secret Required? How to create it
Master credential key Required A base64 32-byte key encrypting stored infra credentials and alert-channel configs (standard base64 of exactly 32 bytes, as the recipe below makes; a credential.key of any other shape is refused at render). Create it as a Secret (key credential-key) and point credential.existingSecret at it, as the install commands below do. On a real install or upgrade the chart checks that Secret's key when the Secret already exists (credential.preflightExistingSecret; helm template, client-side --dry-run and Argo CD cannot look it up). --set credential.key=… also works, but puts the key on the helm command line, readable by every local user with ps, and in the release's stored values. BACK IT UP — losing it makes every encrypted credential unrecoverable.
NATS auth material Required Generated by deploy/nats/gen-auth.sh — the per-service creds (ingestion.creds, gateway.creds, alerting.creds, inventory.creds, control.creds), plus core-agent.creds when coreAgent.enabled=true, plus sys.creds, operator.nk, spokes-account.nk, SPOKES.jwt, resolver.conf. All of them — each is referenced by key name in a volume projection, so a missing one leaves that pod crash-looping (or stuck in ContainerCreating). The chart cannot generate it (needs nsc) but does preflight it on install (nats.preflightAuthSecret). Loaded as the Secret named by nats.authSecret (default fleeger-nats-auth). See Deploying Fleeger on Kubernetes for the one-liner.
DB password Optional Leave surrealdb.password/existingSecret empty to have the chart generate a strong password into a Secret (kept across upgrades; not under Argo CD or helm template, see GitOps), or pin your own in a Secret (key db-password) referenced by surrealdb.existingSecret; --set surrealdb.password=… puts it on the helm command line. A pinned password takes effect only on a new datastore.
S3 credentials Optional Only if backup.enabled=true. A Secret with keys access-key-id, secret-access-key, referenced by backup.existingSecret.
Backup signing key Optional Only if backup.enabled=true. An Ed25519 private key in PEM (key signing-key) that only the backup CronJob mounts and signs every backup set with: openssl genpkey -algorithm ed25519 | kubectl -n <ns> create secret generic fleeger-backup-signing --from-file=signing-key=/dev/stdin, referenced by backup.signing.existingSecret. Empty ⇒ the chart generates one into <fullname>-backup-signing (kept across upgrades; not in a client-side render, see GitOps). Its public key goes into backup.signing.publicKeys (the install notes print the command) and, as a copy, somewhere off the hub: scripts/restore.sh restores only sets whose signature verifies under one of them.

Install

From a checkout, with the fleeger/*:local images built from it and loaded into the nodes:

kubectl create namespace fleeger
# The master key goes through a pipe, never onto a command line. BACK IT UP.
head -c 32 /dev/urandom | base64 | tr -d '\n' | \
  kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin
helm install fleeger charts/fleeger-hub -n fleeger \
  --set image.tag=local \
  --set credential.existingSecret=fleeger-credential-key

From the OCI registry (the project's GitLab container registry — replace registry.example.com/fleeger with your ${CI_REGISTRY_IMAGE}):

helm install fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
  --version 1.0.0-rc1 -n fleeger \
  --set image.registry=registry.example.com/fleeger \
  --set credential.existingSecret=fleeger-credential-key   # the Secret created above

Leave image.tag unset: a published chart runs the images its own release pipeline pushed (its appVersion), so each chart upgrade brings the matching images with it. A tag you set is stored in the release and carried into every later upgrade, which would leave the old images running under the new chart.

The NATS auth Secret must already exist in the namespace before install. A post-install nats-push Job loads the SPOKES account into the resolver, after which the services finish authenticating and become Ready. Create the first admin with kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service user add …, which prompts for the password (see the post-install notes).

Upgrade

# from a checkout
helm upgrade fleeger charts/fleeger-hub -n fleeger

# from OCI
helm upgrade fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
  --version 1.0.0-rc1 -n fleeger

These pass no values, and that is what makes them safe. Helm carries a release's values forward to the next revision only when the upgrade is given no values at all. The moment you add a --set, -f or --values, it stops: every value you set at install and do not repeat falls back to its chart default, and the upgrade usually still reports success. An ingress you enabled is deleted, edge.* tuning such as loginRateLimit reverts, and so does everything else you set.

So whenever an upgrade passes a value, add --reset-then-reuse-values:

helm upgrade fleeger charts/fleeger-hub -n fleeger --reset-then-reuse-values \
  --set networkPolicy.enabled=true

It starts from the defaults of the chart you are upgrading to, applies the release's own values on top, then yours. Use it rather than --reuse-values, which reuses the previous revision's values together with the previous chart's defaults: a key that a newer chart version added then has no value at all, and the render fails (for example on edge.loginRateLimit.rate) or quietly gets nothing. Do not combine the two, either — Helm ignores --reset-then-reuse-values when --reuse-values or --reset-values is also given. Passing your complete values file with -f on every upgrade would also work, but only while that file really holds everything you set; the recipes here use --reset-then-reuse-values so they are safe either way.

That carrying-forward includes image.tag. A release installed with a pinned tag (older install guides set --set image.tag=<version>) keeps running those images after every upgrade form above, new chart or not. Clear the pin once, on the next upgrade, so the chart's appVersion picks the images from then on:

helm upgrade fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
  --version 1.0.0-rc1 -n fleeger --reset-then-reuse-values --set image.tag=

If you pass a values file with -f, deleting image.tag from it is not enough: with --reset-then-reuse-values a key missing from the file keeps its stored value. Set it to empty there instead (tag: "" under image:), or add --set image.tag= as above. helm get values fleeger -n fleeger shows what the release has stored.

inventory-svc is deployed with strategy: Recreate: every rollout of it, one that only changes its configuration included, stops the old pod before the new one migrates and starts, so it is briefly gone (typically seconds; the inventory API, the gateway's per-resource and per-credential permission lookups and control actions fail with 502/503, and that cycle's agent reports are dropped and show as a collect error). Its migrations must never run beside a writing inventory-svc, and a rollout must not run two of its pods at once; a failing db migrate keeps it down until it succeeds. Recreate covers rollouts only: restart it with kubectl rollout restart deploy/inventory-svc, not by deleting the pod (a deleted or evicted pod can overlap its replacement; the UNIQUE discovery key still keeps that from duplicating rows). See Deploying Fleeger on Kubernetes. Do not override this to RollingUpdate.

The internal token, the DB password and VictoriaMetrics' two admin-API keys are generated once and reused on every upgrade, where Helm can read the cluster while it renders; a render that cannot fails instead (see GitOps). The credential master key is reused too, but on stricter terms, because it is the only thing that decrypts stored infra credentials and alert-channel secrets — replacing it makes all of them permanently undecryptable, and nothing fails until the next decrypt:

  • Omit credential.key when upgrading. The key already in the cluster is reused. The upgrade commands above deliberately do not pass it. This, and the refusal below, work only when Helm can read the cluster while it renders (it looks the live key up). helm template and --dry-run without =server, and GitOps tools that render client-side, such as Argo CD, cannot: there an upgrade without the key fails to render (credential.key is required, pointing here), and a key that differs from the live one is not refused. If you deploy that way, set credential.existingSecret, as the install commands do; the chart then never generates, compares or replaces the key. A release that already runs keeps its key in the chart-made Secret, <fullname>-credential-key (for release fleeger, fleeger-fleeger-hub-credential-key): copy that key into a Secret of your own first, and point credential.existingSecret at the copy. A Secret holding any other key makes every stored credential undecryptable, so the copy stops when it finds no key rather than create an empty one:

    key=$(kubectl -n fleeger get secret fleeger-fleeger-hub-credential-key \
      -o jsonpath='{.data.credential-key}' | base64 -d) &&
      [ -n "$key" ] &&
      printf '%s' "$key" | kubectl -n fleeger create secret generic \
        fleeger-credential-key --from-file=credential-key=/dev/stdin ||
      echo "no credential key copied; nothing created" >&2

    Pointing credential.existingSecret at the chart-made Secret itself is not enough under GitOps: Helm keeps it when it leaves the chart, but a GitOps tool may prune it. Helm prints a warning on every install and upgrade while no existingSecret is set; helm template, and so Argo CD, never shows it.

  • Passing a different credential.key is refused at render time rather than applied. Re-running an install line that passed --set credential.key=$(head -c 32 /dev/urandom | base64) from shell history is the usual way this happens: it mints a new key every time.

  • To rotate on purpose: put the current key in a Secret referenced by credential.retiredKeysSecret so existing ciphertexts still decrypt, then upgrade with helm upgrade … --reset-then-reuse-values --set credential.allowKeyChange=true and the new key, and once it is live run service rekey in both inventory-svc (credentials) and alerting-svc (alert-channel secrets). Once a second run of each reports everything already current, drop the retired key and re-arm the refusal with helm upgrade … --reset-then-reuse-values --set credential.retiredKeysSecret= --set credential.allowKeyChange=false: --reset-then-reuse-values keeps every value you set, allowKeyChange=true included, so until you do, the next stray credential.key is applied instead of refused. Keep the old key itself, off-box, for as long as you keep any backup taken before both rekeys were confirmed: restoring one needs it back in credential.retiredKeysSecret, and both rekey runs again afterwards (see Backup & disaster recovery).

The Secret is annotated helm.sh/resource-policy: keep, so it also survives helm uninstall.

Edge trusted proxies (BREAKING, first shipped in chart 0.8.1)

Chart 0.8.1 is the first release with edge.trustedProxies, and it refuses to render without it when an ingress is in front of the edge. An upgrade that succeeded on an earlier chart — and equally a fresh install with ingress.enabled=true — now stops, before anything is applied, with:

ingress.enabled is true but edge.trustedProxies is empty.

The refusal is deliberate, and it is the fix, not the regression. The /api/auth/login rate limit is keyed on the address the edge resolves for the caller, and with an ingress in front and no hop trusted that is the controller's own pod address for every caller. The limit then collapses into a single shared bucket of 10r/m for the whole deployment, which any unauthenticated client can exhaust to lock every operator out of login.

The audit log degrades too, though differently from before. Earlier charts appended to X-Forwarded-For, so the client's address was somewhere in the chain — unverified and trivially forgeable, but present. This chart overwrites X-Forwarded-For with the address it actually resolved, which is what stops a caller forging their own entry; the cost is that with nothing trusted the log records the ingress controller for every action and the real client is not recorded at all.

Both failures were silent — no crash-loop, no error, nothing in any log — so the chart refuses at render time rather than merely warning, and the edge also answers such requests with a 503 (below).

To upgrade, name the CIDR of the one hop directly in front of the edge:

helm upgrade fleeger charts/fleeger-hub -n fleeger --reset-then-reuse-values \
  --set 'edge.trustedProxies[0]=10.42.0.0/16'   # e.g. the ingress controller's pod CIDR

Trust only that hop. A wider range lets anyone who can reach the edge forge their own source address, dodge the rate limit and write what they like into the audit log — worse than having no forwarded header at all.

Every entry is also checked at render to be an IPv4 or IPv6 address or CIDR, whether or not the ingress is enabled or the check above is waived: an unreplaced placeholder such as <ingress-controller-pod-CIDR>, a host name or a prefix out of range is refused with an error naming the entry, instead of installing an edge whose nginx will not start.

Beyond that, the render-time check sees only the chart's own ingress, and only whether edge.trustedProxies is empty. So the edge also checks at runtime: a request that carries X-Forwarded-For or X-Real-IP but whose client address the edge did not take from it came through a hop that is not trusted, and gets a 503 whose body starts "Fleeger is behind a proxy the edge does not trust". That catches a CIDR that misses the controller, an HTTP proxy or load balancer from outside the chart (a Gateway, an L7 cloud LB), a waived check with a proxy in the path after all, and a trusted proxy that does not send the header edge.realIpHeader names. The 503 covers everything through that proxy: the UI, the API, agents on /nats (their bootstrap log shows only "503"), and auth-svc when nats.publicUrl is a wss:// URL through the ingress (then revoking or offboarding an agent fails, so the agent stays admitted). Requests without either header (port-forward, NodePort, in-cluster clients) are served as before; a client that sends one itself, for example through an office HTTP proxy or an egress proxy in a pod's HTTP_PROXY, gets the 503.

If you see that 503 through your ingress, read the peer's address off the edge's log (kubectl -n fleeger logs deploy/edge, the first field of each line). If it is not in edge.trustedProxies, set the CIDR that covers it. If it already is, the proxy does not send edge.realIpHeader: set that to the header it does send. The edge restarts by itself when its config changes. Before an upgrade from a chart without this check, the same log tells you in advance: outside requests logged with the controller's address as the first field come through a hop the new edge will refuse.

What the edge sees as the peer is not always the proxy's pod address: a cloud LB in NodePort/instance mode with externalTrafficPolicy: Cluster arrives from a node address, and a service-mesh sidecar on the edge pod (Istio, Linkerd) hides the peer behind its own loopback address (127.0.0.6 with Istio, 127.0.0.1 with Linkerd), which cannot be trusted without trusting every meshed pod. Read the address from the log rather than guessing. With a mesh, keep the edge pod out of it (sidecar.istio.io/inject: "false" as a pod label, or linkerd.io/inject: disabled as a pod annotation), and if the mesh enforces mTLS, let gateway, auth-svc and NATS accept plaintext from the edge; or keep the sidecar and bypass it only for the edge's inbound port 80 (traffic.sidecar.istio.io/excludeInboundPorts or config.linkerd.io/skip-inbound-ports; see Fleeger on Kubernetes — plain manifests). Set either through edge.podLabels and edge.podAnnotations, which go on the edge pod only:

edge:
  podLabels:
    sidecar.istio.io/inject: "false"     # Istio
  podAnnotations:
    linkerd.io/inject: disabled          # Linkerd

A Cloudflare Tunnel whose ingress rule points at the edge Service makes cloudflared the hop: trust its pods' addresses and take the client address from the header Cloudflare sends:

helm upgrade fleeger charts/fleeger-hub -n fleeger --reset-then-reuse-values \
  --set 'edge.trustedProxies[0]=<cloudflared-pod-CIDR>' \
  --set edge.realIpHeader=CF-Connecting-IP

If the rule points at an ingress controller instead, the controller stays the edge's hop and has to trust cloudflared itself. Cloudflare's published IP ranges are the hop only when Cloudflare's proxy connects to the edge itself through a load balancer that keeps the source address; with a tunnel, trusting them leaves the 503 in place.

An L4 load balancer that sends no forwarded header (the usual LoadBalancer Service, an NLB) is not detected, and every client may then share a node's address; put an HTTP ingress in front of the edge instead.

If the ingress genuinely is not in the request path (clients reach the edge directly over NodePort or a port-forward), leave edge.trustedProxies empty and waive the check with --set edge.preflightTrustedProxies=false (on an upgrade: helm upgrade … --reset-then-reuse-values --set edge.preflightTrustedProxies=false). Like any value you set, the waiver then stays in force on later upgrades; set it back to true if an ingress is ever put in the request path.

edge.realIpHeader, edge.realIpRecursive and edge.loginRateLimit tune the same machinery and are documented in the Edge / ingress / network policy table under Values reference.

NATS volume grown to 3Gi (upgrading from a chart that shipped 1Gi)

natsServer.persistence.size now defaults to 3Gi, up from 1Gi. NATS declares 75% of the volume as its JetStream store, and the SPOKES account's 2 GB JetStream grant does not fit in less. An upgrade therefore asks Kubernetes to grow the existing nats-data claim.

  • The StorageClass allows volume expansion (allowVolumeExpansion: true): nothing to do. The claim grows in place, keeping its data.

  • It does not (k3s local-path, for one): the upgrade fails with

    cannot patch "nats-data" with kind PersistentVolumeClaim: persistentvolumeclaims "nats-data" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
    

    but Helm has already applied everything else, so NATS restarts declaring a 2.25 GiB store on the old 1 GiB disk, and fills it before any stream limit is reached. Migrate the claim with the recipe below, then run the same upgrade again.

The recipe stops NATS, copies nats-data into a temporary claim, replaces nats-data with a 3Gi claim and copies the data back. Keep the data: nats-data holds the metrics and logs ingestion-svc has not forwarded yet, and the resolver's SPOKES account JWT, which carries auth-svc's agent revocations. (A NATS started on an empty claim gets SPOKES back from the nats-push hook, and auth-svc adds the revocations back within about 10 seconds of seeing them missing, but only while it is running; until then every offboarded agent's credentials work again.) Agents cannot publish while NATS is down, so pick a quiet moment.

Set the first two lines, then paste the whole block. It runs in a subshell that stops at the first failing step, and it is safe to run again after a failure: it never copies over the temporary claim once that holds a complete copy, and never deletes nats-data before then. STORAGE_CLASS must match natsServer.persistence.storageClass (empty for the cluster default): a claim's class cannot be changed, so a mismatch makes the upgrade fail again. The copy runs as root in a busybox pod, to keep the files' owners; a namespace that enforces the restricted Pod Security Standard, or a quota that requires resource limits, refuses that pod, so relax them for the migration.

NS=fleeger RELEASE=fleeger SIZE=3Gi   # your namespace, release name and natsServer.persistence.size
STORAGE_CLASS=""                      # natsServer.persistence.storageClass, if you set one
(
set -e

# Copy one claim's files onto another. A failed copy pod is left for
# `kubectl logs`, and removed by the next run.
copy() {
  kubectl -n "$NS" delete pod nats-data-copy --ignore-not-found --wait
  kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata: {name: nats-data-copy}
spec:
  restartPolicy: Never
  containers:
    - name: copy
      image: busybox:1.38
      command: ["sh", "-c", "cp -a /from/. /to/ && sync"]
      volumeMounts: [{name: from, mountPath: /from}, {name: to, mountPath: /to}]
  volumes:
    - {name: from, persistentVolumeClaim: {claimName: $1}}
    - {name: to, persistentVolumeClaim: {claimName: $2}}
EOF
  while :; do
    phase="$(kubectl -n "$NS" get pod nats-data-copy -o jsonpath='{.status.phase}')"
    case "$phase" in
      Succeeded) break ;;
      Failed) echo "copying $1 to $2 failed: kubectl -n $NS logs nats-data-copy" >&2; return 1 ;;
    esac
    sleep 2
  done
  kubectl -n "$NS" delete pod nats-data-copy
}

# A new, empty claim. It carries Helm's marks, like the chart's own claim, and
# one of ours that says it is the new one.
claim() {
  kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: $1
  labels: {app.kubernetes.io/managed-by: Helm}
  annotations:
    meta.helm.sh/release-name: $RELEASE
    meta.helm.sh/release-namespace: $NS
    fleeger.dev/nats-migration: new
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: $SIZE}}
${STORAGE_CLASS:+  storageClassName: $STORAGE_CLASS}
EOF
}

note() { kubectl -n "$NS" get pvc "$1" --ignore-not-found -o jsonpath='{.metadata.annotations.fleeger\.dev/nats-migration}'; }

kubectl -n "$NS" scale deploy/nats --replicas=0
while [ -n "$(kubectl -n "$NS" get pods -l app.kubernetes.io/name=nats,app.kubernetes.io/instance="$RELEASE" -o name)" ]; do sleep 2; done

# 1. Copy nats-data aside, unless an earlier run already has.
if [ "$(note nats-data-move)" != copied ]; then
  claim nats-data-move
  copy nats-data nats-data-move
  kubectl -n "$NS" annotate pvc nats-data-move fleeger.dev/nats-migration=copied --overwrite
fi
# 2. Replace nats-data with the larger claim, unless an earlier run already has.
if [ "$(note nats-data)" != new ]; then
  kubectl -n "$NS" delete pvc nats-data --ignore-not-found --wait
  claim nats-data
fi
# 3. Copy the data back and drop the temporary claim.
copy nats-data-move nats-data
kubectl -n "$NS" delete pvc nats-data-move
)

Then repeat the upgrade command you started with. It brings NATS back on the new claim. The old 1Gi volume is released with its claim, and deleted or kept according to the StorageClass's reclaim policy.

A few more things to know:

  • --reuse-values keeps the old 1Gi and is refused at render. Use --reset-then-reuse-values (see above) whenever the upgrade passes a value.
  • An upgrade run with --atomic (or --rollback-on-failure) does not half-apply: Helm rolls the release back when the claim is refused. Migrate the claim the same way, then upgrade again.
  • Once the claim has grown, rolling back to a chart that shipped 1Gi fails on the claim, because Kubernetes never shrinks one. Pin natsServer.persistence.size to the claim's size if you need to roll back.

The raw manifests in deploy/k8s/base/ grew the claim the same way.

The NATS auth Secret must not hold the credential key

The chart now refuses to render when nats.authSecret names the Secret that holds the credential master key: the chart-made <fullname>-credential-key, credential.existingSecret or credential.retiredKeysSecret. That Secret is mounted whole into auth-svc and the NATS pod, which would both then hold the key that decrypts every stored infrastructure credential and alert-channel secret. If you keep both in one Secret today, move the credential key (and any retired keys) into a Secret of its own and point credential.existingSecret at it before upgrading.

VictoriaMetrics admin-API keys and the alerting hub list

VictoriaMetrics now requires keys for its admin endpoints. The chart generates both into <fullname>-victoriametrics on the upgrade and keeps them: delete-key (-deleteAuthKey), which only ingestion-svc holds, and snapshot-key (-snapshotAuthKey, -forceMergeAuthKey, -forceFlushAuthKey, -search.resetCacheAuthKey, -reloadAuthKey), which only the vmbackup sidecar holds, in both of its snapshot URLs. Nothing to do unless you call those endpoints yourself, which now needs authKey, or bring your own Secret with victoriametrics.auth.existingSecret (both keys, hex). ingestion-svc checks at startup that a delete without the key is refused and one with it accepted, and logs an error otherwise; a delete VictoriaMetrics refuses for its key is retried rather than dropped.

alerting-svc now also refuses notification channels aimed at the host of loki.url and at anything in alerting.hubServices. Add your pod and service CIDRs there: alerting-svc knows the ClusterIP a service name resolves to, not the pod IPs behind it. The core agent's FLEEGER_HUB_SERVICES gets the host of loki.url added too (the default names are written out with it when coreAgent.hubServices is empty).

Ingestion streams created by an earlier release (one-off, manual)

Earlier releases created the metrics and logs JetStream streams unbounded: they keep every message after ingestion has forwarded it, so on a hub that ran one of those releases they grow until the NATS account's disk is full, at which point every agent's publishes are refused and metric and log ingestion stop. Current releases create them bounded, but cannot convert an existing stream — NATS refuses to change a stream's retention in place — so an upgraded hub keeps the old stream, and ingestion-svc logs this at every start until it is fixed:

the 'metrics' stream already exists with Limits retention, not WorkQueue: … Run `service stream migrate` …

Fix it once, after the upgrade has fully rolled out and while ingestion-svc is running (it is what drains the backlog):

kubectl -n fleeger exec deploy/ingestion-svc -- /usr/local/bin/service stream migrate
# the compose stack:
#   docker compose -f deploy/compose/compose.yaml exec ingestion-svc /usr/local/bin/service stream migrate

For each stream it waits for ingestion-svc to forward everything the stream holds, stops the stream accepting new publishes so nothing can arrive after that check, then deletes it and creates the bounded one. It never deletes a message that has not reached VictoriaMetrics or Loki. ingestion-svc needs no restart: it reattaches to the new stream within seconds, losing nothing. The command checks for that and tells you to restart ingestion-svc if it has not reattached within 30 seconds. --stream metrics or --stream logs limits it to one stream. A fresh install never needs any of this.

What to expect:

  • Usually a few milliseconds of failed publishes per stream (measured at 1–5 ms against a local broker; roughly five JetStream round trips). If a few messages arrived just before the stream stopped accepting, the command first waits for ingestion-svc to forward them — up to 5 seconds, with publishes still failing — and restores the stream and refuses if they are not. An agent whose publish lands in that window logs it (publish failed for …, heartbeat publish failed, k8s log publish … failed) and drops that sample — for logs, that pod's batch for the cycle. A logs publish fails at once; a metrics publish fails when the agent's 5-second ack wait runs out (gateway-svc and alerting-svc also listen on metrics.>), holding that agent's cycle up by 5 seconds. Where the stream is the NATS account's only one (metrics on a hub without Loki), NATS now and then refuses to create the new stream right after the old one is deleted (error creating store for stream); the command retries that for up to about 1.5 seconds, publishes still failing, rather than give up.
  • Settings you tuned are kept. The new stream is built from the old one's config — replicas, duplicate window, extra subjects, per-subject limits, description, placement. Only these are reset to ingestion-svc's configuration, and the command prints each one it changed: retention, discard policy, the three size/age limits, discard_new_per_subject (incompatible with the new discard policy), and a duplicate window longer than the new max_age (shortened to it; NATS refuses otherwise).
  • It refuses rather than guesses, exits non-zero and changes nothing, when: the backlog does not drain within --drain-timeout-secs (default 300 — check that ingestion-svc is running and can reach VictoriaMetrics / Loki; on a very busy hub the backlog may never be empty at a check, so try a quieter moment); the stream has a consumer other than ingestion-svc's own (remove it first), or its consumer is not shaped the way ingestion-svc creates it (push-based, not explicitly acked, filtered, or not starting at the beginning of the stream); or the stream does not look like one ingestion-svc created (sealed, a mirror, sourced, template-owned, or subjects that do not cover metrics.> / logs.>). If an ingestion-svc still on the previous release recreates the old, unbounded stream mid-run, the command reports a failure.
  • It is safe to re-run. A stream that is already bounded, or does not exist, is left alone. The command warns just before the stream stops accepting publishes; if a run is cut off after that without reporting an outcome, the stream may be left accepting no publishes, and the running ingestion-svc does not notice (it logs it only at its next start). Run the command again: it finishes the job. For about 35 seconds such a stream stays reserved for the run that stopped it, and a re-run is refused rather than interfering. Two runs on the same stream at once do not both act on it: at least one backs off, and if both do, the stream is left as it was — run it again. That protection assumes a run gets from its last check to its delete within those 35 seconds; a run frozen for longer at exactly that point could remove a replacement another run had just created. An interruption between deleting the old stream and creating the new one leaves no stream: then ingestion-svc recreates it within seconds, bounded but with default settings.
  • Hubs with log ingestion switched off. Nothing drains an old logs stream there. If everything in it was forwarded while log ingestion was still on, it is migrated (and the command warns that nothing reattached — expected). If it holds messages never forwarded, the command waits out the drain timeout on it and exits non-zero; --stream metrics skips it, but that stream then stays unbounded, shares the account's disk budget with metrics, and gets no startup warning, because nothing opens it. There is no lossless fix from here: switch log ingestion on long enough for the command to drain and migrate it, or accept losing those buffered lines and remove the stream by hand.

GitOps

Argo CD and other tools that render the chart client-side, like helm template, --dry-run without =server and the helm-diff plugin without its --dry-run=server, cannot read the cluster: lookup sees nothing. There the chart cannot reuse the internal token, the DB password and VictoriaMetrics' keys it generated before, and would generate new ones on every render, changing them under running pods. So it refuses to generate them there, and the render fails naming the values still to set: set the values listed, bringing your own Secrets as the credential key already is (or pin internal.token and surrealdb.password inline).

# A new install: fresh values, each through a pipe, never on a command line.
head -c 30 /dev/urandom | base64 | tr -d '\n+/=' | \
  kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n+/=' | \
  kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=/dev/stdin
# bash, for the <(…); VictoriaMetrics does not trim its keys, so no newline.
kubectl -n fleeger create secret generic fleeger-victoriametrics \
  --from-file=delete-key=<(openssl rand -hex 32 | tr -d '\n') \
  --from-file=snapshot-key=<(openssl rand -hex 32 | tr -d '\n')

A release that already runs keeps its values in the chart-made Secrets <fullname>-internal, <fullname>-db and <fullname>-victoriametrics (for release fleeger, fleeger-fleeger-hub-internal and so on). Copy them into Secrets of your own instead, so the pods keep the values they run with. Set full to your release's fullname. The copy (bash, for the <(…), which keeps the values off the command line) stops, creating nothing, when a value cannot be read, rather than create a Secret with an empty one. If a create fails (say, the Secret already exists from an earlier run), the ones after it are skipped; delete what was created and run it again:

full=fleeger-fleeger-hub
from() { kubectl -n fleeger get secret "$full-$1" -o go-template="{{index .data \"$2\" | base64decode}}"; }
if it=$(from internal internal-token) && db=$(from db db-password) &&
  dk=$(from victoriametrics delete-key) && sk=$(from victoriametrics snapshot-key) &&
  [ -n "$it" ] && [ -n "$db" ] && [ -n "$dk" ] && [ -n "$sk" ]; then
  kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=<(printf '%s' "$it") &&
  kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=<(printf '%s' "$db") &&
  kubectl -n fleeger create secret generic fleeger-victoriametrics \
    --from-file=delete-key=<(printf '%s' "$dk") --from-file=snapshot-key=<(printf '%s' "$sk")
else
  echo "a value could not be read; nothing created" >&2
fi
unset it db dk sk full; unset -f from

Then deploy with

credential:
  existingSecret: fleeger-credential-key
internal:
  existingSecret: fleeger-internal
surrealdb:
  existingSecret: fleeger-db
victoriametrics:
  auth:
    existingSecret: fleeger-victoriametrics

With scheduled backups on (backup.enabled), the backup signing key is one more: bring it as backup.signing.existingSecret, a new one with openssl genpkey -algorithm ed25519 | kubectl -n fleeger create secret generic fleeger-backup-signing --from-file=signing-key=/dev/stdin, or a copy of the chart-made <fullname>-backup-signing (its signing-key entry, as above) for a release that already signs with it; a new key signs sets that only its own public key in backup.signing.publicKeys accepts.

Pointing the values at the chart-made Secrets themselves is not enough: Helm keeps them when they leave the chart, but a GitOps tool may prune them. A release that was already deployed client-side before this refusal has had these values replaced on every sync; the DB password, though, takes effect only on a new datastore, so if the services cannot log in after the copy, the datastore still has the first one.

A real install or upgrade reads the cluster and is not refused, with --create-namespace too; the chart lists the namespace's ServiceAccounts to tell, so on an install (or an upgrade that first creates VictoriaMetrics' keys) the deploying user needs list on them, as the built-in edit role has. To see what an upgrade would change, render against the cluster: helm-diff 3.9 or later with its --dry-run=server, or Helm's own --dry-run=server. To only look at a render with helm template, turn allowOfflineSecretGeneration on for that render; never deploy such a render.

Toggling core services

Every core service is enabled by default and can be turned off individually — e.g. bring your own OIDC by disabling auth-svc:

helm install fleeger charts/fleeger-hub -n fleeger \
  --set image.tag=local \
  --set services.auth-svc.enabled=false

services.<name>.enabled (default true) exists for: auth-svc, inventory-svc, gateway-svc, ingestion-svc, alerting-svc, control-svc.

Values reference

Only the important values are listed. See values.yaml for the full set and inline notes.

Images

Key Default Description
image.registry fleeger Registry prefix; service images are <registry>/<service>:<tag>. Set it for an OCI install: the default names Docker Hub, where Fleeger publishes nothing.
image.tag "" Image tag for all Fleeger images (services, frontend, spoke-agent, backup). Empty means the chart's appVersion, the tag its release published; set local for images built from a checkout. A tag set here stays pinned across upgrades until cleared with --set image.tag=. The frontend image runs as uid/gid 101 with no capabilities, so it must be built from a Dockerfile.frontend that makes its nginx config world-readable (this chart version's does); an older frontend image built from a group-only checkout fails with open() "/etc/nginx/conf.d/default.conf" failed (13: Permission denied).
image.pullPolicy IfNotPresent
imagePullSecrets [] Names of pull secrets for a private (e.g. GitLab) registry.
logLevel info

Secrets

Key Default Description
credential.key "" Base64 32-byte master key. Required unless existingSecret is set. Checked at render as the services decode it: trimmed, standard base64, exactly 32 bytes. Never checked: the key already in the cluster, which an upgrade reuses as it is.
credential.existingSecret "" Secret holding the key under credential-key.
credential.preflightExistingSecret true Fail helm install/upgrade when credential.existingSecret exists but has no credential-key entry, or its entry is not a valid key (standard base64 of 32 bytes; the message names the Secret, never the value), instead of leaving the pods in CreateContainerConfigError or crash-looping. An absent Secret is not refused (the install notes remind you). The check uses lookup, which sees nothing under helm template, --dry-run without =server or Argo CD, so it runs only on a real install or upgrade. Needs Secret read access; set false if your RBAC forbids it.
credential.retiredKeysSecret "" Secret with retired-keys (comma-separated base64) so old ciphertexts decrypt during a rotation; clear after service rekey has run in both inventory-svc and alerting-svc.
internal.token "" Shared token the gateway attaches and backends require. Empty ⇒ chart generates one (kept across upgrades; not in a client-side render, see GitOps).
internal.existingSecret "" Secret with an internal-token key.
allowOfflineSecretGeneration false Generate the internal token, the DB password and VictoriaMetrics' keys even where the render cannot read the cluster (helm template, --dry-run without =server, Argo CD). Off, such a render fails naming the existingSecret values to set instead (GitOps). Only for a render that is looked at: deployed, every render changes those values.
victoriametrics.auth.existingSecret "" Secret with VictoriaMetrics' admin-API keys: delete-key (-deleteAuthKey, reaches only VictoriaMetrics and ingestion-svc) and snapshot-key (-snapshotAuthKey and the other maintenance keys, reaches only VictoriaMetrics and the vmbackup sidecar). Two different hex keys (openssl rand -hex 32). Empty ⇒ the chart generates both into <fullname>-victoriametrics (kept across upgrades; not in a client-side render, see GitOps).

NATS

Key Default Description
nats.authSecret fleeger-nats-auth Name of the Secret holding the gen-auth.sh material. Required. Must not be the Secret holding the credential key (refused at render).
nats.preflightAuthSecret true Fail helm install/upgrade (naming the missing keys) when that Secret exists but lacks a key the render references, instead of letting the pods fail later. Needs Secret read access; set false if your RBAC forbids it.
nats.publicUrl "" NATS URL agents connect to. Empty ⇒ edge WebSocket (ws://edge/nats, in-cluster). For external agents set your wss:// endpoint, e.g. wss://fleeger.example.com/nats.

SurrealDB

Key Default Description
surrealdb.image surrealdb/surrealdb:v3.3.4
surrealdb.user root
surrealdb.password "" Root password. Empty ⇒ chart generates one. Prefer surrealdb.existingSecret for your own: --set puts it on the command line.
surrealdb.existingSecret "" Secret with a db-password key (BYO password).
surrealdb.persistence.size 5Gi
surrealdb.persistence.storageClass ""

NATS server (dependency)

Key Default Description
natsServer.image nats:2.15-alpine
natsServer.persistence.size 3Gi
natsServer.persistence.storageClass ""

VictoriaMetrics

Key Default Description
victoriametrics.image victoriametrics/victoria-metrics:v1.153.0
victoriametrics.retentionPeriod "6" Retention in months.
victoriametrics.maxQueryDuration 30s Query-cost bound.
victoriametrics.maxConcurrentRequests "16"
victoriametrics.maxQueryLen 16KB
victoriametrics.maxHourlySeries 1000000 Hub-wide backstop on series: the most distinct series written in one hour of VictoriaMetrics' uptime (-storage.maxHourlySeries), every location and the hub's own counted, existing series included: once reached, every series not yet written in that hour loses its samples. ingestion-svc's per-location ingestion.maxNewSeriesPerLocationHour comes first. Size it above your hub-wide active series plus their hourly churn; VictoriaMetrics needs roughly 1 KiB of memory per active series. VictoriaMetrics stores its own metrics (-selfScrapeInterval=30s), and the built-in vm_series_limit_reached alert fires on its drops. 0 is no limit; not set is refused (as an upgrade with --reuse-values from an older chart leaves it: use --reset-then-reuse-values).
victoriametrics.maxDailySeries 5000000 The same per day of uptime (-storage.maxDailySeries): above your hub-wide active series plus their daily churn. 0 is no limit.
victoriametrics.auth.existingSecret "" See Secrets.
victoriametrics.persistence.size 10Gi
victoriametrics.persistence.storageClass ""

Core service toggles

Key Default Description
services.auth-svc.enabled true Disable to bring your own OIDC.
services.inventory-svc.enabled true
services.gateway-svc.enabled true
services.ingestion-svc.enabled true
services.alerting-svc.enabled true
services.control-svc.enabled true

Bundled core agent (opt-in)

Key Default Description
coreAgent.enabled false Serves every location flagged local_collector. Requires a core-agent.creds entry in the NATS auth Secret. That credential needs only pub inv._core.> for inventory, and pub metrics._core.>/logs._core.>/_INBOX._core.> and sub commands._core.> for its metrics, logs and commands: after rolling out a release with both changes (every replica of every hub service, not only the core agent: an older ingestion-svc drops the _core subjects, an older control-svc or inventory-svc sends no command there, an older alerting-svc credits no _core heartbeat), narrow an older one with FLEEGER_AUTH_SVC="kubectl -n <namespace> exec deploy/auth-svc -- /usr/local/bin/service" deploy/nats/gen-auth.sh fix-perms core-agent (it also revokes the core agent's earlier JWTs in auth-svc's revocation ledger, which every later push of the SPOKES account re-asserts, and disconnects it), update the Secret with the reissued core-agent.creds and kubectl rollout restart the core agent. Until then it keeps working on each location's own subjects, and logs a warning at start-up. To roll back to a release without the _core subjects afterwards, widen the credential first (Agent credentials & revocation).
coreAgent.pollIntervalSecs 15
coreAgent.tmpSizeLimit 2Gi Writable /tmp emptyDir — the agent's scratch space (rootfs is read-only).
coreAgent.hubServices [] The hub's own services, which the core agent never probes or collects from: host names, IPs or CIDR ranges. Empty refuses this chart's service names, by name and by the ClusterIP they resolve to. Setting it replaces that list. The host of loki.url is always added. Whatever the list, this release's own pods are refused by their pod IPs too, listed every cycle by their app.kubernetes.io/part-of and app.kubernetes.io/instance labels, which needs pods list in the namespace (granted by coreAgent.rbac.create). Loopback, link-local, the agent's NATS host and this cluster's own Kubernetes API (kubernetes.default.svc and the KUBERNETES_SERVICE_HOST address) are always refused.
coreAgent.resources requests 50m/64Mi, limit 256Mi

Alerting

Key Default Description
alerting.hubServices [] More of the hub's own services no notification channel may be aimed at: host names, IPs or CIDR ranges, rendered into alerting-svc's [alerting] hub_services next to the host of loki.url. Add the pod and service CIDRs to refuse the pod IPs of VictoriaMetrics and Loki, unless a notification target of yours lives in them. An invalid entry stops alerting-svc from starting.

Ingestion streams

The bounds of the JetStream metrics and logs streams that buffer what the agents send while ingestion-svc is down, rendered into ingestion-svc's [ingestion] section. Each takes a whole number; empty keeps ingestion-svc's own default, and ingestion-svc refuses to start on a value outside its bounds, naming the key. Set them here: ingestion-svc refuses BUMACS_INGESTION_* environment variables, which never reached these keys.

Key Default Description
ingestion.streamMaxAgeSecs "" How long a stream keeps a message ingestion-svc has not forwarded yet, in seconds: the longest ingestion outage that loses nothing. Empty ⇒ 259200 (3 days). At least 60.
ingestion.streamMaxBytes "" Disk budget of each stream, in bytes: the bound that makes a stream drop its oldest messages instead of filling the NATS volume. Empty ⇒ 201326592 (192 MiB). Both streams reserve it, so twice it must fit under JetStream's max_file_store (75% of natsServer.persistence.size) and the SPOKES account's 2000000000-byte grant: at most 1000000000 on any volume the chart accepts (its size floor keeps max_file_store at or above the grant). The render refuses a larger value. It counts both streams even on a hub without loki.url, so that enabling Loki later does not make ingestion-svc halve the cap.
ingestion.streamMaxMsgs "" The most messages each stream keeps. Empty ⇒ 1000000.
ingestion.maxNewSeriesPerLocationHour "" The most new metric series one location may make per hour, by the hour its messages were published to NATS. Past it ingestion-svc drops the points of further new series (the location's existing series and every other location are written), logs it once per location and hour, and fires the built-in series_bound_exceeded alert. Empty ⇒ 50000; ingestion-svc takes 1000 to 250000 and keeps up to about 2.3 MiB per location at the default (18 MiB at 250000) of resources.services' 256Mi limit. After an ingestion-svc restart, or an agent upgrade that renames its series, every series of a location counts as new: with S series a location writes per hour, S up to the bound loses nothing, up to twice it loses the excess until the next hour, more keeps losing points. Size it above your largest location (a Redfish-heavy one, 200 hosts with a few hundred sensors each, reaches the default). victoriametrics.maxHourlySeries is the hub-wide backstop.

Scheduled backups (opt-in)

Key Default Description
backup.enabled false CronJob dumps every SurrealDB service DB to S3 + pushes the dead-man's-switch metric. Needs backup.existingSecret, backup.s3Bucket and, while backup.vmbackup.enabled, backup.vmbackup.dst; the render names whichever are missing.
backup.image "" Empty ⇒ <registry>/backup:<tag>, with <tag> resolved as for image.tag.
backup.schedule "0 3 * * *" Cron (cluster tz).
backup.s3Bucket "" e.g. s3://fleeger-backups.
backup.s3EndpointUrl "" Optional. Only for non-AWS S3-compatible targets (Hetzner object storage, MinIO, Ceph). Empty ⇒ the AWS default endpoint for backup.region.
backup.region us-east-1
backup.existingSecret "" S3 creds Secret (access-key-id, secret-access-key).
backup.retentionDays "" Prune sets older than N days (empty ⇒ keep all).
backup.tmpSizeLimit 2Gi Scratch for dumps.
backup.signing.existingSecret "" Secret with the backup signing key, signing-key (Ed25519, PEM), mounted by the CronJob only. See Secrets. Empty ⇒ chart-generated <fullname>-backup-signing.
backup.signing.publicKeys [] The public keys scripts/restore.sh accepts a set's signature from, each the base64 line of its PEM (openssl pkey -in <key> -pubout | sed -n 2p); given to the CronJob as FLEEGER_BACKUP_VERIFY_KEYS (it warns when its own key is not among them), and checked at render. A restore runs in a pod without the signing key (deploy/k8s/optional/restore-s3-pod.yaml), given the same keys. List the old key too while rotating; drop a leaked one at once. Keep a copy off the hub.
backup.vmbackup.enabled true VictoriaMetrics vmbackup sidecar (incremental → S3; restore with vmrestore). Gets the snapshot key and sends it in both snapshot URLs.
backup.vmbackup.image victoriametrics/vmbackup:v1.153.0
backup.vmbackup.intervalSeconds 3600
backup.vmbackup.dst "" e.g. s3://fleeger-backups/vm (distinct from the DB sets).

Edge / ingress / network policy

Key Default Description
edge.enabled true nginx edge serving the UI and proxying /api.
edge.image nginx:1.31.8-alpine Runs as uid/gid 101 with no capabilities, writing only to emptyDirs at /var/cache/nginx and /run. An override must work that way too, and be the official nginx image at 1.29.7 or later: the edge follows a recreated hub Service with server ... resolve and turns upstream keepalive off with keepalive 0, which older nginx refuses, and its entrypoint's 15-local-resolvers.envsh supplies the resolver.
edge.service.type ClusterIP Set NodePort (+ edge.service.nodePort) for quick access without an ingress.
edge.service.port 80
edge.trustedProxies [] CIDRs of the one hop directly in front of the edge, as the edge sees it — your ingress controller's pod CIDR, or the source range of the cloud LB (node addresses for an LB in NodePort mode; read it from the edge's log). For a peer inside these ranges nginx replaces the client address with the one from edge.realIpHeader; for anyone else the header is ignored. Empty means trust nobody. Each entry must be an IPv4/IPv6 address or CIDR (no host names); anything else is refused at render, naming the entry. Set this whenever ingress.enabled=true — see Upgrade for what goes wrong if you do not.
edge.preflightTrustedProxies true Refuse to render when edge.enabled=true and ingress.enabled=true and edge.trustedProxies is empty, instead of silently shipping a broken rate limit and audit log. Set false only when the ingress is genuinely not in the request path. Requests through an untrusted proxy still get a 503 at runtime either way.
edge.realIpHeader X-Forwarded-For Header the trusted hop puts the client address in. Suits ingress-nginx/Traefik/HAProxy; some setups use X-Real-IP, and Cloudflare uses CF-Connecting-IP (with a Cloudflare Tunnel, trust the cloudflared pods, not Cloudflare's published ranges; see Edge trusted proxies). Only emitted while edge.trustedProxies is non-empty — with nothing trusted there is no header to believe.
edge.realIpRecursive true Walk the header right-to-left past every trusted address and take the first untrusted one as the client. What stops a client-prepended fake entry from being believed; leave on for a chain of proxies. Only emitted while edge.trustedProxies is non-empty.
edge.podLabels / edge.podAnnotations {} / {} Extra labels and annotations for the edge pod, e.g. to keep it out of a service mesh (see the mesh paragraph). Values are strings or bools; the chart's own pod labels and checksum/edge-conf cannot be overridden.
edge.loginRateLimit.rate / .burst 10r/m / 5 /api/auth/login budget, applied per resolved client address — so it is only per-client once edge.trustedProxies is right.
ingress.enabled false Needs an ingress controller and edge.enabled=true (the Ingress routes to the edge Service; the render is refused without it). Use TLS — login and agent WebSockets are otherwise cleartext.
ingress.className ""
ingress.host "" e.g. fleeger.example.com.
ingress.annotations {} e.g. cert-manager cluster-issuer.
ingress.tls [] TLS hosts + secretName.
loki.url "" A Loki you run yourself, for pod-log ingestion, dead-log purging and the log tab; empty turns them off. The release namespace's pod logs (the hub's own, collected with coreAgent.podLogs: all) are read only by a holder of hub_logs.view: ingestion-svc stamps them hub="true" (see logs.hubNamespaces).
loki.maxPushBytes "" The most one push to Loki may come to as the protobuf Loki's distributor hands its ingester over gRPC, by a conservative estimate: ingestion-svc splits a larger log message into several pushes. Keep it under your Loki's server.grpc_server_max_recv_msg_size (4 MiB by default). Empty ⇒ 3145728 (3 MiB). From 1048576 to 268435456; the render refuses anything else.
logs.hubNamespaces [] More namespaces whose pods are the hub's own, added to the release namespace (always one) in ingestion-svc's [logs] hub_namespaces, each once. Their pods' log streams, as the core agent ships them for a local_collector location, are stamped hub="true", which only a holder of hub_logs.view reads. Add any namespace that runs hub components next to the release (a Loki, an ingress controller, an operator you installed for the hub) whose logs a viewer should not read. Streams written before the upgrade that added this setting, or before fix-perms core-agent narrowed the core agent's credential, are not stamped and stay visible until Loki's retention ages them out; a hub cluster collected by a regular spoke agent instead of the core agent is never stamped. Each entry must be a namespace name; with loki.url only.
networkPolicy.enabled false Default-deny + per-service allow rules. CNI-specific; validate before enabling.
networkPolicy.probeCIDRs [] Node CIDR(s) to allow kubelet health probes through.

Scheduling

Key Default Description
resources.services.requests 50m / 64Mi Per-service requests.
resources.services.limits.memory 256Mi
nodeSelector / tolerations / affinity {} / [] / {}

Full walkthrough

See Deploying Fleeger on Kubernetes for the complete deployment guide, including generating the NATS auth Secret, TLS/ingress, backups, and NetworkPolicy validation.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.