// Deployment-Referenz
Helm chart: fleeger-hub
Helm chart for the Fleeger hub — the control plane and admin UI. Chart version 1.0.0-rc1 (appVersion 1.0.0-rc1).
Install one hub release per namespace: service names are fixed (surrealdb, nats, auth-svc, …) so the services resolve each other by in-namespace DNS.
What it deploys
- Core services —
auth-svc,inventory-svc,ingestion-svc,alerting-svc,control-svc,gateway-svc. Each can be toggled off individually (see Toggling services). - Frontend + edge — the admin UI and an nginx edge that serves it and proxies
/apito the gateway (and exposesws://edge/nats). - Bundled dependencies — SurrealDB (v3.3.4), NATS/JetStream (2.15), VictoriaMetrics (v1.153.0).
- Optional — a bundled "core" agent (
coreAgent), scheduled backups + a VictoriaMetricsvmbackupsidecar (backup), an ingress, and NetworkPolicies.
VictoriaMetrics is the metrics store. Log aggregation is off by default — this chart deploys no Loki and no log shipper (only the compose stack does). Point loki.url at a Loki you run yourself to enable per-stream dead-log purging, pod-log ingestion and the Observability Explorer's log tab; leave it empty and those features stay cleanly disabled (/api/logs/* returns 404).
Prerequisites
A Kubernetes cluster with a default StorageClass (or set *.persistence.storageClass), Helm 3.14 or later (the upgrade recipes below use --reset-then-reuse-values, which 3.14 added), and the following Secrets in the target namespace:
| Secret | Required? | How to create it |
|---|---|---|
| Master credential key | Required | A base64 32-byte key encrypting stored infra credentials and alert-channel configs (standard base64 of exactly 32 bytes, as the recipe below makes; a credential.key of any other shape is refused at render). Create it as a Secret (key credential-key) and point credential.existingSecret at it, as the install commands below do. On a real install or upgrade the chart checks that Secret's key when the Secret already exists (credential.preflightExistingSecret; helm template, client-side --dry-run and Argo CD cannot look it up). --set credential.key=… also works, but puts the key on the helm command line, readable by every local user with ps, and in the release's stored values. BACK IT UP — losing it makes every encrypted credential unrecoverable. |
| NATS auth material | Required | Generated by deploy/nats/gen-auth.sh — the per-service creds (ingestion.creds, gateway.creds, alerting.creds, inventory.creds, control.creds), plus core-agent.creds when coreAgent.enabled=true, plus sys.creds, operator.nk, spokes-account.nk, SPOKES.jwt, resolver.conf. All of them — each is referenced by key name in a volume projection, so a missing one leaves that pod crash-looping (or stuck in ContainerCreating). The chart cannot generate it (needs nsc) but does preflight it on install (nats.preflightAuthSecret). Loaded as the Secret named by nats.authSecret (default fleeger-nats-auth). See Deploying Fleeger on Kubernetes for the one-liner. |
| DB password | Optional | Leave surrealdb.password/existingSecret empty to have the chart generate a strong password into a Secret (kept across upgrades; not under Argo CD or helm template, see GitOps), or pin your own in a Secret (key db-password) referenced by surrealdb.existingSecret; --set surrealdb.password=… puts it on the helm command line. A pinned password takes effect only on a new datastore. |
| S3 credentials | Optional | Only if backup.enabled=true. A Secret with keys access-key-id, secret-access-key, referenced by backup.existingSecret. |
| Backup signing key | Optional | Only if backup.enabled=true. An Ed25519 private key in PEM (key signing-key) that only the backup CronJob mounts and signs every backup set with: openssl genpkey -algorithm ed25519 | kubectl -n <ns> create secret generic fleeger-backup-signing --from-file=signing-key=/dev/stdin, referenced by backup.signing.existingSecret. Empty ⇒ the chart generates one into <fullname>-backup-signing (kept across upgrades; not in a client-side render, see GitOps). Its public key goes into backup.signing.publicKeys (the install notes print the command) and, as a copy, somewhere off the hub: scripts/restore.sh restores only sets whose signature verifies under one of them. |
Install
From a checkout, with the fleeger/*:local images built from it and loaded into the nodes:
kubectl create namespace fleeger
# The master key goes through a pipe, never onto a command line. BACK IT UP.
head -c 32 /dev/urandom | base64 | tr -d '\n' | \
kubectl -n fleeger create secret generic fleeger-credential-key --from-file=credential-key=/dev/stdin
helm install fleeger charts/fleeger-hub -n fleeger \
--set image.tag=local \
--set credential.existingSecret=fleeger-credential-keyFrom the OCI registry (the project's GitLab container registry — replace registry.example.com/fleeger with your ${CI_REGISTRY_IMAGE}):
helm install fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
--version 1.0.0-rc1 -n fleeger \
--set image.registry=registry.example.com/fleeger \
--set credential.existingSecret=fleeger-credential-key # the Secret created aboveLeave image.tag unset: a published chart runs the images its own release pipeline pushed (its appVersion), so each chart upgrade brings the matching images with it. A tag you set is stored in the release and carried into every later upgrade, which would leave the old images running under the new chart.
The NATS auth Secret must already exist in the namespace before install. A post-install nats-push Job loads the SPOKES account into the resolver, after which the services finish authenticating and become Ready. Create the first admin with kubectl -n fleeger exec -it deploy/auth-svc -- /usr/local/bin/service user add …, which prompts for the password (see the post-install notes).
Upgrade
# from a checkout
helm upgrade fleeger charts/fleeger-hub -n fleeger
# from OCI
helm upgrade fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
--version 1.0.0-rc1 -n fleegerThese pass no values, and that is what makes them safe. Helm carries a
release's values forward to the next revision only when the upgrade is given
no values at all. The moment you add a --set, -f or --values, it stops:
every value you set at install and do not repeat falls back to its chart
default, and the upgrade usually still reports success. An ingress you enabled is
deleted, edge.* tuning such as loginRateLimit reverts, and so does
everything else you set.
So whenever an upgrade passes a value, add --reset-then-reuse-values:
helm upgrade fleeger charts/fleeger-hub -n fleeger --reset-then-reuse-values \
--set networkPolicy.enabled=trueIt starts from the defaults of the chart you are upgrading to, applies the
release's own values on top, then yours. Use it rather than --reuse-values,
which reuses the previous revision's values together with the previous chart's
defaults: a key that a newer chart version added then has no value at all, and
the render fails (for example on edge.loginRateLimit.rate) or quietly gets
nothing. Do not combine the two, either — Helm ignores
--reset-then-reuse-values when --reuse-values or --reset-values is also
given. Passing your complete values file with -f on every upgrade would
also work, but only while that file really holds everything you set; the
recipes here use --reset-then-reuse-values so they are safe either way.
That carrying-forward includes image.tag. A release installed with a pinned
tag (older install guides set --set image.tag=<version>) keeps running those
images after every upgrade form above, new chart or not. Clear the pin once,
on the next upgrade, so the chart's appVersion picks the images from then on:
helm upgrade fleeger oci://registry.example.com/fleeger/charts/fleeger-hub \
--version 1.0.0-rc1 -n fleeger --reset-then-reuse-values --set image.tag=If you pass a values file with -f, deleting image.tag from it is not
enough: with --reset-then-reuse-values a key missing from the file keeps its
stored value. Set it to empty there instead (tag: "" under image:), or
add --set image.tag= as above. helm get values fleeger -n fleeger shows
what the release has stored.
inventory-svc is deployed with strategy: Recreate: every rollout of
it, one that only changes its configuration included, stops the old pod before
the new one migrates and starts, so it is briefly gone (typically seconds; the
inventory API, the gateway's per-resource and per-credential permission
lookups and control actions fail with 502/503, and that cycle's agent reports
are dropped and show as a collect error). Its migrations must never run beside
a writing inventory-svc, and a rollout must not run two of its pods at once; a
failing db migrate keeps it down until it succeeds. Recreate covers rollouts
only: restart it with kubectl rollout restart deploy/inventory-svc, not by
deleting the pod (a deleted or evicted pod can overlap its replacement; the
UNIQUE discovery key still keeps that from duplicating rows). See
Deploying Fleeger on Kubernetes.
Do not override this to RollingUpdate.
The internal token, the DB password and VictoriaMetrics' two admin-API keys are generated once and reused on every upgrade, where Helm can read the cluster while it renders; a render that cannot fails instead (see GitOps). The credential master key is reused too, but on stricter terms, because it is the only thing that decrypts stored infra credentials and alert-channel secrets — replacing it makes all of them permanently undecryptable, and nothing fails until the next decrypt:
Omit
credential.keywhen upgrading. The key already in the cluster is reused. The upgrade commands above deliberately do not pass it. This, and the refusal below, work only when Helm can read the cluster while it renders (it looks the live key up).helm templateand--dry-runwithout=server, and GitOps tools that render client-side, such as Argo CD, cannot: there an upgrade without the key fails to render (credential.key is required, pointing here), and a key that differs from the live one is not refused. If you deploy that way, setcredential.existingSecret, as the install commands do; the chart then never generates, compares or replaces the key. A release that already runs keeps its key in the chart-made Secret,<fullname>-credential-key(for releasefleeger,fleeger-fleeger-hub-credential-key): copy that key into a Secret of your own first, and pointcredential.existingSecretat the copy. A Secret holding any other key makes every stored credential undecryptable, so the copy stops when it finds no key rather than create an empty one:key=$(kubectl -n fleeger get secret fleeger-fleeger-hub-credential-key \ -o jsonpath='{.data.credential-key}' | base64 -d) && [ -n "$key" ] && printf '%s' "$key" | kubectl -n fleeger create secret generic \ fleeger-credential-key --from-file=credential-key=/dev/stdin || echo "no credential key copied; nothing created" >&2Pointing
credential.existingSecretat the chart-made Secret itself is not enough under GitOps: Helm keeps it when it leaves the chart, but a GitOps tool may prune it. Helm prints a warning on every install and upgrade while noexistingSecretis set;helm template, and so Argo CD, never shows it.Passing a different
credential.keyis refused at render time rather than applied. Re-running an install line that passed--set credential.key=$(head -c 32 /dev/urandom | base64)from shell history is the usual way this happens: it mints a new key every time.To rotate on purpose: put the current key in a Secret referenced by
credential.retiredKeysSecretso existing ciphertexts still decrypt, then upgrade withhelm upgrade … --reset-then-reuse-values --set credential.allowKeyChange=trueand the new key, and once it is live runservice rekeyin both inventory-svc (credentials) and alerting-svc (alert-channel secrets). Once a second run of each reports everything already current, drop the retired key and re-arm the refusal withhelm upgrade … --reset-then-reuse-values --set credential.retiredKeysSecret= --set credential.allowKeyChange=false:--reset-then-reuse-valueskeeps every value you set,allowKeyChange=trueincluded, so until you do, the next straycredential.keyis applied instead of refused. Keep the old key itself, off-box, for as long as you keep any backup taken before both rekeys were confirmed: restoring one needs it back incredential.retiredKeysSecret, and bothrekeyruns again afterwards (see Backup & disaster recovery).
The Secret is annotated helm.sh/resource-policy: keep, so it also survives
helm uninstall.
Edge trusted proxies (BREAKING, first shipped in chart 0.8.1)
Chart 0.8.1 is the first release with edge.trustedProxies, and it
refuses to render without it when an ingress is in front of the edge. An
upgrade that succeeded on an earlier chart — and equally a fresh install with
ingress.enabled=true — now stops, before anything is applied, with:
ingress.enabled is true but edge.trustedProxies is empty.
The refusal is deliberate, and it is the fix, not the regression. The
/api/auth/login rate limit is keyed on the address the edge resolves for the
caller, and with an ingress in front and no hop trusted that is the
controller's own pod address for every caller. The limit then collapses
into a single shared bucket of 10r/m for the whole deployment, which any
unauthenticated client can exhaust to lock every operator out of login.
The audit log degrades too, though differently from before. Earlier charts
appended to X-Forwarded-For, so the client's address was somewhere in the
chain — unverified and trivially forgeable, but present. This chart
overwrites X-Forwarded-For with the address it actually resolved, which
is what stops a caller forging their own entry; the cost is that with nothing
trusted the log records the ingress controller for every action and the real
client is not recorded at all.
Both failures were silent — no crash-loop, no error, nothing in any log — so the chart refuses at render time rather than merely warning, and the edge also answers such requests with a 503 (below).
To upgrade, name the CIDR of the one hop directly in front of the edge:
helm upgrade fleeger charts/fleeger-hub -n fleeger --reset-then-reuse-values \
--set 'edge.trustedProxies[0]=10.42.0.0/16' # e.g. the ingress controller's pod CIDRTrust only that hop. A wider range lets anyone who can reach the edge forge their own source address, dodge the rate limit and write what they like into the audit log — worse than having no forwarded header at all.
Every entry is also checked at render to be an IPv4 or IPv6 address or CIDR, whether or not the ingress is enabled or the check above is waived: an
unreplaced placeholder such as <ingress-controller-pod-CIDR>, a host name or
a prefix out of range is refused with an error naming the entry, instead of
installing an edge whose nginx will not start.
Beyond that, the render-time check sees only the chart's own ingress, and only
whether edge.trustedProxies is empty. So the edge also checks at runtime: a
request that carries X-Forwarded-For or X-Real-IP but whose client address
the edge did not take from it came through a hop that is not trusted, and gets
a 503 whose body starts "Fleeger is behind a proxy the edge does not trust".
That catches a CIDR that misses the controller, an HTTP proxy or load balancer
from outside the chart (a Gateway, an L7 cloud LB), a waived check with a proxy
in the path after all, and a trusted proxy that does not send the header
edge.realIpHeader names. The 503 covers everything through that proxy: the
UI, the API, agents on /nats (their bootstrap log shows only "503"), and
auth-svc when nats.publicUrl is a wss:// URL through the ingress (then
revoking or offboarding an agent fails, so the agent stays admitted).
Requests without either header (port-forward, NodePort, in-cluster clients)
are served as before; a client that sends one itself, for example through an
office HTTP proxy or an egress proxy in a pod's HTTP_PROXY, gets the 503.
If you see that 503 through your ingress, read the peer's address off the
edge's log (kubectl -n fleeger logs deploy/edge, the first field of each
line). If it is not in edge.trustedProxies, set the CIDR that covers it. If it
already is, the proxy does not send edge.realIpHeader: set that to the header
it does send. The edge restarts by itself when its config changes. Before an
upgrade from a chart without this check, the same log tells you in advance:
outside requests logged with the controller's address as the first field come
through a hop the new edge will refuse.
What the edge sees as the peer is not always the proxy's pod address: a cloud
LB in NodePort/instance mode with externalTrafficPolicy: Cluster arrives
from a node address, and a service-mesh sidecar on the edge pod (Istio,
Linkerd) hides the peer behind its own loopback address (127.0.0.6 with
Istio, 127.0.0.1 with Linkerd), which cannot be trusted without trusting
every meshed pod. Read the address from the log rather than guessing. With a
mesh, keep the edge pod out of it (sidecar.istio.io/inject: "false" as a pod
label, or linkerd.io/inject: disabled as a pod annotation), and if the mesh
enforces mTLS, let gateway, auth-svc and NATS accept plaintext from the edge;
or keep the sidecar and bypass it only for the edge's inbound port 80
(traffic.sidecar.istio.io/excludeInboundPorts or
config.linkerd.io/skip-inbound-ports; see Fleeger on Kubernetes — plain manifests). Set
either through edge.podLabels and edge.podAnnotations, which go on the edge
pod only:
edge:
podLabels:
sidecar.istio.io/inject: "false" # Istio
podAnnotations:
linkerd.io/inject: disabled # Linkerd
A Cloudflare Tunnel whose ingress rule points at the edge Service makes
cloudflared the hop: trust its pods' addresses and take the client address
from the header Cloudflare sends:
helm upgrade fleeger charts/fleeger-hub -n fleeger --reset-then-reuse-values \
--set 'edge.trustedProxies[0]=<cloudflared-pod-CIDR>' \
--set edge.realIpHeader=CF-Connecting-IPIf the rule points at an ingress controller instead, the controller stays the
edge's hop and has to trust cloudflared itself. Cloudflare's published IP
ranges are the hop only when Cloudflare's proxy connects to the edge itself
through a load balancer that keeps the source address; with a tunnel, trusting
them leaves the 503 in place.
An L4 load balancer that sends no forwarded header (the usual
LoadBalancer Service, an NLB) is not detected, and every client may then
share a node's address; put an HTTP ingress in front of the edge instead.
If the ingress genuinely is not in the request path (clients reach the edge
directly over NodePort or a port-forward), leave edge.trustedProxies empty
and waive the check with --set edge.preflightTrustedProxies=false (on an
upgrade:
helm upgrade … --reset-then-reuse-values --set edge.preflightTrustedProxies=false).
Like any value you set, the waiver then stays in force on later upgrades; set
it back to true if an ingress is ever put in the request path.
edge.realIpHeader, edge.realIpRecursive and edge.loginRateLimit tune the
same machinery and are documented in the Edge / ingress / network policy
table under Values reference.
NATS volume grown to 3Gi (upgrading from a chart that shipped 1Gi)
natsServer.persistence.size now defaults to 3Gi, up from 1Gi. NATS
declares 75% of the volume as its JetStream store, and the SPOKES account's
2 GB JetStream grant does not fit in less. An upgrade therefore asks
Kubernetes to grow the existing nats-data claim.
The StorageClass allows volume expansion (
allowVolumeExpansion: true): nothing to do. The claim grows in place, keeping its data.It does not (k3s
local-path, for one): the upgrade fails withcannot patch "nats-data" with kind PersistentVolumeClaim: persistentvolumeclaims "nats-data" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resizebut Helm has already applied everything else, so NATS restarts declaring a 2.25 GiB store on the old 1 GiB disk, and fills it before any stream limit is reached. Migrate the claim with the recipe below, then run the same upgrade again.
The recipe stops NATS, copies nats-data into a temporary claim, replaces
nats-data with a 3Gi claim and copies the data back. Keep the data:
nats-data holds the metrics and logs ingestion-svc has not forwarded yet,
and the resolver's SPOKES account JWT, which carries auth-svc's agent
revocations. (A NATS started on an empty claim gets SPOKES back from the
nats-push hook, and auth-svc adds the revocations back within about 10
seconds of seeing them missing, but only while it is running; until then every
offboarded agent's credentials work again.) Agents cannot publish while NATS
is down, so pick a quiet moment.
Set the first two lines, then paste the whole block. It runs in a subshell
that stops at the first failing step, and it is safe to run again after a
failure: it never copies over the temporary claim once that holds a complete
copy, and never deletes nats-data before then. STORAGE_CLASS must match
natsServer.persistence.storageClass (empty for the cluster default):
a claim's class cannot be changed, so a mismatch makes the upgrade fail again.
The copy runs as root in a busybox pod, to keep the files' owners; a
namespace that enforces the restricted Pod Security Standard, or a quota
that requires resource limits, refuses that pod, so relax them for the
migration.
NS=fleeger RELEASE=fleeger SIZE=3Gi # your namespace, release name and natsServer.persistence.size
STORAGE_CLASS="" # natsServer.persistence.storageClass, if you set one
(
set -e
# Copy one claim's files onto another. A failed copy pod is left for
# `kubectl logs`, and removed by the next run.
copy() {
kubectl -n "$NS" delete pod nats-data-copy --ignore-not-found --wait
kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata: {name: nats-data-copy}
spec:
restartPolicy: Never
containers:
- name: copy
image: busybox:1.38
command: ["sh", "-c", "cp -a /from/. /to/ && sync"]
volumeMounts: [{name: from, mountPath: /from}, {name: to, mountPath: /to}]
volumes:
- {name: from, persistentVolumeClaim: {claimName: $1}}
- {name: to, persistentVolumeClaim: {claimName: $2}}
EOF
while :; do
phase="$(kubectl -n "$NS" get pod nats-data-copy -o jsonpath='{.status.phase}')"
case "$phase" in
Succeeded) break ;;
Failed) echo "copying $1 to $2 failed: kubectl -n $NS logs nats-data-copy" >&2; return 1 ;;
esac
sleep 2
done
kubectl -n "$NS" delete pod nats-data-copy
}
# A new, empty claim. It carries Helm's marks, like the chart's own claim, and
# one of ours that says it is the new one.
claim() {
kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: $1
labels: {app.kubernetes.io/managed-by: Helm}
annotations:
meta.helm.sh/release-name: $RELEASE
meta.helm.sh/release-namespace: $NS
fleeger.dev/nats-migration: new
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: $SIZE}}
${STORAGE_CLASS:+ storageClassName: $STORAGE_CLASS}
EOF
}
note() { kubectl -n "$NS" get pvc "$1" --ignore-not-found -o jsonpath='{.metadata.annotations.fleeger\.dev/nats-migration}'; }
kubectl -n "$NS" scale deploy/nats --replicas=0
while [ -n "$(kubectl -n "$NS" get pods -l app.kubernetes.io/name=nats,app.kubernetes.io/instance="$RELEASE" -o name)" ]; do sleep 2; done
# 1. Copy nats-data aside, unless an earlier run already has.
if [ "$(note nats-data-move)" != copied ]; then
claim nats-data-move
copy nats-data nats-data-move
kubectl -n "$NS" annotate pvc nats-data-move fleeger.dev/nats-migration=copied --overwrite
fi
# 2. Replace nats-data with the larger claim, unless an earlier run already has.
if [ "$(note nats-data)" != new ]; then
kubectl -n "$NS" delete pvc nats-data --ignore-not-found --wait
claim nats-data
fi
# 3. Copy the data back and drop the temporary claim.
copy nats-data-move nats-data
kubectl -n "$NS" delete pvc nats-data-move
)Then repeat the upgrade command you started with. It brings NATS back on the new claim. The old 1Gi volume is released with its claim, and deleted or kept according to the StorageClass's reclaim policy.
A few more things to know:
--reuse-valueskeeps the old1Giand is refused at render. Use--reset-then-reuse-values(see above) whenever the upgrade passes a value.- An upgrade run with
--atomic(or--rollback-on-failure) does not half-apply: Helm rolls the release back when the claim is refused. Migrate the claim the same way, then upgrade again. - Once the claim has grown, rolling back to a chart that shipped
1Gifails on the claim, because Kubernetes never shrinks one. PinnatsServer.persistence.sizeto the claim's size if you need to roll back.
The raw manifests in deploy/k8s/base/ grew the claim the same way.
The NATS auth Secret must not hold the credential key
The chart now refuses to render when nats.authSecret names the Secret that
holds the credential master key: the chart-made <fullname>-credential-key,
credential.existingSecret or credential.retiredKeysSecret. That Secret is
mounted whole into auth-svc and the NATS pod, which would both then hold the
key that decrypts every stored infrastructure credential and
alert-channel secret. If you keep both in one Secret today, move the
credential key (and any retired keys) into a Secret of its own and point
credential.existingSecret at it before upgrading.
VictoriaMetrics admin-API keys and the alerting hub list
VictoriaMetrics now requires keys for its admin endpoints. The chart generates
both into <fullname>-victoriametrics on the upgrade and keeps them: delete-key
(-deleteAuthKey), which only ingestion-svc holds, and snapshot-key
(-snapshotAuthKey, -forceMergeAuthKey, -forceFlushAuthKey,
-search.resetCacheAuthKey, -reloadAuthKey), which only the vmbackup sidecar
holds, in both of its snapshot URLs. Nothing to do unless you call those
endpoints yourself, which now needs authKey, or bring your own Secret with
victoriametrics.auth.existingSecret (both keys, hex). ingestion-svc checks at
startup that a delete without the key is refused and one with it accepted, and
logs an error otherwise; a delete VictoriaMetrics refuses for its key is
retried rather than dropped.
alerting-svc now also refuses notification channels aimed at the host of
loki.url and at anything in alerting.hubServices. Add your pod and service
CIDRs there: alerting-svc knows the ClusterIP a service name resolves to, not
the pod IPs behind it. The core agent's FLEEGER_HUB_SERVICES gets the host of
loki.url added too (the default names are written out with it when
coreAgent.hubServices is empty).
Ingestion streams created by an earlier release (one-off, manual)
Earlier releases created the metrics and logs JetStream streams unbounded:
they keep every message after ingestion has forwarded it, so on a hub that ran
one of those releases they grow until the NATS account's disk is full, at which
point every agent's publishes are refused and metric and log ingestion stop.
Current releases create them bounded, but cannot convert an existing stream —
NATS refuses to change a stream's retention in place — so an upgraded hub keeps
the old stream, and ingestion-svc logs this at every start until it is fixed:
the 'metrics' stream already exists with Limits retention, not WorkQueue: … Run `service stream migrate` …
Fix it once, after the upgrade has fully rolled out and while ingestion-svc
is running (it is what drains the backlog):
kubectl -n fleeger exec deploy/ingestion-svc -- /usr/local/bin/service stream migrate
# the compose stack:
# docker compose -f deploy/compose/compose.yaml exec ingestion-svc /usr/local/bin/service stream migrateFor each stream it waits for ingestion-svc to forward everything the stream
holds, stops the stream accepting new publishes so nothing can arrive after
that check, then deletes it and creates the bounded one. It never deletes a
message that has not reached VictoriaMetrics or Loki. ingestion-svc needs no
restart: it reattaches to the new stream within seconds, losing nothing. The
command checks for that and tells you to restart ingestion-svc if it has not
reattached within 30 seconds. --stream metrics or --stream logs limits it
to one stream. A fresh install never needs any of this.
What to expect:
- Usually a few milliseconds of failed publishes per stream (measured at
1–5 ms against a local broker; roughly five JetStream round trips). If a few
messages arrived just before the stream stopped accepting, the command first
waits for
ingestion-svcto forward them — up to 5 seconds, with publishes still failing — and restores the stream and refuses if they are not. An agent whose publish lands in that window logs it (publish failed for …,heartbeat publish failed,k8s log publish … failed) and drops that sample — for logs, that pod's batch for the cycle. Alogspublish fails at once; ametricspublish fails when the agent's 5-second ack wait runs out (gateway-svc and alerting-svc also listen onmetrics.>), holding that agent's cycle up by 5 seconds. Where the stream is the NATS account's only one (metricson a hub without Loki), NATS now and then refuses to create the new stream right after the old one is deleted (error creating store for stream); the command retries that for up to about 1.5 seconds, publishes still failing, rather than give up. - Settings you tuned are kept. The new stream is built from the old one's
config — replicas, duplicate window, extra subjects, per-subject limits,
description, placement. Only these are reset to
ingestion-svc's configuration, and the command prints each one it changed: retention, discard policy, the three size/age limits,discard_new_per_subject(incompatible with the new discard policy), and a duplicate window longer than the newmax_age(shortened to it; NATS refuses otherwise). - It refuses rather than guesses, exits non-zero and changes nothing, when:
the backlog does not drain within
--drain-timeout-secs(default 300 — check thatingestion-svcis running and can reach VictoriaMetrics / Loki; on a very busy hub the backlog may never be empty at a check, so try a quieter moment); the stream has a consumer other thaningestion-svc's own (remove it first), or its consumer is not shaped the wayingestion-svccreates it (push-based, not explicitly acked, filtered, or not starting at the beginning of the stream); or the stream does not look like oneingestion-svccreated (sealed, a mirror, sourced, template-owned, or subjects that do not covermetrics.>/logs.>). If aningestion-svcstill on the previous release recreates the old, unbounded stream mid-run, the command reports a failure. - It is safe to re-run. A stream that is already bounded, or does not
exist, is left alone. The command warns just before the stream stops
accepting publishes; if a run is cut off after that without reporting an
outcome, the stream may be left accepting no publishes, and the running
ingestion-svcdoes not notice (it logs it only at its next start). Run the command again: it finishes the job. For about 35 seconds such a stream stays reserved for the run that stopped it, and a re-run is refused rather than interfering. Two runs on the same stream at once do not both act on it: at least one backs off, and if both do, the stream is left as it was — run it again. That protection assumes a run gets from its last check to its delete within those 35 seconds; a run frozen for longer at exactly that point could remove a replacement another run had just created. An interruption between deleting the old stream and creating the new one leaves no stream: theningestion-svcrecreates it within seconds, bounded but with default settings. - Hubs with log ingestion switched off. Nothing drains an old
logsstream there. If everything in it was forwarded while log ingestion was still on, it is migrated (and the command warns that nothing reattached — expected). If it holds messages never forwarded, the command waits out the drain timeout on it and exits non-zero;--stream metricsskips it, but that stream then stays unbounded, shares the account's disk budget withmetrics, and gets no startup warning, because nothing opens it. There is no lossless fix from here: switch log ingestion on long enough for the command to drain and migrate it, or accept losing those buffered lines and remove the stream by hand.
GitOps
Argo CD and other tools that render the chart client-side, like helm template,
--dry-run without =server and the helm-diff plugin without its
--dry-run=server, cannot read the cluster: lookup sees nothing. There the chart cannot reuse the internal token, the DB password and
VictoriaMetrics' keys it generated before, and would generate new ones on every
render, changing them under running pods. So it refuses to generate them there,
and the render fails naming the values still to set: set the values
listed, bringing your own Secrets as the credential key already is (or pin
internal.token and surrealdb.password inline).
# A new install: fresh values, each through a pipe, never on a command line.
head -c 30 /dev/urandom | base64 | tr -d '\n+/=' | \
kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=/dev/stdin
head -c 24 /dev/urandom | base64 | tr -d '\n+/=' | \
kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=/dev/stdin
# bash, for the <(…); VictoriaMetrics does not trim its keys, so no newline.
kubectl -n fleeger create secret generic fleeger-victoriametrics \
--from-file=delete-key=<(openssl rand -hex 32 | tr -d '\n') \
--from-file=snapshot-key=<(openssl rand -hex 32 | tr -d '\n')A release that already runs keeps its values in the chart-made Secrets
<fullname>-internal, <fullname>-db and <fullname>-victoriametrics (for
release fleeger, fleeger-fleeger-hub-internal and so on). Copy them into
Secrets of your own instead, so the pods keep the values they run with. Set
full to your release's fullname. The copy (bash, for the <(…), which keeps
the values off the command line) stops, creating nothing, when a value cannot
be read, rather than create a Secret with an empty one. If a create fails
(say, the Secret already exists from an earlier run), the ones after it are
skipped; delete what was created and run it again:
full=fleeger-fleeger-hub
from() { kubectl -n fleeger get secret "$full-$1" -o go-template="{{index .data \"$2\" | base64decode}}"; }
if it=$(from internal internal-token) && db=$(from db db-password) &&
dk=$(from victoriametrics delete-key) && sk=$(from victoriametrics snapshot-key) &&
[ -n "$it" ] && [ -n "$db" ] && [ -n "$dk" ] && [ -n "$sk" ]; then
kubectl -n fleeger create secret generic fleeger-internal --from-file=internal-token=<(printf '%s' "$it") &&
kubectl -n fleeger create secret generic fleeger-db --from-file=db-password=<(printf '%s' "$db") &&
kubectl -n fleeger create secret generic fleeger-victoriametrics \
--from-file=delete-key=<(printf '%s' "$dk") --from-file=snapshot-key=<(printf '%s' "$sk")
else
echo "a value could not be read; nothing created" >&2
fi
unset it db dk sk full; unset -f fromThen deploy with
credential:
existingSecret: fleeger-credential-key
internal:
existingSecret: fleeger-internal
surrealdb:
existingSecret: fleeger-db
victoriametrics:
auth:
existingSecret: fleeger-victoriametrics
With scheduled backups on (backup.enabled), the backup signing key is one
more: bring it as backup.signing.existingSecret, a new one with
openssl genpkey -algorithm ed25519 | kubectl -n fleeger create secret generic fleeger-backup-signing --from-file=signing-key=/dev/stdin, or a copy of the
chart-made <fullname>-backup-signing (its signing-key entry, as above) for a
release that already signs with it; a new key signs sets that only its own
public key in backup.signing.publicKeys accepts.
Pointing the values at the chart-made Secrets themselves is not enough: Helm keeps them when they leave the chart, but a GitOps tool may prune them. A release that was already deployed client-side before this refusal has had these values replaced on every sync; the DB password, though, takes effect only on a new datastore, so if the services cannot log in after the copy, the datastore still has the first one.
A real install or upgrade reads the cluster and is not refused, with
--create-namespace too; the chart lists the namespace's ServiceAccounts to
tell, so on an install (or an upgrade that first creates VictoriaMetrics' keys) the
deploying user needs list on them, as the built-in edit role has. To see what an upgrade would change, render against the
cluster: helm-diff 3.9 or later with its --dry-run=server, or Helm's own
--dry-run=server. To only look at a render with helm template,
turn allowOfflineSecretGeneration on for that render; never deploy such a
render.
Toggling core services
Every core service is enabled by default and can be turned off individually — e.g. bring your own OIDC by disabling auth-svc:
helm install fleeger charts/fleeger-hub -n fleeger \
--set image.tag=local \
--set services.auth-svc.enabled=falseservices.<name>.enabled (default true) exists for: auth-svc, inventory-svc, gateway-svc, ingestion-svc, alerting-svc, control-svc.
Values reference
Only the important values are listed. See values.yaml for the full set and inline notes.
Images
| Key | Default | Description |
|---|---|---|
image.registry |
fleeger |
Registry prefix; service images are <registry>/<service>:<tag>. Set it for an OCI install: the default names Docker Hub, where Fleeger publishes nothing. |
image.tag |
"" |
Image tag for all Fleeger images (services, frontend, spoke-agent, backup). Empty means the chart's appVersion, the tag its release published; set local for images built from a checkout. A tag set here stays pinned across upgrades until cleared with --set image.tag=. The frontend image runs as uid/gid 101 with no capabilities, so it must be built from a Dockerfile.frontend that makes its nginx config world-readable (this chart version's does); an older frontend image built from a group-only checkout fails with open() "/etc/nginx/conf.d/default.conf" failed (13: Permission denied). |
image.pullPolicy |
IfNotPresent |
|
imagePullSecrets |
[] |
Names of pull secrets for a private (e.g. GitLab) registry. |
logLevel |
info |
Secrets
| Key | Default | Description |
|---|---|---|
credential.key |
"" |
Base64 32-byte master key. Required unless existingSecret is set. Checked at render as the services decode it: trimmed, standard base64, exactly 32 bytes. Never checked: the key already in the cluster, which an upgrade reuses as it is. |
credential.existingSecret |
"" |
Secret holding the key under credential-key. |
credential.preflightExistingSecret |
true |
Fail helm install/upgrade when credential.existingSecret exists but has no credential-key entry, or its entry is not a valid key (standard base64 of 32 bytes; the message names the Secret, never the value), instead of leaving the pods in CreateContainerConfigError or crash-looping. An absent Secret is not refused (the install notes remind you). The check uses lookup, which sees nothing under helm template, --dry-run without =server or Argo CD, so it runs only on a real install or upgrade. Needs Secret read access; set false if your RBAC forbids it. |
credential.retiredKeysSecret |
"" |
Secret with retired-keys (comma-separated base64) so old ciphertexts decrypt during a rotation; clear after service rekey has run in both inventory-svc and alerting-svc. |
internal.token |
"" |
Shared token the gateway attaches and backends require. Empty ⇒ chart generates one (kept across upgrades; not in a client-side render, see GitOps). |
internal.existingSecret |
"" |
Secret with an internal-token key. |
allowOfflineSecretGeneration |
false |
Generate the internal token, the DB password and VictoriaMetrics' keys even where the render cannot read the cluster (helm template, --dry-run without =server, Argo CD). Off, such a render fails naming the existingSecret values to set instead (GitOps). Only for a render that is looked at: deployed, every render changes those values. |
victoriametrics.auth.existingSecret |
"" |
Secret with VictoriaMetrics' admin-API keys: delete-key (-deleteAuthKey, reaches only VictoriaMetrics and ingestion-svc) and snapshot-key (-snapshotAuthKey and the other maintenance keys, reaches only VictoriaMetrics and the vmbackup sidecar). Two different hex keys (openssl rand -hex 32). Empty ⇒ the chart generates both into <fullname>-victoriametrics (kept across upgrades; not in a client-side render, see GitOps). |
NATS
| Key | Default | Description |
|---|---|---|
nats.authSecret |
fleeger-nats-auth |
Name of the Secret holding the gen-auth.sh material. Required. Must not be the Secret holding the credential key (refused at render). |
nats.preflightAuthSecret |
true |
Fail helm install/upgrade (naming the missing keys) when that Secret exists but lacks a key the render references, instead of letting the pods fail later. Needs Secret read access; set false if your RBAC forbids it. |
nats.publicUrl |
"" |
NATS URL agents connect to. Empty ⇒ edge WebSocket (ws://edge/nats, in-cluster). For external agents set your wss:// endpoint, e.g. wss://fleeger.example.com/nats. |
SurrealDB
| Key | Default | Description |
|---|---|---|
surrealdb.image |
surrealdb/surrealdb:v3.3.4 |
|
surrealdb.user |
root |
|
surrealdb.password |
"" |
Root password. Empty ⇒ chart generates one. Prefer surrealdb.existingSecret for your own: --set puts it on the command line. |
surrealdb.existingSecret |
"" |
Secret with a db-password key (BYO password). |
surrealdb.persistence.size |
5Gi |
|
surrealdb.persistence.storageClass |
"" |
NATS server (dependency)
| Key | Default | Description |
|---|---|---|
natsServer.image |
nats:2.15-alpine |
|
natsServer.persistence.size |
3Gi |
|
natsServer.persistence.storageClass |
"" |
VictoriaMetrics
| Key | Default | Description |
|---|---|---|
victoriametrics.image |
victoriametrics/victoria-metrics:v1.153.0 |
|
victoriametrics.retentionPeriod |
"6" |
Retention in months. |
victoriametrics.maxQueryDuration |
30s |
Query-cost bound. |
victoriametrics.maxConcurrentRequests |
"16" |
|
victoriametrics.maxQueryLen |
16KB |
|
victoriametrics.maxHourlySeries |
1000000 |
Hub-wide backstop on series: the most distinct series written in one hour of VictoriaMetrics' uptime (-storage.maxHourlySeries), every location and the hub's own counted, existing series included: once reached, every series not yet written in that hour loses its samples. ingestion-svc's per-location ingestion.maxNewSeriesPerLocationHour comes first. Size it above your hub-wide active series plus their hourly churn; VictoriaMetrics needs roughly 1 KiB of memory per active series. VictoriaMetrics stores its own metrics (-selfScrapeInterval=30s), and the built-in vm_series_limit_reached alert fires on its drops. 0 is no limit; not set is refused (as an upgrade with --reuse-values from an older chart leaves it: use --reset-then-reuse-values). |
victoriametrics.maxDailySeries |
5000000 |
The same per day of uptime (-storage.maxDailySeries): above your hub-wide active series plus their daily churn. 0 is no limit. |
victoriametrics.auth.existingSecret |
"" |
See Secrets. |
victoriametrics.persistence.size |
10Gi |
|
victoriametrics.persistence.storageClass |
"" |
Core service toggles
| Key | Default | Description |
|---|---|---|
services.auth-svc.enabled |
true |
Disable to bring your own OIDC. |
services.inventory-svc.enabled |
true |
|
services.gateway-svc.enabled |
true |
|
services.ingestion-svc.enabled |
true |
|
services.alerting-svc.enabled |
true |
|
services.control-svc.enabled |
true |
Bundled core agent (opt-in)
| Key | Default | Description |
|---|---|---|
coreAgent.enabled |
false |
Serves every location flagged local_collector. Requires a core-agent.creds entry in the NATS auth Secret. That credential needs only pub inv._core.> for inventory, and pub metrics._core.>/logs._core.>/_INBOX._core.> and sub commands._core.> for its metrics, logs and commands: after rolling out a release with both changes (every replica of every hub service, not only the core agent: an older ingestion-svc drops the _core subjects, an older control-svc or inventory-svc sends no command there, an older alerting-svc credits no _core heartbeat), narrow an older one with FLEEGER_AUTH_SVC="kubectl -n <namespace> exec deploy/auth-svc -- /usr/local/bin/service" deploy/nats/gen-auth.sh fix-perms core-agent (it also revokes the core agent's earlier JWTs in auth-svc's revocation ledger, which every later push of the SPOKES account re-asserts, and disconnects it), update the Secret with the reissued core-agent.creds and kubectl rollout restart the core agent. Until then it keeps working on each location's own subjects, and logs a warning at start-up. To roll back to a release without the _core subjects afterwards, widen the credential first (Agent credentials & revocation). |
coreAgent.pollIntervalSecs |
15 |
|
coreAgent.tmpSizeLimit |
2Gi |
Writable /tmp emptyDir — the agent's scratch space (rootfs is read-only). |
coreAgent.hubServices |
[] |
The hub's own services, which the core agent never probes or collects from: host names, IPs or CIDR ranges. Empty refuses this chart's service names, by name and by the ClusterIP they resolve to. Setting it replaces that list. The host of loki.url is always added. Whatever the list, this release's own pods are refused by their pod IPs too, listed every cycle by their app.kubernetes.io/part-of and app.kubernetes.io/instance labels, which needs pods list in the namespace (granted by coreAgent.rbac.create). Loopback, link-local, the agent's NATS host and this cluster's own Kubernetes API (kubernetes.default.svc and the KUBERNETES_SERVICE_HOST address) are always refused. |
coreAgent.resources |
requests 50m/64Mi, limit 256Mi |
Alerting
| Key | Default | Description |
|---|---|---|
alerting.hubServices |
[] |
More of the hub's own services no notification channel may be aimed at: host names, IPs or CIDR ranges, rendered into alerting-svc's [alerting] hub_services next to the host of loki.url. Add the pod and service CIDRs to refuse the pod IPs of VictoriaMetrics and Loki, unless a notification target of yours lives in them. An invalid entry stops alerting-svc from starting. |
Ingestion streams
The bounds of the JetStream metrics and logs streams that buffer what the agents send while ingestion-svc is down, rendered into ingestion-svc's [ingestion] section. Each takes a whole number; empty keeps ingestion-svc's own default, and ingestion-svc refuses to start on a value outside its bounds, naming the key. Set them here: ingestion-svc refuses BUMACS_INGESTION_* environment variables, which never reached these keys.
| Key | Default | Description |
|---|---|---|
ingestion.streamMaxAgeSecs |
"" |
How long a stream keeps a message ingestion-svc has not forwarded yet, in seconds: the longest ingestion outage that loses nothing. Empty ⇒ 259200 (3 days). At least 60. |
ingestion.streamMaxBytes |
"" |
Disk budget of each stream, in bytes: the bound that makes a stream drop its oldest messages instead of filling the NATS volume. Empty ⇒ 201326592 (192 MiB). Both streams reserve it, so twice it must fit under JetStream's max_file_store (75% of natsServer.persistence.size) and the SPOKES account's 2000000000-byte grant: at most 1000000000 on any volume the chart accepts (its size floor keeps max_file_store at or above the grant). The render refuses a larger value. It counts both streams even on a hub without loki.url, so that enabling Loki later does not make ingestion-svc halve the cap. |
ingestion.streamMaxMsgs |
"" |
The most messages each stream keeps. Empty ⇒ 1000000. |
ingestion.maxNewSeriesPerLocationHour |
"" |
The most new metric series one location may make per hour, by the hour its messages were published to NATS. Past it ingestion-svc drops the points of further new series (the location's existing series and every other location are written), logs it once per location and hour, and fires the built-in series_bound_exceeded alert. Empty ⇒ 50000; ingestion-svc takes 1000 to 250000 and keeps up to about 2.3 MiB per location at the default (18 MiB at 250000) of resources.services' 256Mi limit. After an ingestion-svc restart, or an agent upgrade that renames its series, every series of a location counts as new: with S series a location writes per hour, S up to the bound loses nothing, up to twice it loses the excess until the next hour, more keeps losing points. Size it above your largest location (a Redfish-heavy one, 200 hosts with a few hundred sensors each, reaches the default). victoriametrics.maxHourlySeries is the hub-wide backstop. |
Scheduled backups (opt-in)
| Key | Default | Description |
|---|---|---|
backup.enabled |
false |
CronJob dumps every SurrealDB service DB to S3 + pushes the dead-man's-switch metric. Needs backup.existingSecret, backup.s3Bucket and, while backup.vmbackup.enabled, backup.vmbackup.dst; the render names whichever are missing. |
backup.image |
"" |
Empty ⇒ <registry>/backup:<tag>, with <tag> resolved as for image.tag. |
backup.schedule |
"0 3 * * *" |
Cron (cluster tz). |
backup.s3Bucket |
"" |
e.g. s3://fleeger-backups. |
backup.s3EndpointUrl |
"" |
Optional. Only for non-AWS S3-compatible targets (Hetzner object storage, MinIO, Ceph). Empty ⇒ the AWS default endpoint for backup.region. |
backup.region |
us-east-1 |
|
backup.existingSecret |
"" |
S3 creds Secret (access-key-id, secret-access-key). |
backup.retentionDays |
"" |
Prune sets older than N days (empty ⇒ keep all). |
backup.tmpSizeLimit |
2Gi |
Scratch for dumps. |
backup.signing.existingSecret |
"" |
Secret with the backup signing key, signing-key (Ed25519, PEM), mounted by the CronJob only. See Secrets. Empty ⇒ chart-generated <fullname>-backup-signing. |
backup.signing.publicKeys |
[] |
The public keys scripts/restore.sh accepts a set's signature from, each the base64 line of its PEM (openssl pkey -in <key> -pubout | sed -n 2p); given to the CronJob as FLEEGER_BACKUP_VERIFY_KEYS (it warns when its own key is not among them), and checked at render. A restore runs in a pod without the signing key (deploy/k8s/optional/restore-s3-pod.yaml), given the same keys. List the old key too while rotating; drop a leaked one at once. Keep a copy off the hub. |
backup.vmbackup.enabled |
true |
VictoriaMetrics vmbackup sidecar (incremental → S3; restore with vmrestore). Gets the snapshot key and sends it in both snapshot URLs. |
backup.vmbackup.image |
victoriametrics/vmbackup:v1.153.0 |
|
backup.vmbackup.intervalSeconds |
3600 |
|
backup.vmbackup.dst |
"" |
e.g. s3://fleeger-backups/vm (distinct from the DB sets). |
Edge / ingress / network policy
| Key | Default | Description |
|---|---|---|
edge.enabled |
true |
nginx edge serving the UI and proxying /api. |
edge.image |
nginx:1.31.8-alpine |
Runs as uid/gid 101 with no capabilities, writing only to emptyDirs at /var/cache/nginx and /run. An override must work that way too, and be the official nginx image at 1.29.7 or later: the edge follows a recreated hub Service with server ... resolve and turns upstream keepalive off with keepalive 0, which older nginx refuses, and its entrypoint's 15-local-resolvers.envsh supplies the resolver. |
edge.service.type |
ClusterIP |
Set NodePort (+ edge.service.nodePort) for quick access without an ingress. |
edge.service.port |
80 |
|
edge.trustedProxies |
[] |
CIDRs of the one hop directly in front of the edge, as the edge sees it — your ingress controller's pod CIDR, or the source range of the cloud LB (node addresses for an LB in NodePort mode; read it from the edge's log). For a peer inside these ranges nginx replaces the client address with the one from edge.realIpHeader; for anyone else the header is ignored. Empty means trust nobody. Each entry must be an IPv4/IPv6 address or CIDR (no host names); anything else is refused at render, naming the entry. Set this whenever ingress.enabled=true — see Upgrade for what goes wrong if you do not. |
edge.preflightTrustedProxies |
true |
Refuse to render when edge.enabled=true and ingress.enabled=true and edge.trustedProxies is empty, instead of silently shipping a broken rate limit and audit log. Set false only when the ingress is genuinely not in the request path. Requests through an untrusted proxy still get a 503 at runtime either way. |
edge.realIpHeader |
X-Forwarded-For |
Header the trusted hop puts the client address in. Suits ingress-nginx/Traefik/HAProxy; some setups use X-Real-IP, and Cloudflare uses CF-Connecting-IP (with a Cloudflare Tunnel, trust the cloudflared pods, not Cloudflare's published ranges; see Edge trusted proxies). Only emitted while edge.trustedProxies is non-empty — with nothing trusted there is no header to believe. |
edge.realIpRecursive |
true |
Walk the header right-to-left past every trusted address and take the first untrusted one as the client. What stops a client-prepended fake entry from being believed; leave on for a chain of proxies. Only emitted while edge.trustedProxies is non-empty. |
edge.podLabels / edge.podAnnotations |
{} / {} |
Extra labels and annotations for the edge pod, e.g. to keep it out of a service mesh (see the mesh paragraph). Values are strings or bools; the chart's own pod labels and checksum/edge-conf cannot be overridden. |
edge.loginRateLimit.rate / .burst |
10r/m / 5 |
/api/auth/login budget, applied per resolved client address — so it is only per-client once edge.trustedProxies is right. |
ingress.enabled |
false |
Needs an ingress controller and edge.enabled=true (the Ingress routes to the edge Service; the render is refused without it). Use TLS — login and agent WebSockets are otherwise cleartext. |
ingress.className |
"" |
|
ingress.host |
"" |
e.g. fleeger.example.com. |
ingress.annotations |
{} |
e.g. cert-manager cluster-issuer. |
ingress.tls |
[] |
TLS hosts + secretName. |
loki.url |
"" |
A Loki you run yourself, for pod-log ingestion, dead-log purging and the log tab; empty turns them off. The release namespace's pod logs (the hub's own, collected with coreAgent.podLogs: all) are read only by a holder of hub_logs.view: ingestion-svc stamps them hub="true" (see logs.hubNamespaces). |
loki.maxPushBytes |
"" |
The most one push to Loki may come to as the protobuf Loki's distributor hands its ingester over gRPC, by a conservative estimate: ingestion-svc splits a larger log message into several pushes. Keep it under your Loki's server.grpc_server_max_recv_msg_size (4 MiB by default). Empty ⇒ 3145728 (3 MiB). From 1048576 to 268435456; the render refuses anything else. |
logs.hubNamespaces |
[] |
More namespaces whose pods are the hub's own, added to the release namespace (always one) in ingestion-svc's [logs] hub_namespaces, each once. Their pods' log streams, as the core agent ships them for a local_collector location, are stamped hub="true", which only a holder of hub_logs.view reads. Add any namespace that runs hub components next to the release (a Loki, an ingress controller, an operator you installed for the hub) whose logs a viewer should not read. Streams written before the upgrade that added this setting, or before fix-perms core-agent narrowed the core agent's credential, are not stamped and stay visible until Loki's retention ages them out; a hub cluster collected by a regular spoke agent instead of the core agent is never stamped. Each entry must be a namespace name; with loki.url only. |
networkPolicy.enabled |
false |
Default-deny + per-service allow rules. CNI-specific; validate before enabling. |
networkPolicy.probeCIDRs |
[] |
Node CIDR(s) to allow kubelet health probes through. |
Scheduling
| Key | Default | Description |
|---|---|---|
resources.services.requests |
50m / 64Mi |
Per-service requests. |
resources.services.limits.memory |
256Mi |
|
nodeSelector / tolerations / affinity |
{} / [] / {} |
Full walkthrough
See Deploying Fleeger on Kubernetes for the complete deployment guide, including generating the NATS auth Secret, TLS/ingress, backups, and NetworkPolicy validation.