// Betrieb
Observability & self-monitoring
Fleeger monitors the infrastructure it manages and itself. This document covers the self-monitoring path — how the control plane's own services expose metrics and logs — and how it relates to the infra-metrics pipeline.
Two metric sources, one store
Everything lands in the same VictoriaMetrics instance, distinguished by a
source label:
| Source | source |
How it arrives | Example series |
|---|---|---|---|
| Managed infra | proxmox, idrac, … |
spoke-agent → NATS → ingestion-svc → VM | temperature_celsius{location="homelab",source="idrac"} |
| The platform itself | fleeger |
VictoriaMetrics scrapes each service's /metrics |
fleeger_uptime_seconds{location="meta",source="fleeger",service="auth-svc"} |
The control plane registers as a synthetic meta Location (record
location:meta, kind meta) so it appears in its own dashboard next to
homelab / hetzner.
Its name and kind are fixed and it cannot be deleted or its agent offboarded
(which would purge the hub's own state and metrics), since nothing
recreates it; its region, network config and collector mode stay editable.
Series bound
An agent chooses its resource ids, point names and labels, so the managed-infra
source has no fixed cardinality. ingestion-svc admits at most
[ingestion] max_new_series_per_location_hour (50,000 by default; Helm
ingestion.maxNewSeriesPerLocationHour) new series per location per hour, by
the hour each message was published to NATS; past it the points of further new
series are dropped while the location's existing series keep flowing, one WARN
per location and hour is logged, and ingestion-svc's storage sampler writes
fleeger_ingestion_new_series_dropped_total{location="meta",source="fleeger",service="ingestion-svc",pod,series_location},
which the built-in series_bound_exceeded alert reads. The sampler is also what
writes this counter: while it fails (NATS unreachable, which
nats_sampler_stale reports), drops are not written and the alert stays
silent.
Each replica and each restart keeps sets of its own, so VictoriaMetrics' own
-storage.maxHourlySeries (1,000,000) and -storage.maxDailySeries
(5,000,000) are the hub-wide backstop: chart victoriametrics.maxHourlySeries
/ maxDailySeries, compose FLEEGER_VM_MAX_HOURLY_SERIES /
FLEEGER_VM_MAX_DAILY_SERIES, deploy/k8s the same flags in
base/30-dependencies.yaml; 0 disables either. They count the distinct
series written in an hour or a day of VictoriaMetrics' uptime, existing series
included: once one is reached, every series not yet written in that hour or day
loses its samples, for every source alike, the hub's own series included. Size
each above your hub-wide active series plus their churn in that hour (or day);
the defaults suit a hub up to several hundred thousand active series, and a hub
near them raises them (roughly 1 KiB of VictoriaMetrics memory per active
series).
VictoriaMetrics stores its own metrics (-selfScrapeInterval=30s, labelled
job="victoria-metrics", instance="self", with no location), among them
vm_hourly_series_limit_rows_dropped_total and
vm_daily_series_limit_rows_dropped_total, which it exposes only while that
limit is set. The built-in vm_series_limit_reached alert (warning, migration
0022) fires, hub-wide, on their 15-minute increase above 0. Its own series
count against the limits like any other. Written every 30s, they are known
before a flood within a running hour; but the count restarts with
VictoriaMetrics, and on a hub whose series reach a limit before its first
self-scrape after a restart, its own series, these counters too, are dropped
and the alert is blind until the next hour of its uptime. Its log still says
which limit dropped what (skip series … because -storage.maxHourlySeries=… reached).
Service /metrics endpoints
Every hub service (auth-svc, inventory-svc, gateway-svc, ingestion-svc,
alerting-svc) exposes Prometheus text at /metrics, provided by
fleeger_shared::metrics:
fleeger_build_info{service,version}— gauge, always1fleeger_uptime_seconds{service}— seconds since process startfleeger_process_resident_memory_bytes{service}— RSS (Linux/proc/self)fleeger_process_cpu_seconds_total{service}— cumulative CPUfleeger_http_requests_total{service,method,status}— request counter
The counter is fed by the track_metrics middleware, which each service wraps
around its business router. The registry is a small process-global set of
atomics — no prometheus/metrics crate dependency. Path is deliberately not
a label (bounded cardinality).
Scraping
deploy/victoriametrics/scrape.yml
drives VictoriaMetrics' built-in scraper (--promscrape.config). In local dev
the services run on the host, so targets are host.docker.internal:<port>
(compose maps this via extra_hosts: host-gateway). Every scraped series is
labelled location=meta, source=fleeger. In production these become in-cluster
Service names in the VictoriaMetrics Helm values.
Verify:
curl -s 'http://localhost:8428/api/v1/targets' | jq '.data.activeTargets[] | {instance: .labels.instance, health}'
curl -s 'http://localhost:8428/api/v1/query?query=fleeger_uptime_seconds' | jq '.data.result[].metric.service'Logs — Loki + Promtail
Hub services log plain text to stdout/stderr (the tracing fmt layer). Loki
aggregates; Promtail ships:
- Production compose —
deploy/compose/loki+deploy/compose/promtail. Promtail discovers thefleeger-*containers over the Docker socket and stamps each line with a cleanservicelabel (e.g.fleeger-auth-svc-1→service=auth-svc), pluscontainer,stream,location=meta,source=fleeger. - Dev deps — the root
docker-compose.ymlruns a minimal Loki+Promtail (deploy/loki,deploy/promtail) that also tails/var/log/fleeger/*.logfor host-run services.
Loki has no auth of its own and is never published to a host port. The
gateway is the only way in: it proxies authenticated LogQL at
/api/logs/query[_range] (bearer token + resource.view, and the same
query-cost caps as the VictoriaMetrics proxy), and the UI's Services page
shows a per-service log tail built on it to a holder of hub_logs.view. A
user who holds resource.view only at some locations gets their query pinned
to those locations' streams (see RBAC). The hub's own logs need
hub_logs.view on top, held globally (admin has it through *, the
built-in operator and viewer do not): the hub services' location="meta"
streams, and, on Kubernetes, the streams of the hub's pods the core agent
collects under a local_collector location, which ingestion-svc stamps
hub="true" when the core agent sent them on its logs._core.<location>
subject, ingestion-svc admits that location as local_collector, and
their namespace is one of the hub's ([logs] hub_namespaces of
ingestion-svc, which the chart renders as the release namespace plus
logs.hubNamespaces, and FLEEGER_HUB_NAMESPACE, which the raw manifests set
to ingestion-svc's own). A location's own agent's streams are never stamped:
a hub cluster collected by a regular spoke agent instead of the core agent,
or by the core agent before ./gen-auth.sh fix-perms core-agent narrowed its
credential, has its pod streams visible to every resource.view holder there.
hub is reserved: an agent's own is dropped. gateway-svc leaves out
location="meta" and hub="true" for everyone else, so third-party hub
containers that log secrets (VM's authKey in a failed delete's WARN)
are not read by a viewer; a spoke's pod in a namespace of the same name stays
visible. Hub pod streams written before the upgrade carry no hub label and
stay visible until Loki's retention ages them out (no backfill). A log
shipper that writes the hub's pod logs into Loki itself, bypassing
ingestion-svc (a Promtail DaemonSet, say), must label them itself,
location="meta" or hub="true": ingestion-svc stamps only what passes
through it, and an unlabelled hub pod stream is visible to every
resource.view holder. Still,
the services never log
a credential, a URL's userinfo or query, or a value quoted from stored data,
and log a URL's path only for the URLs our own code builds from a configured
base and fixed paths; see AppError::Internal in fleeger-shared and
fleeger_redact (re-exported as fleeger_shared::redact). Query Loki directly in dev:
curl -s 'http://localhost:3100/loki/api/v1/label/service/values'
curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
--data-urlencode 'query={service="auth-svc"}' --data-urlencode 'limit=20'Monitored-pod logs
Beyond the hub's own service logs, the Kubernetes collector can ship monitored
workload logs to the same Loki. It's opt-in and bounded (off by default):
set FLEEGER_K8S_POD_LOGS on the agent to annotated (only pods carrying the
fleeger.io/logs: "true" annotation — set it on the workload's pod template so it
propagates) or all (whole location). For each selected pod the agent ships each
container's current logs, plus the previous (crashed) container's logs when it
has restarted — the "why did it die?" signal — capped by tail_lines and a lookback
window. Lines carry real container timestamps so Loki dedups overlapping fetches.
Transport mirrors metrics: the agent publishes a LogMessage on its own
logs.<location>.> JetStream subject (a dedicated broker ACL, per-location
confined exactly like metrics), and ingestion-svc's log consumer forwards the
lines to Loki's push API, labelled location/resource_id (pod-<ns>-<name>)/
cluster (the pod's cluster)/namespace/pod_name/container
(+ previous). The pod/resource detail view in
the UI shows a Logs panel (current + previous-container toggle) over the same
gateway Loki proxy. How long a dead pod's logs are kept is governed by the
retention policy.
When Loki refuses a push. A batch Loki answers with a 429 (over its
ingestion rate), a 408 or a 5xx, or does not answer, stays in the logs stream
and is pushed again after the consumer's 30 s ack wait, so the stream's outage
buffer survives the replay when Loki comes back. After each such failure the
consumer pauses before its next push, from 0.5 s doubling to 15 s while
failures continue in a row, and back to full speed at the first success, so a
backlog replays at about the rate Loki accepts. Any other 4xx (for example
entries older than reject_old_samples_max_age) is the batch's own fault: it
is logged as dropping log batch Loki rejected and acked. The shipped Loki
configs (deploy/compose/loki, deploy/loki) therefore accept a whole log
message of up to the stream's 8 MiB at once: ingestion_rate_mb: 8,
ingestion_burst_size_mb: 16 and 16 MiB gRPC messages, where Loki's defaults
(4 MB/s, a 6 MB burst, 4 MiB gRPC) refuse a push of over 6 MB with a 429, and
one of over 4 MiB with a 500, every time. A Loki you run yourself (Helm,
loki.url) needs the same rate and burst: a push over its burst gets the
tenant's 429 every time, which never counts below, so the batch is redelivered
every ack wait until the stream ages it out, holding up the logs behind it; a
push over its gRPC limit gets a 500 and is dropped as below. Agents keep each
message under 4 MiB, so this concerns mainly an unusually large batch.
A batch Loki refuses for good is dropped. A batch Loki fails every
time while it takes others (a 500 it answers to that batch alone) used to come
back every ack wait until the stream aged it out (stream_max_age_secs, 3
days), each time with an error line and a pause for every log behind it, and
nothing alerted. ingestion-svc now counts, per batch, the refusals that may be
the batch's own — a 500 whose body names none of the causes below, or a
5xx other than 502, 503, 504 and 520 to 524 — and of those only the
interleaved ones: the batch's first, and each one after which Loki accepted at least one other
push since the batch's last counted refusal. At 10 counted refusals the batch
is dropped (acked). JetStream's delivery count plays no part. So nothing is
ever dropped during a Loki outage, however long it lasts (no push succeeds, so
the count stays at 1), nor for buffered copies of the same batch pushed with
no accepted push in between. These
never count at all, so a busy or recovering Loki does not cost valid data: any
429 (the tenant's Ingestion rate limit exceeded, and the per-stream Per stream rate limit exceeded, which one hot stream gets while a long outage's
backlog replays and other pushes pass); a 502, 503 or 504 (a gateway in front
of Loki, or Loki overloaded); a 520 to 524 (a proxy such as Cloudflare in
front of Loki that got no valid answer, could not reach it, or timed out); a
timeout or an unreachable Loki; and a 500 whose body names a cause Loki
answers every push with while a part of it is down, matched
case-insensitively: too many unhealthy instances, empty ring, live replicas required (whatever number it names, so a partial outage's at least 2 live replicas required, could only find 1 too), ingester is shutting down, code = Unavailable (gRPC's word for an ingester that is not there),
context deadline exceeded, i/o timeout, transport is closing,
connection refused, no such host. Every failure, counted or not,
still leaves the batch unacked and pauses the consumer as before. Each drop is
one ERROR line, dropping log batch Loki refused 10 times while it accepted other batches in between, with the message's subject, its stream sequence and
Loki's last answer, and counts in
fleeger_ingestion_log_batches_dropped_total{reason="loki_refused"}, which
the storage sampler writes to VictoriaMetrics every 30 s (labelled
location="meta", source="fleeger", service="ingestion-svc" and pod,
the replica's pod name or else a random id per process, from 0 after each
start; a restarted container keeps its pod name, and increase reads the
return to 0 as a counter reset). The built-in log_batches_dropped alert
fires on it, summed over pod (see
Alerting).
The counts are in memory, per consumer run, keyed by stream sequence: a batch
is forgotten once acked, once it has not failed for ten ack waits, and, past
10,000 batches, the one seen least recently first. A restart or a NATS
reconnect starts every count over, so a drop only ever comes later, never
sooner. Redelivered every ack wait with others going through, a refused batch
is dropped nine ack waits after its first refusal at the soonest (4.5 minutes
with the shipped 30 s), later while few other batches get through, and sooner
when buffered copies of it are interleaved with accepted pushes.
Not covered: a distributed Loki with one bad ingester that answers 5xx for the
streams it owns, while the others take theirs, with a body that names none of
the causes above, looks the same as a batch Loki refuses for good: those
streams' batches are dropped after their ten interleaved refusals, and the
alert fires. Fix the ingester; the dropped lines are not retried. The same
holds for an overloaded Loki that answers a bare 500 internal server error
to some pushes at random and accepts the rest: a valid batch that happens to
be refused ten times before one of its pushes passes is dropped (at 50 %
random failures about 1 batch in 1,000), and the alert fires.
Large messages are split. Loki hands a push to its ingester as a
protobuf over gRPC, and that message is what its 4 MiB default limit applies
to, not the JSON ingestion-svc sends. Loki adds a detected_level to every
entry and re-escapes label values, so a push of short lines reaches 4 MiB of
protobuf at about 2.6 MB of JSON; and it copies a stream's level, lvl or
severity label into every entry, so a long one counts once per entry.
ingestion-svc therefore splits a log message whose estimated protobuf size is
over [loki] max_push_bytes (Helm: loki.maxPushBytes; default 3 MiB, from
1 MiB to 256 MiB) into several pushes: between streams first, then between a
stream's entries, which keep their order. The estimate takes every field at
its longest, every label value at its most escaped and the level label once
per entry, so a part under the bound is under it in Loki too; the default
leaves a quarter of Loki's limit for what the estimate cannot see (a stream
Loki shards repeats its labels per shard). Keep the setting under your Loki's
server.grpc_server_max_recv_msg_size.
The message is acked only once every part is stored, and before each further
part ingestion-svc tells JetStream it is still working on it, so a slow push
is not redelivered midway. A part Loki fails as transient leaves the message
for redelivery, which pushes every part again: Loki drops the entries its
ingester still holds, and one it has already flushed is stored twice but
deduplicated at query time. A part Loki refuses with a 4xx does not keep the
other parts from it, and the message is then dropped as before. A single line
too large for a part goes alone: Loki refuses it with a 400 if it is over its
max_line_size (256 KB by default), and otherwise, over the gRPC limit, with
a 500 that is retried until the stream ages the message out.
The spoke-agent's own process logs are still not shipped to the hub — those
stay on the remote host (docker logs / kubectl logs of the agent itself). Only
the monitored pods' logs travel, on the dedicated logs.* subject above.
Enabling it. The agent reads FLEEGER_K8S_POD_LOGS (off default / annotated
/ all). Two prerequisites, or nothing reaches Loki:
- The agent's NATS credential must allow
pub logs.<location>.>. Remote spokes minted by auth-svc get this automatically; the bundled core-agent in the compose/Helm stack uses a pre-provisioned credential, so reissue it after upgrading (a credential minted before this feature lacks the grant and the broker silently denies the publish):deploy/nats/gen-auth.sh fix-perms core-agent, then restart only the core agent (docker compose --profile core-agent up -d --force-recreate --no-deps core-agent). It keeps the operator, the account and every other credential, so NATS needs no restart, nopush-accounts.sh, and no site re-onboards. Do not use a baregen-auth.shfor this: it rotates the whole trust chain. See Agent credentials & revocation. - ingestion-svc must have a
[loki]URL. The compose stack ships one; the Helm chart and the raw manifests do not deploy Loki, so there the URL has to be supplied (loki.urlin values.yaml; the commented[loki]/loki_urlpairs indeploy/k8s/base/10-config.yaml). Without it the config field staysNone, the log consumer never starts, the gateway 404s/api/logs/query, and ingestion-svc removes thelogsJetStream stream and itsingestion-logsconsumer (it checks every 30s and logs how many messages it dropped; those are not counted infleeger_nats_stream_discarded_unforwarded_messages_total), so an unread stream never fills and alerts. It waits while the consumer is still active (another ingestion-svc with Loki) and leaves a stream it did not shape alone. Agents' log publishes are then answered "no stream" and dropped; each agent logs that once, until the stream is back. Configuring Loki again and restarting ingestion-svc recreates both.
Per install path:
deploy/compose— setFLEEGER_K8S_POD_LOGS=all(orannotated) in.env, regenerate the NATS creds as above, anddocker compose --profile core-agent up -d core-agent.- Helm (
charts/fleeger-hub) —--set coreAgent.enabled=true --set coreAgent.podLogs=all --set loki.url=http://…:3100(on a running hub, withhelm upgrade … --reset-then-reuse-values, or every other value you set is reset — see the Helm chart reference's Upgrade).coreAgent.podLogsis what setsFLEEGER_K8S_POD_LOGSon the bundled agent; leaving it at itsoffdefault collects nothing even withloki.urlset. Withallit collects the hub's own pods too; their streams are the hub's logs, which only a holder ofhub_logs.viewreads. - Raw manifests (
deploy/k8s) — editFLEEGER_K8S_POD_LOGSinoptional/cluster-monitoring/core-agent.yamland uncomment the[loki]/loki_urlentries inbase/10-config.yaml.
Live agent connections — /connz
The NATS server's monitoring endpoint (:8222, internal-only) exposes each live
client connection. The gateway proxies it at /api/agents/connz (bearer token +
location.view); agents name their NATS connection agent-<location>, so the
UI correlates a connection to a location and shows its RTT, uptime, throughput,
and subscription count on the Services → Agents tab. Because agents tunnel
NATS over wss through the single public edge, the peer IP the broker sees is the
edge, not the agent's own address.
Dashboard & the Services page
The dashboard's Hub services (meta) panel queries VictoriaMetrics for the
{location="meta"} self-metrics — per-service version, uptime, memory, and total
requests — merged by the service label.
The Services page (nav: Services) goes deeper: a compact, expandable row
per service with its endpoint (instance label), version, uptime, memory, CPU,
and requests-by-status, plus the inline log tail above; a second Agents tab
carries the connection detail from /connz.
Notes & limitations
- The
metaLocation does not report a spoke-agent heartbeat (it isn't polled over NATS); its liveness comes from scrape target health, not theingestion-svcreachability state used forhomelab/hetzner. page_size(4096) and_SC_CLK_TCK(100) are hard-coded in the metrics module to avoid a libc dependency — correct on all common Linux hosts; these two process gauges would be slightly off on an exotic kernel config.