Zum Inhalt springen
Produkte

// Betrieb

Observability & self-monitoring

Fleeger monitors the infrastructure it manages and itself. This document covers the self-monitoring path — how the control plane's own services expose metrics and logs — and how it relates to the infra-metrics pipeline.

Two metric sources, one store

Everything lands in the same VictoriaMetrics instance, distinguished by a source label:

Source source How it arrives Example series
Managed infra proxmox, idrac, … spoke-agent → NATS → ingestion-svc → VM temperature_celsius{location="homelab",source="idrac"}
The platform itself fleeger VictoriaMetrics scrapes each service's /metrics fleeger_uptime_seconds{location="meta",source="fleeger",service="auth-svc"}

The control plane registers as a synthetic meta Location (record location:meta, kind meta) so it appears in its own dashboard next to homelab / hetzner. Its name and kind are fixed and it cannot be deleted or its agent offboarded (which would purge the hub's own state and metrics), since nothing recreates it; its region, network config and collector mode stay editable.

Series bound

An agent chooses its resource ids, point names and labels, so the managed-infra source has no fixed cardinality. ingestion-svc admits at most [ingestion] max_new_series_per_location_hour (50,000 by default; Helm ingestion.maxNewSeriesPerLocationHour) new series per location per hour, by the hour each message was published to NATS; past it the points of further new series are dropped while the location's existing series keep flowing, one WARN per location and hour is logged, and ingestion-svc's storage sampler writes fleeger_ingestion_new_series_dropped_total{location="meta",source="fleeger",service="ingestion-svc",pod,series_location}, which the built-in series_bound_exceeded alert reads. The sampler is also what writes this counter: while it fails (NATS unreachable, which nats_sampler_stale reports), drops are not written and the alert stays silent.

Each replica and each restart keeps sets of its own, so VictoriaMetrics' own -storage.maxHourlySeries (1,000,000) and -storage.maxDailySeries (5,000,000) are the hub-wide backstop: chart victoriametrics.maxHourlySeries / maxDailySeries, compose FLEEGER_VM_MAX_HOURLY_SERIES / FLEEGER_VM_MAX_DAILY_SERIES, deploy/k8s the same flags in base/30-dependencies.yaml; 0 disables either. They count the distinct series written in an hour or a day of VictoriaMetrics' uptime, existing series included: once one is reached, every series not yet written in that hour or day loses its samples, for every source alike, the hub's own series included. Size each above your hub-wide active series plus their churn in that hour (or day); the defaults suit a hub up to several hundred thousand active series, and a hub near them raises them (roughly 1 KiB of VictoriaMetrics memory per active series).

VictoriaMetrics stores its own metrics (-selfScrapeInterval=30s, labelled job="victoria-metrics", instance="self", with no location), among them vm_hourly_series_limit_rows_dropped_total and vm_daily_series_limit_rows_dropped_total, which it exposes only while that limit is set. The built-in vm_series_limit_reached alert (warning, migration 0022) fires, hub-wide, on their 15-minute increase above 0. Its own series count against the limits like any other. Written every 30s, they are known before a flood within a running hour; but the count restarts with VictoriaMetrics, and on a hub whose series reach a limit before its first self-scrape after a restart, its own series, these counters too, are dropped and the alert is blind until the next hour of its uptime. Its log still says which limit dropped what (skip series … because -storage.maxHourlySeries=… reached).

Service /metrics endpoints

Every hub service (auth-svc, inventory-svc, gateway-svc, ingestion-svc, alerting-svc) exposes Prometheus text at /metrics, provided by fleeger_shared::metrics:

  • fleeger_build_info{service,version} — gauge, always 1
  • fleeger_uptime_seconds{service} — seconds since process start
  • fleeger_process_resident_memory_bytes{service} — RSS (Linux /proc/self)
  • fleeger_process_cpu_seconds_total{service} — cumulative CPU
  • fleeger_http_requests_total{service,method,status} — request counter

The counter is fed by the track_metrics middleware, which each service wraps around its business router. The registry is a small process-global set of atomics — no prometheus/metrics crate dependency. Path is deliberately not a label (bounded cardinality).

Scraping

deploy/victoriametrics/scrape.yml drives VictoriaMetrics' built-in scraper (--promscrape.config). In local dev the services run on the host, so targets are host.docker.internal:<port> (compose maps this via extra_hosts: host-gateway). Every scraped series is labelled location=meta, source=fleeger. In production these become in-cluster Service names in the VictoriaMetrics Helm values.

Verify:

curl -s 'http://localhost:8428/api/v1/targets' | jq '.data.activeTargets[] | {instance: .labels.instance, health}'
curl -s 'http://localhost:8428/api/v1/query?query=fleeger_uptime_seconds' | jq '.data.result[].metric.service'

Logs — Loki + Promtail

Hub services log plain text to stdout/stderr (the tracing fmt layer). Loki aggregates; Promtail ships:

  • Production compose — deploy/compose/loki + deploy/compose/promtail. Promtail discovers the fleeger-* containers over the Docker socket and stamps each line with a clean service label (e.g. fleeger-auth-svc-1 → service=auth-svc), plus container, stream, location=meta, source=fleeger.
  • Dev deps — the root docker-compose.yml runs a minimal Loki+Promtail (deploy/loki, deploy/promtail) that also tails /var/log/fleeger/*.log for host-run services.

Loki has no auth of its own and is never published to a host port. The gateway is the only way in: it proxies authenticated LogQL at /api/logs/query[_range] (bearer token + resource.view, and the same query-cost caps as the VictoriaMetrics proxy), and the UI's Services page shows a per-service log tail built on it to a holder of hub_logs.view. A user who holds resource.view only at some locations gets their query pinned to those locations' streams (see RBAC). The hub's own logs need hub_logs.view on top, held globally (admin has it through *, the built-in operator and viewer do not): the hub services' location="meta" streams, and, on Kubernetes, the streams of the hub's pods the core agent collects under a local_collector location, which ingestion-svc stamps hub="true" when the core agent sent them on its logs._core.<location> subject, ingestion-svc admits that location as local_collector, and their namespace is one of the hub's ([logs] hub_namespaces of ingestion-svc, which the chart renders as the release namespace plus logs.hubNamespaces, and FLEEGER_HUB_NAMESPACE, which the raw manifests set to ingestion-svc's own). A location's own agent's streams are never stamped: a hub cluster collected by a regular spoke agent instead of the core agent, or by the core agent before ./gen-auth.sh fix-perms core-agent narrowed its credential, has its pod streams visible to every resource.view holder there. hub is reserved: an agent's own is dropped. gateway-svc leaves out location="meta" and hub="true" for everyone else, so third-party hub containers that log secrets (VM's authKey in a failed delete's WARN) are not read by a viewer; a spoke's pod in a namespace of the same name stays visible. Hub pod streams written before the upgrade carry no hub label and stay visible until Loki's retention ages them out (no backfill). A log shipper that writes the hub's pod logs into Loki itself, bypassing ingestion-svc (a Promtail DaemonSet, say), must label them itself, location="meta" or hub="true": ingestion-svc stamps only what passes through it, and an unlabelled hub pod stream is visible to every resource.view holder. Still, the services never log a credential, a URL's userinfo or query, or a value quoted from stored data, and log a URL's path only for the URLs our own code builds from a configured base and fixed paths; see AppError::Internal in fleeger-shared and fleeger_redact (re-exported as fleeger_shared::redact). Query Loki directly in dev:

curl -s 'http://localhost:3100/loki/api/v1/label/service/values'
curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service="auth-svc"}' --data-urlencode 'limit=20'

Monitored-pod logs

Beyond the hub's own service logs, the Kubernetes collector can ship monitored workload logs to the same Loki. It's opt-in and bounded (off by default): set FLEEGER_K8S_POD_LOGS on the agent to annotated (only pods carrying the fleeger.io/logs: "true" annotation — set it on the workload's pod template so it propagates) or all (whole location). For each selected pod the agent ships each container's current logs, plus the previous (crashed) container's logs when it has restarted — the "why did it die?" signal — capped by tail_lines and a lookback window. Lines carry real container timestamps so Loki dedups overlapping fetches.

Transport mirrors metrics: the agent publishes a LogMessage on its own logs.<location>.> JetStream subject (a dedicated broker ACL, per-location confined exactly like metrics), and ingestion-svc's log consumer forwards the lines to Loki's push API, labelled location/resource_id (pod-<ns>-<name>)/ cluster (the pod's cluster)/namespace/pod_name/container (+ previous). The pod/resource detail view in the UI shows a Logs panel (current + previous-container toggle) over the same gateway Loki proxy. How long a dead pod's logs are kept is governed by the retention policy.

When Loki refuses a push. A batch Loki answers with a 429 (over its ingestion rate), a 408 or a 5xx, or does not answer, stays in the logs stream and is pushed again after the consumer's 30 s ack wait, so the stream's outage buffer survives the replay when Loki comes back. After each such failure the consumer pauses before its next push, from 0.5 s doubling to 15 s while failures continue in a row, and back to full speed at the first success, so a backlog replays at about the rate Loki accepts. Any other 4xx (for example entries older than reject_old_samples_max_age) is the batch's own fault: it is logged as dropping log batch Loki rejected and acked. The shipped Loki configs (deploy/compose/loki, deploy/loki) therefore accept a whole log message of up to the stream's 8 MiB at once: ingestion_rate_mb: 8, ingestion_burst_size_mb: 16 and 16 MiB gRPC messages, where Loki's defaults (4 MB/s, a 6 MB burst, 4 MiB gRPC) refuse a push of over 6 MB with a 429, and one of over 4 MiB with a 500, every time. A Loki you run yourself (Helm, loki.url) needs the same rate and burst: a push over its burst gets the tenant's 429 every time, which never counts below, so the batch is redelivered every ack wait until the stream ages it out, holding up the logs behind it; a push over its gRPC limit gets a 500 and is dropped as below. Agents keep each message under 4 MiB, so this concerns mainly an unusually large batch.

A batch Loki refuses for good is dropped. A batch Loki fails every time while it takes others (a 500 it answers to that batch alone) used to come back every ack wait until the stream aged it out (stream_max_age_secs, 3 days), each time with an error line and a pause for every log behind it, and nothing alerted. ingestion-svc now counts, per batch, the refusals that may be the batch's own — a 500 whose body names none of the causes below, or a 5xx other than 502, 503, 504 and 520 to 524 — and of those only the interleaved ones: the batch's first, and each one after which Loki accepted at least one other push since the batch's last counted refusal. At 10 counted refusals the batch is dropped (acked). JetStream's delivery count plays no part. So nothing is ever dropped during a Loki outage, however long it lasts (no push succeeds, so the count stays at 1), nor for buffered copies of the same batch pushed with no accepted push in between. These never count at all, so a busy or recovering Loki does not cost valid data: any 429 (the tenant's Ingestion rate limit exceeded, and the per-stream Per stream rate limit exceeded, which one hot stream gets while a long outage's backlog replays and other pushes pass); a 502, 503 or 504 (a gateway in front of Loki, or Loki overloaded); a 520 to 524 (a proxy such as Cloudflare in front of Loki that got no valid answer, could not reach it, or timed out); a timeout or an unreachable Loki; and a 500 whose body names a cause Loki answers every push with while a part of it is down, matched case-insensitively: too many unhealthy instances, empty ring, live replicas required (whatever number it names, so a partial outage's at least 2 live replicas required, could only find 1 too), ingester is shutting down, code = Unavailable (gRPC's word for an ingester that is not there), context deadline exceeded, i/o timeout, transport is closing, connection refused, no such host. Every failure, counted or not, still leaves the batch unacked and pauses the consumer as before. Each drop is one ERROR line, dropping log batch Loki refused 10 times while it accepted other batches in between, with the message's subject, its stream sequence and Loki's last answer, and counts in fleeger_ingestion_log_batches_dropped_total{reason="loki_refused"}, which the storage sampler writes to VictoriaMetrics every 30 s (labelled location="meta", source="fleeger", service="ingestion-svc" and pod, the replica's pod name or else a random id per process, from 0 after each start; a restarted container keeps its pod name, and increase reads the return to 0 as a counter reset). The built-in log_batches_dropped alert fires on it, summed over pod (see Alerting). The counts are in memory, per consumer run, keyed by stream sequence: a batch is forgotten once acked, once it has not failed for ten ack waits, and, past 10,000 batches, the one seen least recently first. A restart or a NATS reconnect starts every count over, so a drop only ever comes later, never sooner. Redelivered every ack wait with others going through, a refused batch is dropped nine ack waits after its first refusal at the soonest (4.5 minutes with the shipped 30 s), later while few other batches get through, and sooner when buffered copies of it are interleaved with accepted pushes.

Not covered: a distributed Loki with one bad ingester that answers 5xx for the streams it owns, while the others take theirs, with a body that names none of the causes above, looks the same as a batch Loki refuses for good: those streams' batches are dropped after their ten interleaved refusals, and the alert fires. Fix the ingester; the dropped lines are not retried. The same holds for an overloaded Loki that answers a bare 500 internal server error to some pushes at random and accepts the rest: a valid batch that happens to be refused ten times before one of its pushes passes is dropped (at 50 % random failures about 1 batch in 1,000), and the alert fires.

Large messages are split. Loki hands a push to its ingester as a protobuf over gRPC, and that message is what its 4 MiB default limit applies to, not the JSON ingestion-svc sends. Loki adds a detected_level to every entry and re-escapes label values, so a push of short lines reaches 4 MiB of protobuf at about 2.6 MB of JSON; and it copies a stream's level, lvl or severity label into every entry, so a long one counts once per entry. ingestion-svc therefore splits a log message whose estimated protobuf size is over [loki] max_push_bytes (Helm: loki.maxPushBytes; default 3 MiB, from 1 MiB to 256 MiB) into several pushes: between streams first, then between a stream's entries, which keep their order. The estimate takes every field at its longest, every label value at its most escaped and the level label once per entry, so a part under the bound is under it in Loki too; the default leaves a quarter of Loki's limit for what the estimate cannot see (a stream Loki shards repeats its labels per shard). Keep the setting under your Loki's server.grpc_server_max_recv_msg_size.

The message is acked only once every part is stored, and before each further part ingestion-svc tells JetStream it is still working on it, so a slow push is not redelivered midway. A part Loki fails as transient leaves the message for redelivery, which pushes every part again: Loki drops the entries its ingester still holds, and one it has already flushed is stored twice but deduplicated at query time. A part Loki refuses with a 4xx does not keep the other parts from it, and the message is then dropped as before. A single line too large for a part goes alone: Loki refuses it with a 400 if it is over its max_line_size (256 KB by default), and otherwise, over the gRPC limit, with a 500 that is retried until the stream ages the message out.

The spoke-agent's own process logs are still not shipped to the hub — those stay on the remote host (docker logs / kubectl logs of the agent itself). Only the monitored pods' logs travel, on the dedicated logs.* subject above.

Enabling it. The agent reads FLEEGER_K8S_POD_LOGS (off default / annotated / all). Two prerequisites, or nothing reaches Loki:

  1. The agent's NATS credential must allow pub logs.<location>.>. Remote spokes minted by auth-svc get this automatically; the bundled core-agent in the compose/Helm stack uses a pre-provisioned credential, so reissue it after upgrading (a credential minted before this feature lacks the grant and the broker silently denies the publish): deploy/nats/gen-auth.sh fix-perms core-agent, then restart only the core agent (docker compose --profile core-agent up -d --force-recreate --no-deps core-agent). It keeps the operator, the account and every other credential, so NATS needs no restart, no push-accounts.sh, and no site re-onboards. Do not use a bare gen-auth.sh for this: it rotates the whole trust chain. See Agent credentials & revocation.
  2. ingestion-svc must have a [loki] URL. The compose stack ships one; the Helm chart and the raw manifests do not deploy Loki, so there the URL has to be supplied (loki.url in values.yaml; the commented [loki] / loki_url pairs in deploy/k8s/base/10-config.yaml). Without it the config field stays None, the log consumer never starts, the gateway 404s /api/logs/query, and ingestion-svc removes the logs JetStream stream and its ingestion-logs consumer (it checks every 30s and logs how many messages it dropped; those are not counted in fleeger_nats_stream_discarded_unforwarded_messages_total), so an unread stream never fills and alerts. It waits while the consumer is still active (another ingestion-svc with Loki) and leaves a stream it did not shape alone. Agents' log publishes are then answered "no stream" and dropped; each agent logs that once, until the stream is back. Configuring Loki again and restarting ingestion-svc recreates both.

Per install path:

  • deploy/compose — set FLEEGER_K8S_POD_LOGS=all (or annotated) in .env, regenerate the NATS creds as above, and docker compose --profile core-agent up -d core-agent.
  • Helm (charts/fleeger-hub) — --set coreAgent.enabled=true --set coreAgent.podLogs=all --set loki.url=http://…:3100 (on a running hub, with helm upgrade … --reset-then-reuse-values, or every other value you set is reset — see the Helm chart reference's Upgrade). coreAgent.podLogs is what sets FLEEGER_K8S_POD_LOGS on the bundled agent; leaving it at its off default collects nothing even with loki.url set. With all it collects the hub's own pods too; their streams are the hub's logs, which only a holder of hub_logs.view reads.
  • Raw manifests (deploy/k8s) — edit FLEEGER_K8S_POD_LOGS in optional/cluster-monitoring/core-agent.yaml and uncomment the [loki] / loki_url entries in base/10-config.yaml.

Live agent connections — /connz

The NATS server's monitoring endpoint (:8222, internal-only) exposes each live client connection. The gateway proxies it at /api/agents/connz (bearer token + location.view); agents name their NATS connection agent-<location>, so the UI correlates a connection to a location and shows its RTT, uptime, throughput, and subscription count on the Services → Agents tab. Because agents tunnel NATS over wss through the single public edge, the peer IP the broker sees is the edge, not the agent's own address.

Dashboard & the Services page

The dashboard's Hub services (meta) panel queries VictoriaMetrics for the {location="meta"} self-metrics — per-service version, uptime, memory, and total requests — merged by the service label.

The Services page (nav: Services) goes deeper: a compact, expandable row per service with its endpoint (instance label), version, uptime, memory, CPU, and requests-by-status, plus the inline log tail above; a second Agents tab carries the connection detail from /connz.

Notes & limitations

  • The meta Location does not report a spoke-agent heartbeat (it isn't polled over NATS); its liveness comes from scrape target health, not the ingestion-svc reachability state used for homelab/hetzner.
  • page_size (4096) and _SC_CLK_TCK (100) are hard-coded in the metrics module to avoid a libc dependency — correct on all common Linux hosts; these two process gauges would be slightly off on an exotic kernel config.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.