Zum Inhalt springen
Produkte

// Betrieb

Data retention (dead-resource retention)

Fleeger keeps time-series (VictoriaMetrics), logs (Loki), and derived state (SurrealDB) about every resource it monitors. Two pressures pull in opposite directions:

  • Forensics — when a pod/VM/resource dies you often want its metrics and logs to outlive it, so you can investigate why it died.
  • Cost / cardinality — high-churn resources (K8s pods, Jobs, CronJob runs; every rollout mints a new name) otherwise pile up dead series and rows forever, which is the root of the ingestion resource_state growth and the VictoriaMetrics cardinality sawtooth.

The dead-data retention policy serves both: one configurable window that says how long a resource's data is kept after the resource is gone. Set it long for forensics, short for cost.

This is distinct from platform backup retention (backup.retentionDays), which prunes off-box backup sets, not live monitoring data.

The policy model

A single, versioned policy record lives in db=inventory as retention_policy:current (inventory-svc owns platform config). It is edited via the admin API and journalled into retention_policy_revision (append-only), so every change is audited and any prior version can be rolled forward.

Field Meaning
reaper_enabled Master switch. false by default — nothing is deleted until an admin opts in.
sweep_interval_secs How often the reaper sweeps (≥ 60).
default_mode + default_ttl_secs The global dead-data rule.
per_kind[] Per-resource-kind overrides { key, mode, ttl_secs } (e.g. keep k8s_node history 90d but reap dead k8s_pod after 3d).
per_location[] Per-location overrides (stretch).
vm_retention_secs An informational reflection of the deploy-level VictoriaMetrics retention, used only to validate that a keep_for dead-TTL can't exceed the process-level window (VM would drop the series anyway). It does not bound a purge: the reaper's and the offboard's deletes cover all of a series' history. There is no Loki counterpart: the policy used to carry a loki_retention_secs that nothing read; it was removed, and a PUT that still sends it is accepted with the value ignored.
revision_retention_secs / revision_min_per_entity Config-history window: an age ceiling (blank/null = keep forever) and a per-entity floor. A revision is removed only when it is both past the ceiling and outside the newest N for its entity.
alert_retention_secs Age ceiling for a resolved alert, in seconds. null = keep forever, which is the default. A firing alert is never removed whatever its age. Capped at a hundred years (MAX_ALERT_RETENTION_SECS, which is MAX_TTL_SECS) — use null for keep-forever rather than a very large number.

PUT /retention replaces the whole policy. A field omitted from the body is stored as null, not left alone — so a script that builds a request from a subset of this table will silently clear the rest.

Send back the version you read as expected_version (on PUT /retention and on POST /retention/rollback): if the policy was saved since, the request is refused with 409 STALE and nothing is written, so read it again and reapply the edit. Without it the save applies regardless, as before. Either way every save gets its own version and journal row: two saves made at once are serialised, never recorded as the same version.

Modes: keep_forever (never reap), keep_for (reap ttl_secs after last-seen), delete_immediately (reap as soon as dead). A keep_for ttl_secs, like revision_retention_secs and alert_retention_secs, is capped at a hundred years (MAX_TTL_SECS): past roughly 8e12 seconds a sweep's cutoff leaves SurrealDB's datetime range and every sweep fails. A larger value is refused with a 400; one stored before the cap existed is clamped by the sweeps and by a rollback. Use keep_forever (or null for a ceiling) rather than a very large number.

Resolution precedence: per-kind override → per-location override → global default. delete_immediately and short keep_for windows are floored to a liveness floor (LIVENESS_FLOOR_SECS, 300s) so a resource that is merely mid-poll is never mistaken for dead. The shape and resolution live in fleeger_shared::retention so inventory-svc and ingestion-svc resolve identically.

"Dead" detection & re-appearance

"Dead" is measured from last-seen, with no separate liveness bookkeeping:

  • Discovered inventory rows: resource.last_discovered (stamped every poll).
  • Metric-derived state: resource_state.last_seen (stamped on every metric).

A resource that keeps reporting keeps resetting that clock and is never reaped; a resource that vanishes crosses its dead-retention window and is swept. Re-appearance resets the clock for free — the same external_id/resource_id seen again bumps last-seen.

For metric/log stores (which key on source + resource_id, not an inventory kind), the reaper derives a kind hint from the resource id so per-kind overrides still apply: pod-… → k8s_pod, node-… → k8s_node, workload-… → k8s_workload, namespace-… → k8s_namespace; other sources fall back to the source name (proxmox, idrac, synthetic).

These metric-store kinds are their own vocabulary, distinct from inventory kinds, because resource_state records no inventory kind. Notably, a Deployment/StatefulSet/DaemonSet's replica series all share the id shape workload-{ns}-{name} (indistinguishable), so they map to k8s_workload, not k8s_deployment/k8s_statefulset/k8s_daemonset. So a per-kind rule governs a workload's metric series (k8s_workload) and its inventory rows (k8s_deployment/…) separately — set both if you mean to reap both. This is low-impact: workload series are low-cardinality and stable, so the keep-forever default suits them; the per-kind knob matters mainly for the high-churn k8s_pod series, where the hint is exact.

Enforcement (per store)

Enforcement is a scheduled sweep in the service that owns each store. Each reaper re-reads the policy every cycle, so toggling reaper_enabled or changing the interval takes effect without a restart. A cycle that fails (for example the policy cannot be read, or a database query errors) is logged at error level, and that reaper tries again an hour later rather than after sweep_interval_secs. A VictoriaMetrics or Loki purge that fails for one resource does not fail the cycle: it is logged as a warning and tried again on the next sweep.

Store Owner How
SurrealDB discovered resource rows inventory-svc (reaper.rs) Precedence-aware DELETE resource WHERE discovered=true AND … AND duration::secs(now - last_discovered) > ttl.
SurrealDB resource_state ingestion-svc (reaper.rs) DELETE rows past their window, superseded ones included (an older agent's unlabeled row, hidden once its resource reports with a cluster), so their series and streams are purged too. Each purge and the DELETE first re-check that the row is still past its window, so a resource that reports again while it is being reaped keeps its row and is not purged further. With the reaper disabled a superseded row stays, hidden, for good; that is harmless, and its series leave VictoriaMetrics with --retentionPeriod.
VictoriaMetrics series ingestion-svc Targeted POST /api/v1/admin/tsdb/delete_series matched on {location,source,resource_id,cluster} (cluster="" for a row without one, so another cluster's resource with the same id keeps its data).
Loki log streams ingestion-svc Loki compactor delete API (POST /loki/api/v1/delete) matched on {location,resource_id,cluster}, the cluster as for the series, over all of the streams' history (start at the Unix epoch), however long Loki's retention_period is. No end is sent, so the interval ends at Loki's own now: Loki refuses an end past its clock (400 deletes in the future are not allowed), which the hub's clock running even a second ahead of Loki's would otherwise cause. Requires deletion_mode: filter-and-delete.
SurrealDB resolved alert rows alerting-svc (reaper.rs) DELETE resolved alerts whose resolved_at is older than alert_retention_secs, oldest first, in bounded batches; each batch is a range of the alert_resolved index (state, resolved_at, migration 0017), so its cost does not grow with the resolved history. Firing alerts are never touched; with alert_retention_secs unset nothing is reaped.
SurrealDB expired mute rows alerting-svc (reaper.rs) DELETE mutes whose until is more than a day past, every reaper cycle, in bounded batches over the mute_until index (migration 0019). Not governed by the retention policy: nothing reads an expired mute, so they go even with alert_retention_secs unset or the policy unreachable. Shares the reaper's task: if the reaper's HTTP client cannot be built at startup (logged as reaper disabled), expired mutes are not deleted either.

The global VictoriaMetrics --retentionPeriod and Loki retention_period remain the hard outer bounds — they are deploy config (docker-compose / Helm values / k8s manifests), not something the policy changes at runtime, and a dead-TTL is validated never to exceed vm_retention_secs.

Audit

Reaping is not journalled per reaped row — that would reintroduce the very unbounded growth this feature cures. Instead a sweep that reaped something writes one retention_sweep_log row (counts of what was deleted, per store); a sweep with nothing to reap writes none. inventory-svc also records a sweep that failed. ingestion-svc and alerting-svc (as component: "alerting") post their summaries to inventory-svc over the internal API so all retention audit lands in one place, readable at GET /retention/sweeps. Only the reapers write it: the gateway forwards GET /api/retention/sweeps and nothing else there, so no client can add a row. A JetStream drain is not a sweep: it is recorded in the audit log as nats.stream.drain.

The audit log's retention

audit_retention_days (default 400, 0 keeps it forever, otherwise 30 days at least) bounds the hub-wide audit log. The inventory reaper prunes it each cycle while reaper_enabled is set, and journals each prune in the log itself (audit.prune). A PUT without the field keeps the stored value.

Settings → Retention (UI)

An admin-only page (iam.manage) under the settings surfaces lets an operator:

  • toggle the reaper and set the sweep interval;
  • set the global default and per-kind / per-location overrides;
  • set how long the audit log is kept (days, 0 = forever);
  • set the VictoriaMetrics retention bound the dead-data TTLs are validated against (with a note that changing the process-level retention needs a redeploy);
  • preview impact (GET /retention/preview: how many dead resources are past the window now, by kind) before saving;
  • review recent sweeps and roll back to a prior policy version.

Because shortening retention deletes data, saving goes through a destructive-action confirmation.

Safe defaults

Installing the migration changes nothing: the reaper is disabled and the default mode is keep_forever, so dead data keeps ageing out on the deploy-level VM/Loki window exactly as before. The new controls are strictly opt-in.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.