// Betrieb
Data retention (dead-resource retention)
Fleeger keeps time-series (VictoriaMetrics), logs (Loki), and derived state (SurrealDB) about every resource it monitors. Two pressures pull in opposite directions:
- Forensics — when a pod/VM/resource dies you often want its metrics and logs to outlive it, so you can investigate why it died.
- Cost / cardinality — high-churn resources (K8s pods, Jobs, CronJob runs;
every rollout mints a new name) otherwise pile up dead series and rows forever,
which is the root of the ingestion
resource_stategrowth and the VictoriaMetrics cardinality sawtooth.
The dead-data retention policy serves both: one configurable window that says how long a resource's data is kept after the resource is gone. Set it long for forensics, short for cost.
This is distinct from platform backup retention (
backup.retentionDays), which prunes off-box backup sets, not live monitoring data.
The policy model
A single, versioned policy record lives in db=inventory as
retention_policy:current (inventory-svc owns platform config). It is edited via
the admin API and journalled into retention_policy_revision (append-only), so
every change is audited and any prior version can be rolled forward.
| Field | Meaning |
|---|---|
reaper_enabled |
Master switch. false by default — nothing is deleted until an admin opts in. |
sweep_interval_secs |
How often the reaper sweeps (≥ 60). |
default_mode + default_ttl_secs |
The global dead-data rule. |
per_kind[] |
Per-resource-kind overrides { key, mode, ttl_secs } (e.g. keep k8s_node history 90d but reap dead k8s_pod after 3d). |
per_location[] |
Per-location overrides (stretch). |
vm_retention_secs |
An informational reflection of the deploy-level VictoriaMetrics retention, used only to validate that a keep_for dead-TTL can't exceed the process-level window (VM would drop the series anyway). It does not bound a purge: the reaper's and the offboard's deletes cover all of a series' history. There is no Loki counterpart: the policy used to carry a loki_retention_secs that nothing read; it was removed, and a PUT that still sends it is accepted with the value ignored. |
revision_retention_secs / revision_min_per_entity |
Config-history window: an age ceiling (blank/null = keep forever) and a per-entity floor. A revision is removed only when it is both past the ceiling and outside the newest N for its entity. |
alert_retention_secs |
Age ceiling for a resolved alert, in seconds. null = keep forever, which is the default. A firing alert is never removed whatever its age. Capped at a hundred years (MAX_ALERT_RETENTION_SECS, which is MAX_TTL_SECS) — use null for keep-forever rather than a very large number. |
PUT /retentionreplaces the whole policy. A field omitted from the body is stored asnull, not left alone — so a script that builds a request from a subset of this table will silently clear the rest.
Send back the version you read as expected_version (on PUT /retention
and on POST /retention/rollback): if the policy was saved since, the request
is refused with 409 STALE and nothing is written, so read it again and
reapply the edit. Without it the save applies regardless, as before.
Either way every save gets its own version and journal row: two saves made at
once are serialised, never recorded as the same version.
Modes: keep_forever (never reap), keep_for (reap ttl_secs after last-seen),
delete_immediately (reap as soon as dead). A keep_for ttl_secs, like
revision_retention_secs and alert_retention_secs, is capped at a hundred
years (MAX_TTL_SECS): past roughly 8e12 seconds a sweep's cutoff leaves
SurrealDB's datetime range and every sweep fails. A larger value is refused
with a 400; one stored before the cap existed is clamped by the sweeps and by
a rollback. Use keep_forever (or null for a ceiling) rather than a very
large number.
Resolution precedence: per-kind override → per-location override → global
default. delete_immediately and short keep_for windows are floored to a
liveness floor (LIVENESS_FLOOR_SECS, 300s) so a resource that is merely
mid-poll is never mistaken for dead. The shape and resolution live in
fleeger_shared::retention so inventory-svc and ingestion-svc resolve identically.
"Dead" detection & re-appearance
"Dead" is measured from last-seen, with no separate liveness bookkeeping:
- Discovered inventory rows:
resource.last_discovered(stamped every poll). - Metric-derived state:
resource_state.last_seen(stamped on every metric).
A resource that keeps reporting keeps resetting that clock and is never reaped; a
resource that vanishes crosses its dead-retention window and is swept. Re-appearance
resets the clock for free — the same external_id/resource_id seen again bumps
last-seen.
For metric/log stores (which key on source + resource_id, not an inventory
kind), the reaper derives a kind hint from the resource id so per-kind overrides
still apply: pod-… → k8s_pod, node-… → k8s_node, workload-… →
k8s_workload, namespace-… → k8s_namespace; other sources fall back to the
source name (proxmox, idrac, synthetic).
These metric-store kinds are their own vocabulary, distinct from inventory
kinds, because resource_state records no inventory kind. Notably, a
Deployment/StatefulSet/DaemonSet's replica series all share the id shape
workload-{ns}-{name} (indistinguishable), so they map to k8s_workload, not
k8s_deployment/k8s_statefulset/k8s_daemonset. So a per-kind rule governs a
workload's metric series (k8s_workload) and its inventory rows
(k8s_deployment/…) separately — set both if you mean to reap both. This is
low-impact: workload series are low-cardinality and stable, so
the keep-forever default suits them; the per-kind knob matters mainly for the
high-churn k8s_pod series, where the hint is exact.
Enforcement (per store)
Enforcement is a scheduled sweep in the service that owns each store. Each
reaper re-reads the policy every cycle, so toggling reaper_enabled or changing
the interval takes effect without a restart. A cycle that fails (for example
the policy cannot be read, or a database query errors) is logged at error
level, and that reaper tries again an hour later rather than after
sweep_interval_secs. A VictoriaMetrics or Loki purge that fails for one
resource does not fail the cycle: it is logged as a warning and tried again on
the next sweep.
| Store | Owner | How |
|---|---|---|
SurrealDB discovered resource rows |
inventory-svc (reaper.rs) |
Precedence-aware DELETE resource WHERE discovered=true AND … AND duration::secs(now - last_discovered) > ttl. |
SurrealDB resource_state |
ingestion-svc (reaper.rs) |
DELETE rows past their window, superseded ones included (an older agent's unlabeled row, hidden once its resource reports with a cluster), so their series and streams are purged too. Each purge and the DELETE first re-check that the row is still past its window, so a resource that reports again while it is being reaped keeps its row and is not purged further. With the reaper disabled a superseded row stays, hidden, for good; that is harmless, and its series leave VictoriaMetrics with --retentionPeriod. |
| VictoriaMetrics series | ingestion-svc | Targeted POST /api/v1/admin/tsdb/delete_series matched on {location,source,resource_id,cluster} (cluster="" for a row without one, so another cluster's resource with the same id keeps its data). |
| Loki log streams | ingestion-svc | Loki compactor delete API (POST /loki/api/v1/delete) matched on {location,resource_id,cluster}, the cluster as for the series, over all of the streams' history (start at the Unix epoch), however long Loki's retention_period is. No end is sent, so the interval ends at Loki's own now: Loki refuses an end past its clock (400 deletes in the future are not allowed), which the hub's clock running even a second ahead of Loki's would otherwise cause. Requires deletion_mode: filter-and-delete. |
SurrealDB resolved alert rows |
alerting-svc (reaper.rs) |
DELETE resolved alerts whose resolved_at is older than alert_retention_secs, oldest first, in bounded batches; each batch is a range of the alert_resolved index (state, resolved_at, migration 0017), so its cost does not grow with the resolved history. Firing alerts are never touched; with alert_retention_secs unset nothing is reaped. |
SurrealDB expired mute rows |
alerting-svc (reaper.rs) |
DELETE mutes whose until is more than a day past, every reaper cycle, in bounded batches over the mute_until index (migration 0019). Not governed by the retention policy: nothing reads an expired mute, so they go even with alert_retention_secs unset or the policy unreachable. Shares the reaper's task: if the reaper's HTTP client cannot be built at startup (logged as reaper disabled), expired mutes are not deleted either. |
The global VictoriaMetrics --retentionPeriod and Loki retention_period remain
the hard outer bounds — they are deploy config (docker-compose / Helm values /
k8s manifests), not something the policy changes at runtime, and a dead-TTL is
validated never to exceed vm_retention_secs.
Audit
Reaping is not journalled per reaped row — that would reintroduce the very
unbounded growth this feature cures. Instead a sweep that reaped something
writes one retention_sweep_log row (counts of what was deleted, per
store); a sweep with nothing to reap writes none. inventory-svc also records a
sweep that failed. ingestion-svc
and alerting-svc (as component: "alerting") post their summaries to
inventory-svc over the internal API so all retention audit lands in one place,
readable at GET /retention/sweeps. Only the reapers write it: the gateway
forwards GET /api/retention/sweeps and nothing else there, so no client can
add a row. A JetStream drain is not a sweep: it is recorded in the
audit log as nats.stream.drain.
The audit log's retention
audit_retention_days (default 400, 0 keeps it forever, otherwise 30 days
at least) bounds the hub-wide audit log. The inventory reaper
prunes it each cycle while reaper_enabled is set, and journals each prune in
the log itself (audit.prune). A PUT without the field keeps the stored
value.
Settings → Retention (UI)
An admin-only page (iam.manage) under the settings surfaces lets an operator:
- toggle the reaper and set the sweep interval;
- set the global default and per-kind / per-location overrides;
- set how long the audit log is kept (days,
0= forever); - set the VictoriaMetrics retention bound the dead-data TTLs are validated against (with a note that changing the process-level retention needs a redeploy);
- preview impact (
GET /retention/preview: how many dead resources are past the window now, by kind) before saving; - review recent sweeps and roll back to a prior policy version.
Because shortening retention deletes data, saving goes through a destructive-action confirmation.
Safe defaults
Installing the migration changes nothing: the reaper is disabled and the default
mode is keep_forever, so dead data keeps ageing out on the deploy-level VM/Loki
window exactly as before. The new controls are strictly opt-in.