Zum Inhalt springen
Produkte

// Benutzung · Monitoring & Alerting

Alerting

alerting-svc turns the metrics pipeline into notifications. It observes and notifies only — nothing in this service can trigger an action against infrastructure. Rules and channels are user-editable from the dashboard (stored in db=alerting); no redeploy is needed to change what fires or where it goes.

Two evaluators, one alert lifecycle

flowchart LR
    subgraph alerting-svc
        TE["Threshold evaluator\n(polls VictoriaMetrics)"]
        HB["Heartbeat monitor\n(taps NATS metrics.>)"]
        SM["System monitor\n(polls ingestion state →\nbuilt-in location/resource alerts)"]
        EN["Alert engine\n(dedup · mute · dispatch)"]
        DB[("SurrealDB\ndb=alerting\nrules · channels · alerts · mutes")]
    end
    VM[("VictoriaMetrics")]
    NATS["NATS metrics.>"]
    ING["ingestion-svc\n/state/locations · /state/resources"]
    CH["Channels\nntfy · discord · matrix · email"]

    VM --> TE --> EN
    NATS --> HB --> EN
    ING --> SM --> EN
    EN --> DB
    EN --> CH
    DB -.rules/channels.-> TE
    DB -.rules/channels.-> HB
    DB -.built-in rules/channels.-> SM

Threshold rules (kind = "threshold")

Each enabled threshold rule carries a PromQL query, a comparator (gt/gte/lt/lte/eq, i.e. >/≥/</≤/=), and a threshold. On every eval_interval_secs the evaluator runs the query against VictoriaMetrics and, for each returned series, compares the instant value. A breach fires; a series that no longer breaches resolves. This is how "CPU spike", "disk full", "sensor over-temp", and a per-resource "host down" (up == 0 filtered to a resource) are expressed — they're all just queries. A query may filter itself like that: once the host is up again, up == 0 no longer returns it at all, and the evaluator asks VictoriaMetrics whether up of that resource is still reported to tell a recovery from a host that went away (see Resolve below). Use eq (=) for discrete metrics (a state, a count, up 0/1); it does an exact match, so it's a poor fit for continuous ratios that rarely land on an exact value.

The cluster label. Proxmox and Kubernetes series carry a cluster label, so a rule over them alerts once per cluster's resource, unless its query aggregates the label away. When an agent is upgraded, the series it wrote without the label still appears in query results for a few minutes next to the labeled one; the evaluator skips such an unlabeled series while its labeled twin (the same labels plus cluster) is in the result, and carries a firing alert of the unlabeled series over to the labeled one, with its breach clock, so no second alert fires. That holds while the twin is in the result: when it drops out (it went away, or the rule's query filters it out once it stops breaching) while the old series is still within the lookback window, the old series is evaluated on its own again and can fire, or recover, under the old identity once. A new agent never sends a Proxmox or Kubernetes series unlabeled (it labels a cluster it could not scope registration:<id>), so only an older agent's series is ever skipped, and an unscoped cluster's series alert apart from a scoped cluster's.

Sustained-breach window (for_secs). A rule may require the breach to persist before firing: for_secs (0 = fire on the first breaching cycle, the default). The evaluator tracks per-(rule, series) when the breach started and only fires once it has held for for_secs; a series that stops breaching resets the clock. This is essential for workload-availability alerting (below) and useful generally to ride out transient blips. The breach clock is kept in memory, so a restart of alerting-svc starts it again: a window longer than the service usually runs between restarts does not fire in practice. for_secs may be at most ~10 years, the same maximum as stale_secs, which refuses the absurd values that could never be satisfied, and may not be negative (a negative value stored before that reads as 0, and 0 clears it). As with stale_secs, only a changing value is checked on an update. A threshold rule is created only with its query, comparator and a finite threshold.

Alert on workloads, not individual pods. A kubectl rollout restart terminates the old ReplicaSet's pods and starts new ones by design, so alerting on individual pod up produces guaranteed noise during any healthy deploy. Instead, the spoke-agent emits per-workload replica gauges (k8s_workload_desired / _ready / _available / _updated, labelled namespace/workload/kind), and the built-in workload_degraded rule alerts on k8s_workload_desired - k8s_workload_available > 0 held for its for_secs window (default 10 min). A healthy rolling update stays within its maxUnavailable budget and clears well inside the window, and the generous window also rides out a slow first rollout (big image pull / slow startup) where available lags desired for minutes; only a genuinely stuck/failed rollout fires. Raise for_secs to exceed your slowest legitimate rollout. Per pod up remains for visibility, but prefer the workload rule for alerting.

Heartbeat rules (kind = "heartbeat") — "location unreachable"

The heartbeat monitor taps NATS metrics.> directly and tracks the last time each location produced any metric, on its own metrics.<location>.… or on the bundled core agent's metrics._core.<location>.… — the latter only while that location is local_collector (read from inventory-svc's GET /internal/locations/local-collectors at start and every 10 s, a failed read keeping the set; until a read succeeds no _core heartbeat is credited, with one WARN, so while inventory-svc is down when alerting-svc starts, a local_collector location only the core agent reports for can be raised unreachable although it is up, until a read succeeds) — deliberately independent of ingestion-svc (a separate failure domain: alerting still detects silence even if ingestion is down). When a location's last-seen age exceeds the staleness window (stale_secs, default heartbeat_stale_secs), a "location unreachable" alert fires. Every agent sends a heartbeat for each location it serves every poll interval, and at least every 30 s, on a timer of its own: a collection cycle held up by targets that time out does not delay it.

stale_secs must be between 60 and ~10 years, the same maximum the retirement ceilings take. The upper bound exists because the field is signed on the record and every monitor compares it as an unsigned window: a negative value used to wrap to a window no age could reach, silently disabling the rule and — via the widest-window fold the ceiling is derived from — the retirement prune for every other rule with it. 0 is refused too, in the other direction: it makes every age stale, so the rule fires on everything. The floor is two heartbeat intervals: an agent beats at least every 30 s, so a location's age climbs to 30 s between every two beats, and a window of that or less fires on a healthy location every cycle; two intervals leave room for one late or lost beat. The rule form says so, and the System alerts card refuses a shorter window before sending it. A rule stored with a window under 60 before this floor existed is evaluated at heartbeat_stale_secs instead, with a warning naming it, as below.

The [alerting] config key heartbeat_stale_secs — the installation-wide default a rule falls back to — takes the same 60–315360000 range, checked at startup: an out-of-range value fails the pod immediately rather than starting in a broken state, with an error naming the key and the value. The floor is the same two heartbeat intervals: a shorter default window fires every heartbeat rule that sets no window of its own, and both built-in unreachable alerts, on healthy subjects between two beats. Every shipped config sets 90. Unlike its two neighbours resource_stale_ceiling_secs and heartbeat_alert_ceiling_secs, 0 is not an "off" switch here — there is no way to disable the staleness window, because every heartbeat rule, both built-in unreachable alerts and the location-liveness gate for check_down are all defined in terms of it. A 0 there means every subject is unreachable on every cycle — and, because check_down gates on the same window inverted (it only evaluates checks at locations seen within it), that alert goes silent at the same moment the others turn maximally noisy.

Note the floor is two heartbeat intervals, not one eval_interval_secs. A window shorter than the evaluation cycle is legitimate: the monitors compare a wall-clock age (kept by the NATS tap, and by ingestion-svc's last-seen state) that the cycle does not bound, so a short window is merely observed more coarsely than it is written, not rendered meaningless.

Three details of how the range is enforced, none of them obvious:

  • On create, omitting the field uses heartbeat_stale_secs. Out-of-range values are refused.
  • On update, leaving stale_secs out of the request means "leave it as it is", and sending null clears the override, returning the rule to heartbeat_stale_secs. Sending a value equal to the one already stored is accepted even if it is out of range, deliberately: a client that reads a rule and sends every field back would otherwise have that rule's name, severity or channels blocked until its window was fixed too. A genuinely new out-of-range value is always refused.
  • At evaluation, a row holding an out-of-range window — from before the check existed, or restored from an older backup — is evaluated at heartbeat_stale_secs, with a warning naming the rule's id. Such a row keeps its stored value, and GET /rules keeps reporting it in stale_secs until someone overwrites or clears the field.

Every rule answer of a kind that reads a window (heartbeat and the built-in location_unreachable / resource_unreachable) also carries effective_stale_secs, the window the monitors evaluate it at, resolved as above, and default_stale_secs, the configured heartbeat_stale_secs. The alerts page shows those, so an installation that changed heartbeat_stale_secs sees its own default rather than the shipped 90. Other kinds answer null for both.

A heartbeat rule's window has a second, tighter bound when the retirement ceiling heartbeat_alert_ceiling_secs is on (default 604800, 7 days): it must be at most the ceiling minus two eval_interval_secs, so 604740 with the defaults. A location silent for longer than the ceiling is retired: the monitor forgets it and its alert stops. Retirement is switched off entirely while any enabled heartbeat rule has a window at or past the ceiling, so one such rule used to stop retirement for every heartbeat rule, with only a warning in the log. The two cycles of margin leave room for the evaluation that fires the alert before the location is retired. The config-level default gets the same margin: alerting-svc refuses to start when heartbeat_alert_ceiling_secs is on and smaller than heartbeat_stale_secs plus two eval_interval_secs, so the default window is always accepted. With the ceiling set to 0 (retirement off) there is no extra bound.

Each cycle, the heartbeat rules read the last-seen map before locations past the ceiling are dropped from it. A cycle delayed by a slow notifier can carry a location's age past both its window and the ceiling in one step; it still fires "location unreachable" in that cycle, and its alert is retired as no longer monitored once it has been firing for the ceiling. The bound is checked:

  • when a heartbeat rule is created, enabled or not;
  • when a heartbeat rule's stale_secs is changed to a new value;
  • when a disabled heartbeat rule is switched on, against the window it already holds, even if the request sends that window back unchanged. A rule over the bound cannot be switched on until its window is lowered, which can be done in the same request. A rule whose stored window is outside the 1–315360000 range above can be switched on: that window is not used anyway.

Otherwise, as above, re-sending the window a rule already holds is not refused, so an enabled rule stored with a wider window stays editable. Lowering such a window in steps is refused until a step reaches the bound; clearing it with null always works.

The refusal names the limit. A site that needs a longer window raises heartbeat_alert_ceiling_secs instead. A rule stored with a wider window before this check existed, or left over the bound by a later change to heartbeat_alert_ceiling_secs, eval_interval_secs or heartbeat_stale_secs, keeps running as before, but alerting-svc names it at startup: one warning per heartbeat rule past the bound, enabled or not, with its id, window and the bound. If an enabled rule's window reaches the ceiling, retirement stays off for every heartbeat rule and the log also warns on each cycle. If it is over the bound but still below the ceiling, retirement stays on, and a location can be dropped (logged) in the same cycle that rule first fires for it. Other rule kinds, including the built-in resource_unreachable, are not affected.

description and scope_location follow the same update rule as stale_secs: left out, they are untouched; null clears them — an empty description, and a rule widened from one location back to all of them. No other field clears: for query, comparator and threshold a null is the same as leaving the field out, because a threshold rule cannot be evaluated without all three. A blank or whitespace-only scope_location is no scope either, on create and update alike, and a scope is stored trimmed; a rule stored before that is read the same way.

Scoping a heartbeat rule to one location, or moving its scope to another, retires its firing alerts for every other location, the way disabling a rule retires all of them: each is cleared with a [CLEARED] notice saying the rule no longer watches that location, never as "back online". The alert for the location it still watches keeps firing. An alert of that kind left over anyway (raised by a cycle that read the rule just before the change) is cleared the same way on the next heartbeat cycle. Widening a rule back to every location retires nothing.

Deleting a rule retires its firing alerts the same way, before the rule goes. An alert is raised, or re-notified, only while its rule exists and is enabled, checked in the statement that writes it, so a monitor cycle that read the rule just before a delete or a disable raises nothing after it (a delete disables the rule before it retires the alerts and removes the row). A firing alert left anyway, whose rule no longer exists (left by a delete older than that) or is disabled (a fire that committed alongside the disable, or a retirement that failed), is resolved by a sweep once every eval_interval_secs, the first time one interval after startup. A disabled rule's alerts are swept only once the rule was last written more than that interval ago, and at least ten minutes ago (at most a day), so a disable or delete in progress retires them itself, with its notices, however slow they are. A repeated DELETE /rules/{id} of a gone rule resolves its alerts at once. The sweep clears without a notice and logs a warning naming the alert and the rule.

The rule form sends stale_secs (and a threshold rule's for_secs) only when you changed it: a stored value past 2^53 comes back from JSON rounded, and echoing it would be a change the server refuses. It sends description and scope_location as the form holds them; the token below is what keeps an old form from writing back old values. An empty "Stale after" box clears the override; a typed 0 is sent as 0 and refused (unless the rule already holds 0, which the equal-value rule above accepts).

Stale rule edits

An update reads the rule, applies the request to it and writes the whole rule back. A client that read the rule before someone else changed it would put the old values back with a 200. To prevent that, PUT /rules/{id} takes an optional expected_updated_at: the rule's updated_at as the client read it, sent back exactly as read (an RFC 3339 string; a value that is not one is a 422 naming the field). If the rule has been written since, the update is refused with 409 and code STALE, before any other check, and nothing is written: read the rule again, reapply the edit, and send the new updated_at. A rule that no longer exists is a 404, not a 409.

Every write of a rule through the API gives it a new updated_at, so every such write spends the token: an edit, a toggle, a channel tick on a built-in's card, and deleting one of the rule's channels (which removes the channel from the rule). Migrations are the exception: they run under OPTION IMPORT, which skips the VALUE that sets it, so a migration that seeds or changes a rule must set updated_at itself for open forms to notice. The rule form sends the token and, on STALE, reloads the rule, keeps the fields you edited, takes the new values of the ones you did not, and lists the fields both sides changed with the value the server now holds; save again to apply. The channel ticks of the System alerts cards send the token too, so a card on a page older than the rule is refused and the page reloads instead of putting back a channel list (perhaps holding a deleted channel) it read before. The on/off toggles and the cards' value boxes send narrow patches without a token, so a quick toggle is not refused over an unrelated change.

Without expected_updated_at (or with null) the update applies as before, with whatever the request carries, so a script that sends back a rule it read earlier is not protected against having read an old one. The write itself is always conditional, though: a rule written between the update's own read and its write is read again and the update made again. A channel update (PUT /channels/{id}) takes the same expected_updated_at token and is conditional the same way (see Editing a channel); a channel delete (DELETE /channels/{id}) that loses a write conflict (to a concurrent write of the channel, or of a rule being routed to it) is run again too.

Rule creates and updates that route a rule to a channel, and channel updates and deletes, all write the channel's row (see below), so they conflict with each other as a matter of course — many rules provisioned at once onto one shared channel, say. Such a write is retried up to 16 times, after a random pause in a window that doubles each time (10 ms up to 250 ms, about 3 s in all at most), which absorbs parallel provisioning. Only contention that outlasts that is answered 409 CONFLICT ("another write touched … at the same time; try again").

A channel that an update adds to a rule (one the rule does not already hold) must exist, or the update is refused with a 400 naming it and nothing is written. The check is part of the conditional write, and the write touches the added channel's row, so a channel delete committing at the same time makes one of the two run again: the update then finds the channel gone, or the delete detaches it from the rule as well. A channel delete detaches the channel from every rule and deletes it in one transaction. So no update, with or without a token, routes a rule to a deleted channel. A channel id the rule already holds is not checked, so a rule stored with one from before is still editable by a client that sends its channels back.

Creating a rule (POST /rules) checks every channel it names the same way: one that does not exist — never did, or was deleted since the client listed it — refuses the create with a 400 naming it, and no rule is stored. The check and the create are one transaction that touches each channel's row, so a channel delete committing at the same time makes one of the two run again: the create then finds the channel gone, or the delete detaches it from the new rule. Creates routed to the same channel conflict with each other the same way, and are retried as above.

Migration 0016 gives every rule without an updated_at one: the built-ins were seeded in import mode, which skips the VALUE time::now() that sets it. A restore does not undo that: it replaces the alerting database with the backup's, migration state included, so a backup from before 0016 has no record of it, and the migrate step runs it again before alerting-svc starts. If a rule ever shows updated_at: null anyway, it refuses any token and applies a request without one until its next write; this covers it at once (namespace fleeger, database alerting):

UPDATE alert_rule SET updated_at = time::now() WHERE updated_at = NONE;

Each kind of rule accepts only the condition fields it is evaluated from, and the API refuses any other with a 400 naming the field: a threshold rule reads query, comparator, threshold and for_secs; a heartbeat rule stale_secs and scope_location; the built-in location_unreachable and resource_unreachable only stale_secs; check_down none of them. Before, such a field was stored and never read, so a request that meant something else succeeded without effect. The built-in threshold rules (backup_stale, cert_expiring, cert_invalid, workload_degraded, nats_storage_high, nats_stream_near_cap, nats_stream_discarding, nats_sampler_stale, nats_account_jetstream_disabled, log_batches_dropped, series_bound_exceeded, vm_series_limit_reached) count as threshold rules here, so they refuse stale_secs and scope_location. Their threshold and for_secs are editable; their query and comparator are what makes each that built-in, so a changing one is refused with a 400 as well. No built-in can be renamed: a new name gets a 400 too. A null, a blank scope_location and a for_secs of 0 (what an omitted key means) set nothing and are accepted; on an update they clear a leftover value. Otherwise only a changing value is refused, against the kind the rule was created with, so a client that sends back a rule as it read it is not refused over a field the row already holds from before.

To stop a rule firing, set enabled = false; a window long enough that it never fires is not a supported way to park one.

This is distinct from a per-resource "host down" alert: host down is a threshold rule about one resource inside a location that is still reporting; location unreachable means the whole location (its spoke-agent) has gone silent, so we can no longer say anything about the resources within it. A scoped rule watches one named location; an unscoped rule watches every location that has ever reported.

Built-in "system" alerts (always on, no rule to write)

Two rules are seeded (migration 0002_system_alerts) and always present: location_unreachable and resource_unreachable. They're marked builtin — the API refuses to delete them, refuses a new name, and refuses condition fields other than stale_secs; only their channels, enabled, severity, description, stale_secs, and mute state are user-editable. (A null empties a built-in's description like any other rule's; the seeded text is not restored automatically, but can be sent again.) A user picks where each notifies (they ship with no channels, so they track silently until a channel is chosen). Each built-in's card in the System alerts section of Alerts → Rules sets its channels, enabled, mute and the condition values its kind reads: stale_secs for these two, threshold and for_secs for the built-in threshold rules.

A third worker, the system monitor, drives them from ingestion-svc's persisted per-location and per-resource last-seen state (/state/locations, /state/resources) rather than an in-memory NATS tally. Because that state is persisted, an entity that was already silent when alerting-svc started still alerts — there's no startup blind spot (unlike the in-memory heartbeat monitor, which only sees a location go silent after it has observed traffic from it).

  • location_unreachable — a location whose newest metric is older than the stale window. Overlaps conceptually with an unscoped heartbeat rule, but it's the zero-config, always-on version.

  • resource_unreachable — a single resource whose metrics went stale while its location is still fresh (from 20 of one location firing at once, one grouped alert for that location; see below). A full-location outage is left to the location alert (so one dead site doesn't emit one alert per resource under it). "Fresh" is judged by location_unreachable's own window (its stale_secs, else heartbeat_stale_secs), the one its alert fires on, so a site is either covered by the location alert or by its resources' alerts, never by both or neither. The window applies while location_unreachable is disabled too. The check monitor's dark locations (below) use it as well, but never later than heartbeat_stale_secs: a silent site's probe samples leave its lookback window, so a raised window does not keep its checks judged. Crucially, the spoke-agent reports a stopped guest as up 0 on every cycle — it keeps reporting — so deliberately powered-off guests never go stale and never fire here. What does fire: a resource that genuinely stops being reported at all (an iDRAC host offline, a Proxmox node down, a vanished k8s node/pod).

    A staleness ceiling (resource_stale_ceiling_secs, default 30 min) bounds this: a resource silent longer than the ceiling is treated as decommissioned (a churned k8s pod whose name changed, a deleted VM) rather than a fresh outage, so it stops firing. Without the ceiling a resource that stops reporting alerts until ingestion forgets it, and by default ingestion never does: its resource_state row stays until something deletes it, so every deleted/renamed resource would alert forever. Location unreachable has no ceiling: a whole site down for days is still down.

    In normal operation three things delete a resource_state row (a wiped or restored ingestion database resets them all): deleting the registered resource or a rollback removing it (below), offboarding its location (which clears that location's alerts itself, as [CLEARED] … its location was offboarded; see Offboarding an agent and the check monitor below), and ingestion's retention reaper (services/ingestion-svc/src/reaper.rs). The reaper runs only once the retention policy is enabled (it ships disabled, with a keep_forever default), and deletes a row whose last_seen is older than the resource's resolved window (per kind, then per location, then the default; keep_forever never), at the first sweep after the window passes (sweep_interval_secs, default 3600 s, at least 60), purging the resource's VictoriaMetrics series and Loki streams with it. The shortest window is 300 s (LIVENESS_FLOOR_SECS in crates/fleeger-shared/src/retention.rs: delete_immediately, or a shorter keep_for), well under the default 1800 s ceiling. Whichever comes first, the ceiling or the sweep that reaps the row, ends a silent resource's alert; with neither (the ceiling disabled with 0 or not above the rule's stale_secs, and the reaper off or the window keep_forever) it keeps firing:

    • The ceiling first (always while the reaper is off or the window is keep_forever): it retires the alert with its [CLEARED] … has not reported for longer than the configured ceiling notice; reaping the row later changes nothing for alerting.
    • The reaper first: the resource drops out of ingestion's list and the next cycle clears its firing alert as vanished, [CLEARED] … is no longer reported by ingestion … This is NOT a recovery, whether its location is live or dark (a dark site's suppressed alert included). A resource whose rule window (the resource_unreachable rule's stale_secs) is longer than its retention window can be reaped before it ever fires.

    Either way, once its row is gone nothing fires for that resource again while it stays silent, and when it reports again ingestion records it anew: a resource whose alert was already cleared gets no "back online". location_state is not reaped, so location_unreachable is unaffected.

    Deleting a registered resource in Fleeger (a Proxmox node, a Kubernetes cluster, an iDRAC host, a check), or a full checkpoint rollback removing one , does not wait for the ceiling: the gateway has ingestion drop the state of everything collected through it, its discovered guests, nodes and pods included, so none of them fires, and an alert already firing for one is cleared as vanished ("no longer monitored"), never as recovered. That needs an agent that stamps its messages with the registration's inventory id (rc.3) and ingestion-svc reachable during the delete or rollback; otherwise the rows age out through the ceiling as described above. A resource a rollback recreates under its old id is no longer held back by its delete's 5-minute tombstone: a check reports again on its next run, while a recreated Proxmox node, Kubernetes cluster or iDRAC host comes back without its credential and reports nothing until one is attached again. Deleting a discovered resource on its own (a VM, a pod) clears nothing: the agent keeps collecting it through its registered parent.

    One alert per resource (unless it is grouped, see below): location, resource id and, for a Proxmox or Kubernetes resource, its cluster, so two clusters' vm-100 alert apart; the alert text (see Naming a cluster label) and labels name the cluster. A resource's alert raised while an older agent reported it without a cluster is carried over to the labeled resource once the upgraded agent reports it (ingestion then marks the unlabeled row superseded and leaves it out of the state list), rather than cleared as no longer monitored and raised again. With two clusters in a location this bridge cannot tell which cluster the shared alert belonged to: it moves to whichever cluster's resource is checked first.

    From 20 resources of one location firing at once, one grouped alert covers them (RESOURCE_GROUP_THRESHOLD in services/alerting-svc/src/system_monitor.rs). Counted are the resources that would newly fire this cycle: stale at a live location and within the ceiling, exactly as for one alone, without an alert of their own and not muted on their own (below). The agent chooses its resource ids, so one that invents ids and stops reporting them while its location stays live raised one alert and one notification for every invented resource, up to ingestion.max_resources_per_location (10,000 by default) at once; a real mass outage at a live site paged as often. Under 20, each resource alerts on its own exactly as before, and a single outage is unchanged. The count is per cycle: an agent that lets its invented ids go silent a few at a time (up to 19 per cycle) still pages for each of them, as before.

    • Identity: one grouped alert per location under the resource_unreachable rule, fingerprint resource_unreachable#group|<location>, which no resource's fingerprint can equal. Its labels are location (so a location mute silences it and offboarding the location clears it) and resources, the count; it has no resource_id.
    • Muted resources are never grouped: a resource with a live mute of its own (matched as the engine matches it, location and cluster scope included) keeps its own alert, muted and so silent, and does not count towards the 20. A member muted while the group fires leaves it the same way. A resource mute therefore keeps its promise however many resources are down around it.
    • Text: the count, the first 10 resources in lexicographic order of their text and how many more, and how long the most recent of them has been silent, e.g. 25 resources at 'Site A' unreachable — no metrics for 2m or longer: 'vm-00' ('proxmox'), … and 15 more. Each id, source and cluster name is cut to 64 characters (…) and quoted with \ and ' escaped, so an id cannot pose as several entries. Its subject is the group of resources at '<location>'; the text and the subject are rewritten each cycle they change (members, the location's name, or the silence growing by the minute), without paging.
    • Notifications: one when the group fires, then re-notified every renotify_secs like any alert, with the group as it is that cycle (the alert's text is brought up to date before it is re-notified). A resource joining or leaving the group pages nobody.
    • Alerts already firing when 20 others go down keep their own alert (re-notified, and recovered or retired, as before) and are not counted; the group's text says how many more alert on their own. So forming a group neither clears an alert nor pages twice for one resource (but see Rolling update below: while two pods run, a page can come both ways). Nineteen alerting on their own and one more going down form no group.
    • Sticky membership: a member leaves the group only by reporting again, going silent past the ceiling, leaving the state list or being muted. No member ever gets an alert of its own because the group shrank, so a real outage's recovery tail, or invented ids aging out one by one, pages nobody and nothing flaps around 20. A resource that goes down while the group fires joins it only while the group, counting it, has at least 20 members (20 or more going down at once always join); otherwise it is a new outage and alerts and pages on its own, so a single failure next to a shrunken group is not hidden in it. After an alerting-svc restart the monitor does not know the members, and holds every resource of the location that goes down without an alert of its own in the firing group.
    • Clearing: only once the group has no member left. The notice is [RESOLVED] … is back online only when every resource it covered while it fired reports again; [CLEARED] … has not reported for longer than the configured ceiling when every one went past resource_stale_ceiling_secs, and [CLEARED] … no longer reported by ingestion when every one left the state list, as each one's own alert would have been. Otherwise it is [CLEARED] … has no resource left that is unreachable without an alert of its own … This is NOT a recovery, saying how many were back online, past the ceiling, no longer reported, or are still unreachable but muted (and alert on their own). The monitor keeps what the group covered in memory: after an alerting-svc restart a group is never called a recovery, and its notice says the monitor knows only what it covered since the restart, or nothing of them. A group cleared by someone else while it fired (resolved by hand, or by the other pod during a rolling update) and formed again is a new group: its notice counts only what the new one covered. A clear by hand is not a way to silence a mass outage: the next cycle, the members still down form a new group if they are 20 or more, and otherwise each alerts and pages on its own. Mute the location or its resources instead.
    • Rolling update: for the moment two alerting-svc pods run, both run the monitor. Firing and clearing are deduplicated by fingerprint, so the same alert is not raised twice, but the two can disagree: the new pod starts without the group's members and treats it as after a restart, so one pod may fire a resource on its own that the other holds in the group, or clear the group with the unknown-members notice; a page can come both ways for that moment.
    • Ceiling and the backstop: a resource past the ceiling neither fires nor counts, as before; one that leaves the group that way gets no notice of its own (the group's text drops it, and its clear notice counts it). A resource let through by the backstop counts like any stale one.
    • Dark site: a location gone dark suppresses the group like its resources' alerts: a firing group is left exactly as it is, members and all, and none is raised there. Members going past the ceiling while the site is dark do not clear it; once the site is back they are judged as above. A location whose members all left the state list has its group cleared as no longer reported.

    The monitor samples each resource's age once per cycle and tests the ceiling before the window, so a slow cycle could carry an age from under heartbeat_stale_secs to past the ceiling between two samples, and the resource was retired without ever firing. alerting-svc therefore refuses to start when resource_stale_ceiling_secs is on and smaller than heartbeat_stale_secs plus two eval_interval_secs, the same margin the heartbeat ceiling gets. The margin covers ordinary cycles, but not a cycle held up for longer than two intervals (a hung upstream holds one for the whole http_timeout_secs) nor a resource_unreachable window set just under the ceiling with the rule's own stale_secs. For those, the monitor remembers when it last sampled the resource list (on the monotonic clock, so an NTP step does not move it). A resource past the ceiling that was not yet stale at that sample, judged by the rule's current window, has never been reported, so it is judged once more: if its location is up it fires, and the next cycle retires it as no longer monitored as usual. If its location is dark it is suppressed, like any stale resource there, and kept waiting, since the same hang that skipped its sample can make every location look dark; it fires once its location is up again, and leaves the queue if it reports again or drops out of ingestion's list. At most 10,000 resources wait; past that, one is retired unseen as before, with a warning. A resource already stale at the previous sample, or long gone, is retired without firing as before, and so is every resource on the first cycle after alerting-svc starts, after the rule is re-enabled, or after cycles failed for longer than the ceiling: then there is no usable previous sample, and every resource that went past the ceiling in that time would otherwise fire once and clear.

The system monitor reuses the same engine (dedup, mute, re-notify, dispatch) as the two evaluators, so built-in alerts behave identically to user rules. Metrics are keyed by a location's stable id, so the monitor also pulls inventory-svc's id → name map each cycle and writes the friendly name into the alert text (falling back to the id if inventory is unreachable).

A third built-in, check_down, is seeded by migration 0004_check_down and covers uptime / synthetic monitoring (see Uptime / synthetic monitoring). A fourth worker, the check monitor, drives it from the probe metrics: each cycle it reads probe_success{source="synthetic"} from VictoriaMetrics and, for every failing check, fires — but only after a few consecutive failed cycles (check_fail_confirmations, default 2) so a single blip never pages. A count whose wait ((count - 1) x eval_interval_secs) passes ~10 years is refused at startup, since check_down would never fire. Each vantage point (location + check) votes once, with its newest sample in the lookback window: the probe's reason label puts a passing and a failing probe in different series, and the older one must not count as a second vantage. Each check has its own window: twice its probe interval plus 3 minutes, never less than 5 minutes nor more than 64 days. That leaves room for one lost probe and for the lateness a check at the default minute always had: the agent probes a due check in its next collection cycle, so a location polling less often than every 3 minutes (poll_interval_secs) can still miss it. Each cycle the monitor asks inventory-svc how the enabled checks are probed (GET /internal/checks/probes: each one's interval, home location and vantages), reads every check from VictoriaMetrics over 5 minutes and the checks with a longer window again over theirs (grouped into windows of 5 minutes doubled, so a handful of queries, each naming only its checks), and drops every sample older than its own check's window, by VictoriaMetrics' clock, or from a location that no longer probes the check. So a check probed every 10 minutes is read over 23, while a check probed every minute that stops reporting is absent after 5, however long other checks' windows are. A check probed less often than once a month reads as absent between its probes. A longer window applies at once; a shorter one (the interval shortened) only after it has been asked for one whole shorter window, by when the check has been probed at its new interval; shortened again before that, it waits a whole window of the newest interval from then. A check inventory no longer lists (deleted, or its probe disabled) gets the shortest window at once. If inventory does not answer, the windows stand; nor are they shrunk by an answer listing no check while inventory lists no location (wiped or mid-restore), nor by a 404 once it has answered or in the first 5 minutes after the monitor starts (a rollout may be under way). Until inventory has answered once since the monitor started, absence counts as in an unsighted cycle (below). An inventory-svc older than alerting-svc has no such endpoint: every check then gets 5 minutes, as before, with a warning in the log, so upgrade inventory-svc with or before alerting-svc. If inventory still has not answered 5 minutes after the start, that is logged too. VictoriaMetrics reduces each series to its newest sample and the time it was written (last_over_time and tlast_over_time in one query, never from its rollup cache), so the answer is one value per series however long the window, though VictoriaMetrics still scans the window for the checks read over it. It resolves (with the "back online" all-clear) when the check passes again. A check whose whole location has gone unreachable is suppressed — that's location_unreachable's job — so a dark site pages once, not once per check. The alert is one per check, though, and the location it names is only a representative failing vantage point: when that location is dark but other, live locations probe the check, they judge it. Passing there resolves it as back online, the notice naming where it passed and that the named location is dark (Seen passing from 'dc2'; 'homelab' is dark and not reporting.); failing there keeps it — by the same majority as any alert, so a check that fired with 2 of 3 vantages failing is back online once one of the failing two is dark and only 1 of the 2 live ones fails; reported from no live location — including any single-vantage check there — it is left alone, as before, and the dark cycles never count as absence (nor is inventory asked about it). The alert's labels keep naming the dark location: a re-fire does not rewrite them, and the subject of a multi-vantage alert names no location anyway. A check that stops reporting from every live location — deleted, or its location offboarded (dropped from ingestion's location list, not merely dark) — is cleared after the same number of consecutive absent cycles with a [CLEARED] notice saying it is no longer reported from any live location, not "back online": nothing passed, and failing results coming back fire it again. A cycle in which the alert's location is dark does not count as absent, and restarts the count. So does a blind cycle: locations are live (their heartbeats arrive) but VictoriaMetrics holds no recent probe_success sample from any of them — none within the shortest window of a check probed from a live location, since a daily check's sample stays in its window for two days after the path broke — because its data was lost or restored, ingestion is not writing to it, or the metrics stream is down while heartbeats still flow. A check reported nowhere then may only be unseen, and so it may for its own window after the samples return, while that refills (each check reappears with its next probe): 5 minutes for a check at the default minute, two days for one probed daily (the agent currently probes a check with an interval over an hour about hourly). The same holds for each check's first window after the monitor starts (a restart may follow a blind spell) or the rule is re-enabled (nothing watched for one meanwhile), and for every check until inventory has told the monitor how they are probed. In such an unsighted cycle a missing check counts as absent only when inventory-svc (GET /resources/{id}) shows it gone: deleted (a 404, trusted only while inventory's /locations answers with some locations, not while it is empty or failing), its probe disabled, or neither its home location nor any of its vantages live — a check whose only location was offboarded. Any other alert, at a live or an offboarded location, is held, its count restarted, and so is every one when inventory cannot be asked (its /locations failing that cycle, no check is looked up; lookups run eight at a time with a 5-second timeout each, and an alert without a resource_id is never looked up). Nothing fires in a blind cycle (it has no failing sample either). One probe from any live location, for any check, makes a cycle sighted, so a single check whose samples stopped while others still report is cleared as before. A sample lingering from a dark or offboarded location does not count, and with no live location at all the dark and gone rules apply instead. The monitor logs a warning once per blind spell (a note after a start or a re-enable of the rule), only when it holds an alert (an install with no check reports nothing at all, which is not a fault), and notes when the samples return and when absence counts again. While ingestion's location list is empty (wiped or mid-restore) a missing location says nothing and its checks are left alone — except a location the offboard endpoint recorded (below). A check whose alert names an offboarded location but which passes from another live location is resolved as back online, and the notice names the location it was seen passing from.

Offboarding a location (POST /internal/locations/{id}/offboard) clears its other alerts itself, but not its check_down ones: one is per check, and the location it names is only a representative failing vantage point, so clearing it would close an alert for a check still failing elsewhere (and fire it again next cycle) or call a recovery an offboard. The endpoint instead records the location in offboarded_location (migration 0018_offboarded_location), and the check monitor reads it every cycle, before ingestion's list: a recorded location that is not live reads as gone even while the list is empty, as it is on a single-location hub after the offboard, so the rules above apply — failing from another live location keeps the alert, passing there resolves it with the note, reported from none clears it as silent. A recorded location reporting again counts as onboarded anew only once ingestion's state for it spans more than two stale windows (first_seen to last_seen, both ingestion's clock): messages published before the offboard can be delivered after the purge (a JetStream backlog) and re-create the state with a fresh last_seen, but such a backlog drains in one burst. Until then the location is dark (its samples are no evidence, its alerts are left alone unless another, live location sees the check pass, the note then saying it was offboarded) and, once silent again, gone. Onboarded anew, it is judged as any other, and the monitor drops its record then, or once no firing check_down alert names it and two stale windows have passed since the offboard (so a backlog delivered within them, even for a check not yet confirmed at the offboard, still meets the record). The rule assumes ingestion keeps up with its stream: one minutes behind at the offboard can deliver a backlog long enough to count as a re-onboard, which running the offboard again fixes. While the check_down rule is disabled the monitor does not run, so the endpoint clears its alerts like any other's. Persisted, unlike the heartbeat monitor's in-memory last-seen map, so a restart between the offboard and the clear does not leave the alert firing. The endpoint's cleared counts only the alerts it cleared. All of these are location ids: ingestion's list, the probes' and the alerts' location label and the offboard path name a location by the id its agent reports under (the alert text shows inventory's name for it). The failing reason (timeout, status_mismatch, tls_error, …) rides into the alert text and labels.

A fifth built-in, workload_degraded (migration 0007_workload_alerts), alerts on Kubernetes workload availability rather than individual pods. It's an ordinary threshold rule over k8s_workload_desired - k8s_workload_available (> 0) with a 10-minute for_secs window, so a healthy rolling update's brief dip self-resolves and only a stuck rollout fires. It ships disabled — enable it and attach channels — and, like every built-in, its enabled, severity, channels and description are editable; as a threshold rule its threshold and for_secs are too.

Another built-in, nats_storage_high — "NATS storage almost full" (migration 0012_nats_storage_alert) — watches the NATS account's JetStream budget. A full account refuses every agent publish and stops metric and log ingestion platform-wide, so it ships enabled (severity warning). It is a threshold rule over

100 * last_over_time(fleeger_nats_account_storage_used_bytes{…}[7d])
    / (  (last_over_time(fleeger_nats_account_storage_limit_bytes{…}[7d]) > 0)
     and (tlast_over_time(fleeger_nats_account_storage_limit_bytes{…}[7d])
            > tlast_over_time(fleeger_nats_account_storage_used_bytes{…}[7d]) - 15))

(each {…} is {location="meta",source="fleeger"}; migration 0014_nats_sampler_deadman gave it this form)

with comparator = gte, threshold = 80 (percent) and for_secs = 300, so a backlog that builds during an ingestion restart and drains right after does not page; the account fills over days, so five minutes of dwell costs nothing. An account without a storage limit of its own writes no limit gauge, so the query returns no series and the rule stays silent. ingestion-svc samples the gauges every 30s and writes them to VictoriaMetrics directly, not through NATS, so the signal survives a full account. The query reads each gauge's last sample in the past 7 days: while ingestion-svc writes, that is the current one, so a drained account resolves the alert at once; if ingestion-svc (or its sampler) stops, the value holds at the last known fill, so a firing alert keeps firing instead of resolving as if the account had been drained, and after 7 days of silence it resolves. A drain writes a fresh sample at once, so an account drained just before ingestion-svc stopped does not keep reading as full. To clear a held alert, bring ingestion-svc back (its next sample replaces the held one), or disable the rule (enabled = false), which retires it; re-enable it once ingestion-svc samples again. The limit counts only when the latest sample wrote it too, so an account whose limit is removed while ingestion-svc runs goes silent at the next sample, not a week later. A held value is the fill before ingestion-svc stopped: an account filling while it is down shows only in the dead-man below. A time-to-full projection was deliberately not built. The stream gauges (fleeger_nats_stream_bytes / _max_bytes) feed the two stream rules below. What to do when it fires: the NATS storage panel on the Services page (see NATS & JetStream capacity).

The ingest streams have caps of their own, and at a cap the broker deletes the oldest messages instead of refusing the publish, so data is lost without any error. Migration 0013_nats_stream_alerts seeds two more built-ins, both enabled, one alert per stream (every series keeps its stream label, and the evaluator fingerprints a series by its labels, so metrics and logs fire and resolve separately):

  • nats_stream_near_cap — "NATS stream near its cap", warning: the stream's fill against the nearest of its caps, in percent,

    100 * max by (location, source, service, stream) (
          label_replace(fleeger_nats_stream_bytes{…} / (fleeger_nats_stream_max_bytes{…} > 0), "cap", "bytes", "", "")
       or label_replace(fleeger_nats_stream_messages{…} / (fleeger_nats_stream_max_messages{…} > 0), "cap", "messages", "", "")
       or label_replace(fleeger_nats_stream_oldest_age_seconds{…} / (fleeger_nats_stream_max_age_seconds{…} > 0), "cap", "age", "", ""))
    

    (each {…} is {location="meta",source="fleeger"}), with comparator = gte, threshold = 90 and for_secs = 300. A cap the stream does not have has no series, so a stream without caps (the older Limits shape) is silent; the account-level nats_storage_high covers that one.

  • nats_stream_discarding — "NATS stream discarding unforwarded data", critical: increase(fleeger_nats_stream_discarded_unforwarded_messages_total{location="meta",source="fleeger"}[15m]) with comparator = gt, threshold = 0 and for_secs = 0, since every message counted is lost data. It resolves 15 minutes after the last loss. The counter counts only messages that left the stream without being acked, never acked ones nor an operator's drain; how, and its limits, are in NATS & JetStream capacity.

  • series_bound_exceeded — "Metric series bound exceeded: a location makes too many new series", warning, migration 0022_series_bound_alert : sum by (series_location) (increase(fleeger_ingestion_new_series_dropped_total{location="meta",source="fleeger"}[15m])) with comparator = gt, threshold = 0 and for_secs = 0, one alert per location (summed over ingestion-svc's replicas). It fires when ingestion-svc dropped points of new series of a location past its hourly bound ([ingestion] max_new_series_per_location_hour), about a minute after the first drop (the sampler writes the location's counter first as 0), and resolves 15 minutes after the last. The location's existing series and every other location are still written. Causes: an agent inventing resource ids, metric names or label values, a site larger than the bound, or an ingestion-svc restart (or an agent upgrade that renames its series), after which every series of a location counts as new. ingestion-svc's storage sampler writes the counter, so while the sampler fails (nats_sampler_stale) this alert is silent.

  • vm_series_limit_reached — "VictoriaMetrics series limit reached: samples dropped hub-wide", warning, migration 0022_series_bound_alert : sum(increase({__name__=~"vm_hourly_series_limit_rows_dropped_total|vm_daily_series_limit_rows_dropped_total"}[15m])) with comparator = gt, threshold = 0 and for_secs = 0, one alert for the hub. VictoriaMetrics stores its own metrics (-selfScrapeInterval=30s) and exposes each counter only while its limit (-storage.maxHourlySeries, -storage.maxDailySeries) is set; a limit of 0 is no series. Once a limit is reached every series not yet written in that hour or day of VictoriaMetrics' uptime loses its samples, existing series included, for every location and the hub's own: raise the limit above the hub-wide active series plus their churn, or find the source of the new series (Observability & self-monitoring).

Their threshold and for_secs are editable like the other threshold built-ins. If ingestion-svc stops writing the gauges, both go stale and resolve (holding the last sample would not help: a stream fills only while ingestion is not forwarding, which is after that sample); nats_sampler_stale below fires instead, and the loss of such an outage is counted when ingestion-svc restarts and fires the second rule then. (ingestion-svc writes each stream's counter as 0 in its first sample after starting and the count from the next, 30s later: VictoriaMetrics' increase takes a series' first sample as its baseline when it is large against the next change, so a counter starting at the outage's loss would read 0.) A hub without Loki has no logs stream: ingestion-svc deletes it, so neither rule has a logs series there (see NATS & JetStream capacity). nats_stream_near_cap does not say which cap is near; GET /api/nats/jetstream or the fleeger_nats_stream_* gauges against their _max_* show it.

All three rules read gauges only ingestion-svc writes, so they are blind while it is down, which is when the account and the streams fill. Migration 0014_nats_sampler_deadman seeds a dead-man's switch for them, in the shape of backup_stale: nats_sampler_stale — "NATS storage sampler silent", warning, enabled — the seconds since ingestion-svc last wrote the account usage gauge (the one gauge every sample writes, with or without a limit),

time() - max(tlast_over_time(fleeger_nats_account_storage_used_bytes{location="meta",source="fleeger"}[7d]) or vector(0))

with comparator = gt, threshold = 300 (seconds) and for_secs = 300, so it fires about 10–11 minutes after the last sample (VictoriaMetrics' default 30s search latency offset hides the newest half-minute of data, plus up to one evaluation tick) and resolves once a new sample is visible. A gauge missing from the whole 7-day window (never written, or silent for longer) reads as epoch 0 and keeps it firing: silence never means "all clear". The dwell also keeps a fresh install quiet while ingestion-svc writes its first sample. When it fires, check that ingestion-svc runs and that its log has no JetStream storage sampling failed warning (it cannot reach NATS or VictoriaMetrics, or JetStream is off for the account, which the alert below reports first). Its threshold and for_secs are editable like the other threshold built-ins.

NATS can be up, with JetStream on at the server, while JetStream is off for the SPOKES account every agent publishes into: when the account's grant (--js-disk-storage in deploy/nats/gen-auth.sh, loaded by the nats-push job) exceeds the server's max_file_store, the server refuses to enable it for that account, logs Error configuring jetstream for account … (10047) once and carries on. The NATS readiness probe (/healthz?js-enabled-only=true) checks only the server, so every pod stays Ready while every agent publish is refused. ingestion-svc's sampler reads the account as a SPOKES user, so it sees this: when the read is answered "JetStream not enabled for account" (10039), and for no other failure, it writes fleeger_nats_account_jetstream_enabled 0 to VictoriaMetrics (every successful read writes 1). Migration 0020_nats_account_jetstream_alert seeds nats_account_jetstream_disabled — "NATS: JetStream disabled for the agents' account — ingestion halted", critical, enabled — over

last_over_time(fleeger_nats_account_jetstream_enabled{location="meta",source="fleeger"}[7d])

with comparator = eq, threshold = 0 and for_secs = 0: the server's answer is its configured state, not a transient, so it fires about 30–90 s after the account loses JetStream (up to one sample, VictoriaMetrics' 30 s search latency offset and one evaluation tick) and resolves at the first sample that reads the account again. Like nats_storage_high it holds the last sample, so it keeps firing if ingestion-svc stops while the account is disabled. A hub whose ingestion-svc never wrote the gauge has no series, and the rule stays silent. A disabled account has no storage to sample, so nats_sampler_stale fires as well about ten minutes later; it stays the backstop for every other sampler failure. To fix it, raise max_file_store to at least the grant (on Helm, natsServer.persistence.size, of which it is 75%; the chart refuses a smaller volume) and restart NATS, or re-sign SPOKES with a smaller grant and re-run the push job. Its threshold and for_secs are editable like the other threshold built-ins.

A log batch Loki refuses in a way that may be its own (a 500 whose body names no transient cause, or a 5xx other than 502 to 504 and 520 to 524) 10 times, with Loki accepting other batches in between, is dropped by ingestion-svc; a Loki outage, a timeout, any 429, a 502 to 504, a 520 to 524 (a proxy in front of Loki) or a 500 naming an unavailable ring or ingester never drops one (see Observability & self-monitoring). Each drop counts in fleeger_ingestion_log_batches_dropped_total{reason="loki_refused"}, which the storage sampler writes to VictoriaMetrics from 0 after each start, one series per replica (pod). Migration 0021_log_batches_dropped_alert seeds log_batches_dropped — "Log batches dropped: Loki refused them", warning, enabled — over

sum without (pod) (increase(fleeger_ingestion_log_batches_dropped_total{location="meta",source="fleeger",reason="loki_refused"}[15m]))

with comparator = gt, threshold = 0 and for_secs = 0, the shape of nats_stream_discarding, one alert however many replicas drop: every batch counted is log lines Loki never stored, so it fires at the next evaluation after the sample that counts the drop, and resolves 15 minutes after the last one. The batch's subject, stream sequence and Loki's last answer are in ingestion-svc's ERROR line dropping log batch Loki refused. A hub whose ingestion-svc predates this has no series, and the rule stays silent. Its threshold and for_secs are editable like the other threshold built-ins.

A latency SLO needs no new machinery: it's an ordinary threshold rule over the probe latency, e.g. quantile_over_time(0.95, probe_duration_seconds{phase="total"}[10m]) with comparator = gt. (A one-click template for it lands with the status UI.)

Naming a cluster label

Alert summaries and subjects (threshold alerts and resource_unreachable) show a series' cluster label the way the UI does, never as the raw label:

  • cluster:<name> shows as the name, k8s-cluster:<uid> as the first 8 characters of the UID.
  • A label that names an inventory registration (registration:<id>, a cluster the agent could not read, or resource:<id>, a Proxmox node in no cluster) shows as that resource's name, else its address, from inventory-svc's GET /locations/{id}/resources for the alert's location. Only that location's Proxmox and Kubernetes registrations count. Without a name, a registration reads registration <first 8 characters of its id> and a node in no cluster its id.
  • A name that two such registrations of the location show, or that a cluster:<name> label seen there shows, gets the start of the id, as the UI's clusterNamer does: pve (registration abcd1234). Names are sanitized and cut to 64 characters before that.
  • A name is not unique across locations the way the raw label was, so a threshold alert on a series with a resource_id that is named with one also names its location: cpu on vm-100 (pve) at 'Site A'. (A series without one already lists its location label.) resource_unreachable always names the location.

Looking a name up never waits on inventory-svc. alerting-svc answers from a cache and reads a location's names in the background: on its first lookup, and every 5 minutes. For ids the last read lacked it reads once after 30 s, then after 1, 2, 4 and 5 minutes while no read finds any of them, however many are missing. One read per location runs at a time; it gives up after 5 s and keeps the names it had. A name read for the first time therefore shows from the next cycle on. The system monitor looks up every labeled resource each cycle, so names are normally read while a resource still reports, before any alert needs them. Every age a system-monitor cycle judges is taken when it reads the state lists, so a slow call later in the cycle does not age a subject past its window.

Only the text changes: the fingerprint, the alert's cluster label and mutes keep the raw label. The text is written when an alert is raised, so a firing alert keeps its summary and its recovery notice its subject; only its re-notifications use the current name.

The alert lifecycle (dedup, mute, re-notify)

All firing/resolving flows through one engine so the two evaluators behave identically:

  • Fingerprint — a stable key per (rule, subject) (rule id + the series' sorted labels, or rule id + location). Dedup is by fingerprint: a condition that stays breached keeps one alert row rather than spamming a new one each cycle. A threshold series' reason label is left out: it says what state the series is in, not which subject it is (probe_tls_cert_valid is 0{reason=…} while invalid and a bare 1 once fixed), so a changed or dropped reason is the same alert. The alert keeps the reason it was raised with until it resolves: in its stored labels, and in its text when the series is named by its labels (… on check_id c1, location homelab, reason self-signed; a resource_id name leaves it out). A later change of reason updates neither; the resolve notice names neither (see Resolve). An alert raised by an older release is keyed with its reason; its rule's next evaluation moves it to the new key in place, keeping its start and its notification stamp, so the upgrade neither pages nor clears it. Should two rows of one subject be firing under different reasons at the upgrade, the older is kept and the other retired without a notice.
  • Subject — a threshold alert's notification (and its summary in the alert list) names the series by its resource_id. A series without one is named by its own labels, compactly (stream metrics), leaving out __name__, source, service, and location when it is the hub's own meta; any other location is kept. A series left with nothing reads the hub when it is hub-wide (location="meta", e.g. nats_storage_high) and the aggregate otherwise (a label-less result such as backup_stale's). The name depends only on the series' own labels, never on what else the query returned. A cluster label is never shown raw; it reads as the UI shows it (see Naming a cluster label). Label values have control characters (newlines included) replaced by spaces and are cut to 64 characters, the whole name to 200, with an ellipsis. When the query computes its value (a ratio, increase(), max by, desired - available), so the series has no metric name, the text leads with the value: value 95.000 >= 90 on stream metrics, value 2.000 > 0 on k3s. The name is display text only; the fingerprint is the raw labels, less a threshold series' reason.
  • Fire — first breach creates a firing alert and notifies. While it stays firing it only re-notifies after renotify_secs, so a persistent root cause doesn't storm the channels.
  • Resolve — when the condition clears, the alert flips to resolved and a one-off "resolved" notice is sent. A threshold alert whose series the query still returns, back on the right side of the threshold, recovers ([RESOLVED] … recovered.). One whose series the query no longer returns is asked about, all of a rule's in one instant query per cycle (more only past 8 KiB of query): does VictoriaMetrics still hold a current sample of it? If so, the query left it out because it is healthy (up == 0, rate(x[5m]) > 5), and it recovers. If not (an offline agent, a deleted workload, a check set to accept invalid certificates, which then stops reporting probe_tls_cert_valid), nothing was measured, so it is cleared as [CLEARED] … is no longer reported … This is NOT a recovery, as the system monitor clears a subject that vanished. Both leave the alert resolved.
    • Which series is asked about. When the alert's series has a metric name (a plain selector, a comparison filter such as up == 0, last_over_time), exactly that series: that metric with exactly its labels, so a check that stops sending probe_tls_cert_valid but goes on sending probe_success is still cleared, not recovered. Its current sample is also judged by the rule's threshold: still breaching, the alert is held, not resolved, for one cycle. That covers a sample written after the rule's query ran, which the query returns next cycle, so the alert goes on firing as usual. Breaching again the next cycle, the raw sample is not what the query judges (a filter on something else, x > 90 and on(host) maintenance == 0, unless, topk, offset, a rollup that keeps the name such as max_over_time, or keep_metric_names): the series is present, so the alert recovers, a cycle late. The hold is kept in memory, so a restart in between can hold an alert one cycle more. A query that computes its value (rate, increase, arithmetic, max by) has no name to ask by; then any series carrying all of its labels counts, whatever its metric, and counts as healthy: no raw sample is the rule's value, so only presence is asked. That is weaker: another metric of the same subject keeps it current, so such a rule can recover a subject one of whose metrics stopped. Among the built-in rules cert_expiring and cert_invalid ask by their exact series; workload_degraded, nats_storage_high, nats_stream_near_cap and nats_stream_discarding by their labels; the dead-man rules (backup_stale, nats_sampler_stale) always return their one series. Any label name is asked about, escaped as MetricsQL needs (service.name, app.kubernetes.io/name, a control character as \uXXXX).
    • An alert with no labels to ask by (a label-less aggregate such as count(probe_success == 0) > 3 or sum(x > 5) > 0, or one whose only label is a reason) recovers when its query returns nothing, as every alert did in older releases: no subject of it can stop being reported, and the rule's own result is the whole measurement, so an empty result is the aggregate back under the threshold. Nothing is asked of VictoriaMetrics.
    • Current means within the lookback VictoriaMetrics applies to an instant query. For a rule that reads its metric at an instant (a plain selector, up == 0, max by, arithmetic), that is the window its own query just used: a longer one would find the last sample of a series that just stopped and call it a recovery, a shorter one would miss a healthy series pushed less often. A rule that names a range of its own is still asked at the lookback, not over its range, so the two windows differ. nats_stream_discarding (increase(…[15m])) and nats_storage_high (last_over_time(…[7d])) go on returning a series that stopped until it leaves their range (15 minutes, 7 days); only then is it asked about, and both ask by labels, so any series with the same labels reported within the lookback recovers it. A rule asking by exact series with such a range (last_over_time(x[1h]) > 5) over a metric pushed less often than the lookback finds its healthy series, which the query filtered out, not current, and clears the alert as no longer reported instead of recovering it.
    • If VictoriaMetrics does not answer, nothing is resolved on a guess: the alerts that query asked about stay firing and are asked about again next cycle (logged once, not every cycle); when a rule needs several queries, the alerts of those that answered are resolved on their answers. The warning clears once that rule's queries all answer (or nothing is left to ask), whatever other rules do.
    • The notice names the subject without the reason it was raised with (probe_tls_cert_valid on check_id c1, location homelab recovered.): that reason no longer holds. The stored alert keeps it.
  • Mute / maintenance windows — a rule can be snoozed (mute_until), and a mute row can snooze a rule, a resource, or a whole location for a window. A window can start now (a plain snooze) or be scheduled ahead (starts_in_minutes) and carry a human label — a maintenance window for planned work. The engine honors starts_at ≤ now < until. While muted the alert is still recorded (visible in the UI) but no notification is sent. A "resolved" notice is never muted — the all-clear always gets through. (Recurring windows are a later addition.) A muted resource is never part of a grouped resource_unreachable alert (above), so its mute holds even in a mass outage; a location mute silences that location's group. A resource mute can also carry a location and a cluster, compared with the alert's labels of those names: each one, when set, must match, so muting one cluster's vm-100 leaves another's alone. Left out, it matches the id in any location or cluster, which is how every mute stored before rc.3 keeps working. A set scope never matches an alert lacking that label: only Proxmox and Kubernetes resources reported by rc.3 agents carry cluster. Scope a mute only for those, whose ids can repeat across clusters or locations. Check ids are unique, and a check alert's location is the location id of one failing vantage, which can change from cycle to cycle, so a location-scoped check mute can stop matching. The target, and a scope when set, are compared exactly, so the API refuses one that is blank or has leading or trailing whitespace (a pasted id with a stray space) instead of listing a mute that silences nothing. Migration 0019 deletes such rows stored before, and its ASSERTs refuse new ones written directly (an import skips ASSERTs). A blank location or cluster scope, which only a write past the API could store, did match the resource's alerts lacking that label, so its deletion can unmute them. A mute is deleted by the alerting reaper a day after its window ends, whatever the retention policy says: GET /mutes lists live and upcoming windows only, so nothing reads an expired one.

Notification channels

Channels are pluggable behind a single Channel abstraction; this build ships ntfy, Discord, Matrix, and email (SMTP). Adding a provider is a new variant plus its send — the rest of the system is unaware.

Channel config (webhook URLs, SMTP passwords, Matrix tokens) is encrypted at rest with the same service-held AES-256-GCM key as the inventory credentials (FLEEGER_CREDENTIAL_KEY); only ciphertext is stored, and it's decrypted only at dispatch. Nor does it reach the logs: a failed delivery or channel test logs the provider and what went wrong (discord request failed: error sending request), never the request URL, since a webhook URL or ntfy topic is the credential itself and alerting-svc's logs are readable through /api/logs with resource.view. A rule lists the channel(s) it notifies.

A rule's or a channel's name is 1 to 100 characters (not bytes), on update as on create: an empty or longer new name is refused with a 400 before anything is stored. A name sent back unchanged is not checked, so a row saved with an over-long name before that check can still have its other fields edited.

A rule name, a rule's scope_location, a channel name, and every string in a channel's config (URLs, topic, room id, tokens, SMTP host, user, password and addresses), may not contain a control character (a line break, tab, NUL, DEL or a C1 code; non-ASCII is fine): saving one is refused with a 400 that names the field but never repeats the value, since it is usually a credential. A scope is checked as it is stored, after its padding is trimmed, so a trailing line break is still just trimmed off. A rule's query may span lines, so it keeps line breaks and tabs (\n, \r, \t) but refuses any other control character the same way. On an update only a new name, scope or query is checked, so a client that sends back the value it read is not refused over it. The rule description may still hold line breaks.

Rows saved before those checks are not changed or migrated; they are checked only when that name, scope, query or config is edited. Until then the runtime copes with them: an ntfy notification's title travels in an HTTP header, so a control character in it (a line break in a rule name, say) is sent as a space; non-ASCII is kept. A request the stored config makes impossible to build (a token with a line break) fails the channel test with a 400, and a rule's delivery reports the channel as unusable rather than retrying it every cycle.

Editing a channel

The API never returns a channel's config, not even masked: GET /channels and GET /channels/{id} give its id, name, kind, enabled, created_at and updated_at only. So an edit cannot send back what it read, and PUT /channels/{id} takes a partial config: an object whose fields are set over the stored config, a field sent as null is removed, and a field left out keeps its stored value. A rename, or a new topic or SMTP recipient, does not need the webhook URL, token or password again. The result must still be a valid config of the channel's kind (a required field removed is a 400, and nothing is written); the kind itself is fixed at create. Only the fields sent are checked for control characters; the destination of the merged config is vetted as on create. A merged config that does not parse is refused with the kind of error and the fields sent, never the parser's text, which could quote a decrypted stored value. A stored config that can no longer be decrypted (its key is gone) has nothing to keep, so the PUT must then carry a whole config, which replaces it; a partial one is refused with a 400 saying so.

A config key the channel's kind does not read (a misspelt pasword, say) is refused with a 400 naming the key, never its value, on create and in a PUT, rather than stored unused, as for a rule's fields. A PUT may still send such a key as null, which removes one stored before this check.

PUT /channels/{id} takes the optional expected_updated_at a rule's PUT does: the channel's updated_at as read, sent back exactly. A channel written since is refused with 409 STALE, before any other check, and nothing is written; a deleted one is a 404. Without the token (or with null) the PUT applies regardless, as a rule's does, but its write is still conditional: a channel written between the PUT's own read and its write is read again and the PUT made again over it, so a partial config is never merged over a stale copy. Every write of the channel's row spends the token: an edit, a rekey (which re-encrypts the config), and a rule create or update that routes a rule to the channel (which writes the channel's row, see above).

Alerts → Channels has an Edit button per channel. The form opens with the channel's name and kind (locked), and every config field empty, since none is shown back: an empty field keeps its current value, so only the fields typed into are sent, and a save with nothing changed closes the form without a write. An optional field (the ntfy server and token, the SMTP port) can be removed with its checkbox. On STALE the form reloads the channel, takes its new name unless you changed the name too (then it says so), keeps the config fields you typed in, and asks you to save again, as the rule form does.

The audit row of an edit (notification_channel.update) names the config's fields as merged and those the edit changes, never a value, and records the expected_updated_at sent. Those are reckoned against the handler's read; the completed row's result names the config fields the write actually changed, against the row it was applied to.

Where a channel may deliver

Whoever may create alert rules may create channels, and alerting-svc can reach the hub's own services, some of which have no authentication of their own (VictoriaMetrics' admin API deletes every series on one POST). So a channel may not point at the hub itself. Refused:

  • Loopback (127.0.0.0/8, ::1, and 0.0.0.0/::, which reach loopback on connect). From inside the hub, "this machine" is the hub.
  • The hub's own services, as alerting-svc's config names them: the [victoriametrics], [ingestion], [inventory], [nats] and [database] URLs. Refused by host name, and by every address that name resolves to, on any port, so http://victoriametrics:8428, the container's IP and another name for the same address are all refused.
  • The hosts, addresses and CIDR ranges in [alerting] hub_services, a list in the same form as the core agent's FLEEGER_HUB_SERVICES. Names are refused like the service URLs above, addresses and ranges by every address a destination resolves to. It is where Loki goes: Loki's delete API takes its parameters from the URL and has no key, so a webhook aimed at it would delete every log. The compose stack sets ["loki"]; the chart sets the host of loki.url plus alerting.hubServices. In Kubernetes a name resolves to its Service's ClusterIP, not to the pod IP behind it, so list the pod and service CIDRs too (alerting.hubServices in the chart) to refuse the pod IPs of VictoriaMetrics and Loki; leave out any range a notification target of yours lives in. An entry that is no host, address or range stops alerting-svc from starting.
  • Link-local and cloud-metadata hosts, as before.

VictoriaMetrics' delete API also needs a key now, which only ingestion-svc holds, so even a channel the guard lets through cannot delete series.

LAN and public hosts stay allowed: a self-hosted ntfy, Gotify or SMTP relay on the LAN works as before.

Services run directly on a host (no container network, the service URLs set to localhost or 127.0.0.1): the guard refuses loopback, but not the host's own LAN addresses, which it does not look up. A service that listens on all interfaces (VictoriaMetrics' default :8428, a SurrealDB or NATS bound to 0.0.0.0) is then reachable from alerting-svc at http://<host-LAN-IP>:<port>. List those addresses in [alerting] hub_services, for example hub_services = ["192.168.1.10", "fd00::10"], or bind the services to 127.0.0.1. A notification target on the same host is then refused as well, as on loopback. The compose and Kubernetes layouts are not affected: there VictoriaMetrics' port is not published and NATS listens on 127.0.0.1.

The check runs when a channel is created or its config is edited (a 400 naming the reason), and again on every delivery, so a channel stored before the check existed fails its test with the same 400 and, attached to a rule, is reported as unusable rather than retried. On delivery the address connected to is the one judged: HTTP host names resolve through the guard itself, and SMTP connects to the address the guard checked (with the certificate still verified against the host name), so a name that changes its DNS answer between check and connect gains nothing. Notification requests do not follow redirects: a webhook that answers with a 3xx is reported as that status, so point the channel at the final URL. An HTTP proxy from the environment (HTTPS_PROXY and friends) is the operator's and is not judged; when one is set, the proxy resolves the destination, and only the name check and the pre-send lookup apply.

Services alerting-svc's config does not name, such as Loki, are not covered by this check; keep them off networks alerting-svc can reach, or behind authentication.

Cycle and delivery tunables

These [alerting] keys are range-checked when the service starts, and an out-of-range value fails the pod immediately rather than starting in a broken state. Only start range-checks them: db migrate and db unlock still parse the whole config, so a value that is not a number at all (or does not fit one) blocks them too, but a merely out-of-range one does not.

Key Default Allowed Why the bound
eval_interval_secs 30 1 or more; while heartbeat_alert_ceiling_secs or resource_stale_ceiling_secs is on, at most half of (that ceiling − heartbeat_stale_secs) It is the pause between monitor cycles. At 0 there is none: all four monitors re-query VictoriaMetrics, SurrealDB, ingestion-svc and inventory-svc back to back, hundreds of times a second. The upper bound keeps two cycles between the default window and each retirement ceiling.
renotify_secs 3600 0–8000000000000 0 is allowed and re-notifies on every cycle. The window is added to a datetime in the database, whose datetimes end in the year 262142; a value a little past the maximum (~250,000 years, kept a few thousand years short of that edge) overflows it and makes the re-notify check error for every still-firing alert. Any value up to it already works as "effectively never".
check_fail_confirmations 2 any count whose wait, (count − 1) × eval_interval_secs (the interval taken as at most 315360000), is at most 315360000 s (~10 years) check_down fires only after that many consecutive failed cycles. A count whose wait passes ~10 years would never fire, so it is refused rather than silently disabling the rule; to stop check_down, disable the rule.
http_timeout_secs 30 1 or more A zero timeout is a deadline already passed, not "no timeout": every request slower than about a millisecond fails.
http_connect_timeout_secs 10 1 or more Same, for the TCP handshake: every webhook to an off-host endpoint would fail.
smtp_timeout_secs 120 1 or more Same, for the whole SMTP conversation: every email notification would fail.

A 0 timeout mostly works against a stub on loopback, which answers faster than the timer ticks — that is how it slips through a local test. There is no upper bound on the timeouts, nor on eval_interval_secs with both retirement ceilings off (see the table for the bound while either is on): nothing overflows, a long timeout only stretches a cycle whose upstream hangs, and an interval longer than the pod lives already means "one cycle, then none" — no cut-off would separate working values from broken ones. The advice to keep http_timeout_secs well under the cycle is advice, not a check: a cycle that outlasts the interval simply starts its pause later.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.