// Benutzung · Monitoring & Alerting
Alerting
alerting-svc turns the metrics pipeline into notifications. It observes and
notifies only — nothing in this service can trigger an action against
infrastructure. Rules and channels are user-editable from the dashboard (stored
in db=alerting); no redeploy is needed to change what fires or where it goes.
Two evaluators, one alert lifecycle
flowchart LR
subgraph alerting-svc
TE["Threshold evaluator\n(polls VictoriaMetrics)"]
HB["Heartbeat monitor\n(taps NATS metrics.>)"]
SM["System monitor\n(polls ingestion state →\nbuilt-in location/resource alerts)"]
EN["Alert engine\n(dedup · mute · dispatch)"]
DB[("SurrealDB\ndb=alerting\nrules · channels · alerts · mutes")]
end
VM[("VictoriaMetrics")]
NATS["NATS metrics.>"]
ING["ingestion-svc\n/state/locations · /state/resources"]
CH["Channels\nntfy · discord · matrix · email"]
VM --> TE --> EN
NATS --> HB --> EN
ING --> SM --> EN
EN --> DB
EN --> CH
DB -.rules/channels.-> TE
DB -.rules/channels.-> HB
DB -.built-in rules/channels.-> SMThreshold rules (kind = "threshold")
Each enabled threshold rule carries a PromQL query, a comparator
(gt/gte/lt/lte/eq, i.e. >/≥/</≤/=), and a threshold. On
every eval_interval_secs the evaluator runs the query against VictoriaMetrics
and, for each returned series, compares the instant value. A breach fires; a
series that no longer breaches resolves. This is how "CPU spike", "disk full",
"sensor over-temp", and a per-resource "host down" (up == 0 filtered to a
resource) are expressed — they're all just queries. A query may filter itself
like that: once the host is up again, up == 0 no longer returns it at all,
and the evaluator asks VictoriaMetrics whether up of that resource is still
reported to tell a recovery from a host that went away (see Resolve
below). Use eq (=) for discrete
metrics (a state, a count, up 0/1); it does an exact match, so it's a poor fit
for continuous ratios that rarely land on an exact value.
The cluster label. Proxmox and Kubernetes series carry a cluster
label, so a rule over them
alerts once per cluster's resource, unless its query aggregates the label away.
When an agent is upgraded, the series it wrote without the label still appears
in query results for a few minutes next to the labeled one; the evaluator
skips such an unlabeled series while its labeled twin (the same labels plus
cluster) is in the result, and carries a firing alert of the unlabeled series
over to the labeled one, with its breach clock, so no second alert fires. That
holds while the twin is in the result: when it drops out (it went away, or the
rule's query filters it out once it stops breaching) while the old series is
still within the lookback window, the old series is evaluated on its own again
and can fire, or recover, under the old identity once. A new agent never sends
a Proxmox or Kubernetes series unlabeled (it labels a cluster it could not
scope registration:<id>), so only an older agent's series is ever skipped,
and an unscoped cluster's series alert apart from a scoped cluster's.
Sustained-breach window (for_secs). A rule may require the breach to
persist before firing: for_secs (0 = fire on the first breaching cycle, the
default). The evaluator tracks per-(rule, series) when the breach started and only
fires once it has held for for_secs; a series that stops breaching resets the
clock. This is essential for workload-availability alerting (below) and useful
generally to ride out transient blips. The breach clock is kept in memory, so a
restart of alerting-svc starts it again: a window longer than the service usually
runs between restarts does not fire in practice. for_secs may be at most
~10 years, the same maximum as stale_secs, which refuses the
absurd values that could never be satisfied, and may not be negative (a
negative value stored before that reads as 0, and 0 clears it). As with
stale_secs, only a changing value is checked on an update. A threshold
rule is created only with its query, comparator and a finite threshold.
Alert on workloads, not individual pods. A kubectl rollout restart terminates the old ReplicaSet's pods and starts new ones by design, so
alerting on individual pod up produces guaranteed noise during any healthy
deploy. Instead, the spoke-agent emits per-workload replica gauges
(k8s_workload_desired / _ready / _available / _updated, labelled
namespace/workload/kind), and the built-in workload_degraded rule
alerts on k8s_workload_desired - k8s_workload_available > 0 held for its
for_secs window (default 10 min). A healthy rolling update stays within its
maxUnavailable budget and clears well inside the window, and the generous
window also rides out a slow first rollout (big image pull / slow startup)
where available lags desired for minutes; only a genuinely stuck/failed
rollout fires. Raise for_secs to exceed your slowest legitimate rollout. Per
pod up remains for visibility, but prefer the workload rule for alerting.
Heartbeat rules (kind = "heartbeat") — "location unreachable"
The heartbeat monitor taps NATS metrics.> directly and tracks the last time
each location produced any metric, on its own metrics.<location>.… or on the
bundled core agent's metrics._core.<location>.… — the latter only while that
location is local_collector (read from inventory-svc's
GET /internal/locations/local-collectors at start and every 10 s, a failed read
keeping the set; until a read succeeds no _core heartbeat is credited, with one
WARN, so while inventory-svc is down when alerting-svc starts, a local_collector
location only the core agent reports for can be raised unreachable although it is
up, until a read succeeds) — deliberately independent of ingestion-svc
(a separate failure domain: alerting still detects silence even if ingestion is
down). When a location's last-seen age exceeds the staleness window
(stale_secs, default heartbeat_stale_secs), a "location unreachable" alert
fires. Every agent sends a heartbeat for each location it serves every poll
interval, and at least every 30 s, on a timer of its own: a collection cycle
held up by targets that time out does not delay it.
stale_secs must be between 60 and ~10 years, the same maximum the
retirement ceilings take. The upper bound exists because the field is signed on the
record and every monitor compares it as an unsigned window: a negative value used
to wrap to a window no age could reach, silently disabling the rule and — via the
widest-window fold the ceiling is derived from — the retirement prune for every
other rule with it. 0 is refused too, in the other direction: it makes every age
stale, so the rule fires on everything. The floor is two heartbeat intervals: an agent beats at least every 30 s, so a location's age climbs to 30 s
between every two beats, and a window of that or less fires on a healthy location
every cycle; two intervals leave room for one late or lost beat. The rule form
says so, and the System alerts card refuses a shorter window before sending it. A
rule stored with a window under 60 before this floor existed is evaluated at
heartbeat_stale_secs instead, with a warning naming it, as below.
The [alerting] config key heartbeat_stale_secs — the installation-wide default
a rule falls back to — takes the same 60–315360000 range, checked at
startup: an out-of-range value fails the pod immediately rather than starting in a
broken state, with an error naming the key and the value. The floor is the same two
heartbeat intervals: a shorter default window fires every heartbeat rule that
sets no window of its own, and both built-in unreachable alerts, on healthy
subjects between two beats. Every shipped config sets 90. Unlike its two neighbours resource_stale_ceiling_secs and
heartbeat_alert_ceiling_secs, 0 is not an "off" switch here — there is no
way to disable the staleness window, because every heartbeat rule, both built-in
unreachable alerts and the location-liveness gate for check_down are all
defined in terms of it. A 0 there means every subject is unreachable on every
cycle — and, because check_down gates on the same window inverted (it only
evaluates checks at locations seen within it), that alert goes silent at the
same moment the others turn maximally noisy.
Note the floor is two heartbeat intervals, not one eval_interval_secs. A window shorter than the
evaluation cycle is legitimate: the monitors compare a wall-clock age (kept by the
NATS tap, and by ingestion-svc's last-seen state) that the cycle does not bound, so
a short window is merely observed more coarsely than it is written, not rendered
meaningless.
Three details of how the range is enforced, none of them obvious:
- On create, omitting the field uses
heartbeat_stale_secs. Out-of-range values are refused. - On update, leaving
stale_secsout of the request means "leave it as it is", and sendingnullclears the override, returning the rule toheartbeat_stale_secs. Sending a value equal to the one already stored is accepted even if it is out of range, deliberately: a client that reads a rule and sends every field back would otherwise have that rule's name, severity or channels blocked until its window was fixed too. A genuinely new out-of-range value is always refused. - At evaluation, a row holding an out-of-range window — from before the check
existed, or restored from an older backup — is evaluated at
heartbeat_stale_secs, with a warning naming the rule's id. Such a row keeps its stored value, andGET /ruleskeeps reporting it instale_secsuntil someone overwrites or clears the field.
Every rule answer of a kind that reads a window (heartbeat and the built-in
location_unreachable / resource_unreachable) also carries
effective_stale_secs, the window the monitors evaluate it at, resolved as above,
and default_stale_secs, the configured heartbeat_stale_secs. The alerts
page shows those, so an installation that changed heartbeat_stale_secs sees its
own default rather than the shipped 90. Other kinds answer null for both.
A heartbeat rule's window has a second, tighter bound when the retirement
ceiling heartbeat_alert_ceiling_secs is on (default 604800, 7 days): it must be
at most the ceiling minus two eval_interval_secs, so 604740 with the defaults.
A location silent for longer than the ceiling is retired: the monitor forgets it
and its alert stops. Retirement is switched off entirely while any enabled
heartbeat rule has a window at or past the ceiling, so one such rule used to stop
retirement for every heartbeat rule, with only a warning in the log. The two
cycles of margin leave room for the evaluation that fires the alert before the
location is retired. The config-level default gets the same margin: alerting-svc
refuses to start when heartbeat_alert_ceiling_secs is on and smaller than
heartbeat_stale_secs plus two eval_interval_secs, so the default
window is always accepted. With the ceiling set to 0 (retirement off) there is
no extra bound.
Each cycle, the heartbeat rules read the last-seen map before locations past the ceiling are dropped from it. A cycle delayed by a slow notifier can carry a location's age past both its window and the ceiling in one step; it still fires "location unreachable" in that cycle, and its alert is retired as no longer monitored once it has been firing for the ceiling. The bound is checked:
- when a heartbeat rule is created, enabled or not;
- when a heartbeat rule's
stale_secsis changed to a new value; - when a disabled heartbeat rule is switched on, against the window it already
holds, even if the request sends that window back unchanged. A rule over the
bound cannot be switched on until its window is lowered, which can be done in
the same request. A rule whose stored window is outside the
1–315360000range above can be switched on: that window is not used anyway.
Otherwise, as above, re-sending the window a rule already holds is not refused,
so an enabled rule stored with a wider window stays editable. Lowering such a
window in steps is refused until a step reaches the bound; clearing it with
null always works.
The refusal names the limit. A site that needs a longer window raises
heartbeat_alert_ceiling_secs instead. A rule stored with a wider window before
this check existed, or left over the bound by a later change to
heartbeat_alert_ceiling_secs, eval_interval_secs or heartbeat_stale_secs,
keeps running as before, but alerting-svc names it at startup: one warning per
heartbeat rule past the bound, enabled or not, with its id, window and the bound. If an enabled rule's window reaches the ceiling, retirement stays off
for every heartbeat rule and the log also warns on each cycle. If it is over the
bound but still below the ceiling, retirement stays on, and a location can be
dropped (logged) in the same cycle that rule first fires for it. Other rule kinds,
including the built-in resource_unreachable, are not affected.
description and scope_location follow the same update rule as stale_secs:
left out, they are untouched; null clears them — an empty description, and a
rule widened from one location back to all of them. No other field clears:
for query, comparator and threshold a null is the same as leaving the
field out, because a threshold rule cannot be evaluated without all three.
A blank or whitespace-only scope_location is no scope either, on create and
update alike, and a scope is stored trimmed; a rule stored before that
is read the same way.
Scoping a heartbeat rule to one location, or moving its scope to another,
retires its firing alerts for every other location, the way disabling a rule
retires all of them: each is cleared with a [CLEARED] notice saying the rule
no longer watches that location, never as "back online". The alert for
the location it still watches keeps firing. An alert of that kind left over
anyway (raised by a cycle that read the rule just before the change) is cleared
the same way on the next heartbeat cycle. Widening a rule back to every
location retires nothing.
Deleting a rule retires its firing alerts the same way, before the rule goes.
An alert is raised, or re-notified, only while its rule exists and is
enabled, checked in the statement that writes it, so a monitor cycle that read
the rule just before a delete or a disable raises nothing after it (a delete
disables the rule before it retires the alerts and removes the row). A firing
alert left anyway, whose rule no longer exists (left by a delete older than
that) or is disabled (a fire that committed alongside the disable, or a
retirement that failed), is resolved by a sweep once every
eval_interval_secs, the first time one interval after startup. A disabled
rule's alerts are swept only once the rule was last written more than that
interval ago, and at least ten minutes ago (at most a day), so a disable or
delete in progress retires them itself, with its notices, however slow they
are. A repeated DELETE /rules/{id} of a gone rule
resolves its alerts at once. The sweep clears without a notice and logs
a warning naming the alert and the rule.
The rule form sends stale_secs (and a threshold rule's for_secs) only when
you changed it: a stored value past 2^53 comes back from JSON rounded, and
echoing it would be a change the server refuses. It sends
description and scope_location as the form holds them; the token below is
what keeps an old form from writing back old values. An empty "Stale
after" box clears the override; a typed 0 is sent as 0 and refused (unless
the rule already holds 0, which the equal-value rule above accepts).
Stale rule edits
An update reads the rule, applies the request to it and writes the whole rule
back. A client that read the rule before someone else changed it would put
the old values back with a 200. To prevent that, PUT /rules/{id} takes an
optional expected_updated_at: the rule's updated_at as the client read it,
sent back exactly as read (an RFC 3339 string; a value that is not one is a
422 naming the field). If the rule has been written since, the update is
refused with 409 and code STALE, before any other check, and nothing is
written: read the rule again, reapply the edit, and send the new
updated_at. A rule that no longer exists is a 404, not a 409.
Every write of a rule through the API gives it a new updated_at, so every
such write spends the token: an edit, a toggle, a channel tick on a built-in's
card, and deleting one of the rule's channels (which removes the channel from
the rule). Migrations are the exception: they run under OPTION IMPORT, which
skips the VALUE that sets it, so a migration that seeds or changes a rule
must set updated_at itself for open forms to notice. The rule form sends the
token and, on STALE, reloads the rule, keeps the fields you edited, takes
the new values of the ones you did not, and lists the fields both sides
changed with the value the server now holds; save again to apply. The channel
ticks of the System alerts cards send the token too, so a card on a page older
than the rule is refused and the page reloads instead of putting back a channel
list (perhaps holding a deleted channel) it read before. The on/off toggles and
the cards' value boxes send narrow patches without a token, so a quick toggle
is not refused over an unrelated change.
Without expected_updated_at (or with null) the update applies as before,
with whatever the request carries, so a script that sends back a rule it read
earlier is not protected against having read an old one. The write itself is
always conditional, though: a rule written between the update's own read and
its write is read again and the update made again. A channel update
(PUT /channels/{id}) takes the same expected_updated_at token and is
conditional the same way (see Editing a
channel); a channel delete (DELETE /channels/{id})
that loses a write conflict (to a concurrent write of the channel, or of a
rule being routed to it) is run again too.
Rule creates and updates that route a rule to a channel, and channel updates
and deletes, all write the channel's row (see below), so they conflict with
each other as a matter of course — many rules provisioned at once onto one
shared channel, say. Such a write is retried up to 16 times, after a random
pause in a window that doubles each time (10 ms up to 250 ms, about 3 s in
all at most), which absorbs parallel provisioning. Only contention that
outlasts that is answered 409 CONFLICT ("another write touched … at the
same time; try again").
A channel that an update adds to a rule (one the rule does not already hold) must exist, or the update is refused with a 400 naming it and nothing is written. The check is part of the conditional write, and the write touches the added channel's row, so a channel delete committing at the same time makes one of the two run again: the update then finds the channel gone, or the delete detaches it from the rule as well. A channel delete detaches the channel from every rule and deletes it in one transaction. So no update, with or without a token, routes a rule to a deleted channel. A channel id the rule already holds is not checked, so a rule stored with one from before is still editable by a client that sends its channels back.
Creating a rule (POST /rules) checks every channel it names the same way: one that does not exist — never did, or was deleted since the client
listed it — refuses the create with a 400 naming it, and no rule is stored.
The check and the create are one transaction that touches each channel's row,
so a channel delete committing at the same time makes one of the two run
again: the create then finds the channel gone, or the delete detaches it from
the new rule. Creates routed to the same channel conflict with each other the
same way, and are retried as above.
Migration 0016 gives every rule without an updated_at one: the built-ins
were seeded in import mode, which skips the VALUE time::now() that sets it.
A restore does not undo that: it replaces the alerting database with the
backup's, migration state included, so a backup from before 0016 has
no record of it, and the migrate step runs it again before alerting-svc
starts. If a
rule ever shows updated_at: null anyway, it refuses any token and applies a
request without one until its next write; this covers it at once (namespace
fleeger, database alerting):
UPDATE alert_rule SET updated_at = time::now() WHERE updated_at = NONE;
Each kind of rule accepts only the condition fields it is evaluated from, and
the API refuses any other with a 400 naming the field: a threshold rule
reads query, comparator, threshold and for_secs; a heartbeat rule
stale_secs and scope_location; the built-in location_unreachable and
resource_unreachable only stale_secs; check_down none of them. Before, such
a field was stored and never read, so a request that meant something else
succeeded without effect. The built-in threshold rules (backup_stale,
cert_expiring, cert_invalid, workload_degraded, nats_storage_high,
nats_stream_near_cap, nats_stream_discarding, nats_sampler_stale,
nats_account_jetstream_disabled, log_batches_dropped,
series_bound_exceeded, vm_series_limit_reached) count as threshold rules here, so they refuse stale_secs and scope_location. Their threshold and
for_secs are editable; their query and comparator are what makes each
that built-in, so a changing one is refused with a 400 as well. No
built-in can be renamed: a new name gets a 400 too. A null, a blank
scope_location and a for_secs of 0 (what an omitted key means) set nothing
and are accepted; on an update they clear a leftover value. Otherwise only a
changing value is refused, against the kind the rule was created with, so a
client that sends back a rule as it read it is not refused over a field the row
already holds from before.
To stop a rule firing, set enabled = false; a window long enough that it never
fires is not a supported way to park one.
This is distinct from a per-resource "host down" alert: host down is a threshold rule about one resource inside a location that is still reporting; location unreachable means the whole location (its spoke-agent) has gone silent, so we can no longer say anything about the resources within it. A scoped rule watches one named location; an unscoped rule watches every location that has ever reported.
Built-in "system" alerts (always on, no rule to write)
Two rules are seeded (migration 0002_system_alerts) and always present:
location_unreachable and resource_unreachable. They're marked builtin — the
API refuses to delete them, refuses a new name, and refuses condition
fields other than stale_secs; only
their channels, enabled, severity, description, stale_secs, and mute
state are user-editable. (A null empties a built-in's description like any
other rule's; the seeded text is not restored automatically, but can be sent
again.) A user picks where each notifies (they ship with no channels, so
they track silently until a channel is chosen). Each built-in's card in the
System alerts section of Alerts → Rules sets its channels, enabled,
mute and the condition values its kind reads: stale_secs for these two,
threshold and for_secs for the built-in threshold rules.
A third worker, the system monitor, drives them from ingestion-svc's
persisted per-location and per-resource last-seen state (/state/locations,
/state/resources) rather than an in-memory NATS tally. Because that state is
persisted, an entity that was already silent when alerting-svc started still
alerts — there's no startup blind spot (unlike the in-memory heartbeat monitor,
which only sees a location go silent after it has observed traffic from it).
location_unreachable— a location whose newest metric is older than the stale window. Overlaps conceptually with an unscoped heartbeat rule, but it's the zero-config, always-on version.resource_unreachable— a single resource whose metrics went stale while its location is still fresh (from 20 of one location firing at once, one grouped alert for that location; see below). A full-location outage is left to the location alert (so one dead site doesn't emit one alert per resource under it). "Fresh" is judged bylocation_unreachable's own window (itsstale_secs, elseheartbeat_stale_secs), the one its alert fires on, so a site is either covered by the location alert or by its resources' alerts, never by both or neither. The window applies whilelocation_unreachableis disabled too. The check monitor's dark locations (below) use it as well, but never later thanheartbeat_stale_secs: a silent site's probe samples leave its lookback window, so a raised window does not keep its checks judged. Crucially, the spoke-agent reports a stopped guest asup 0on every cycle — it keeps reporting — so deliberately powered-off guests never go stale and never fire here. What does fire: a resource that genuinely stops being reported at all (an iDRAC host offline, a Proxmox node down, a vanished k8s node/pod).A staleness ceiling (
resource_stale_ceiling_secs, default 30 min) bounds this: a resource silent longer than the ceiling is treated as decommissioned (a churned k8s pod whose name changed, a deleted VM) rather than a fresh outage, so it stops firing. Without the ceiling a resource that stops reporting alerts until ingestion forgets it, and by default ingestion never does: itsresource_staterow stays until something deletes it, so every deleted/renamed resource would alert forever. Location unreachable has no ceiling: a whole site down for days is still down.In normal operation three things delete a
resource_staterow (a wiped or restored ingestion database resets them all): deleting the registered resource or a rollback removing it (below), offboarding its location (which clears that location's alerts itself, as[CLEARED] … its location was offboarded; see Offboarding an agent and the check monitor below), and ingestion's retention reaper (services/ingestion-svc/src/reaper.rs). The reaper runs only once the retention policy is enabled (it ships disabled, with akeep_foreverdefault), and deletes a row whoselast_seenis older than the resource's resolved window (per kind, then per location, then the default;keep_forevernever), at the first sweep after the window passes (sweep_interval_secs, default 3600 s, at least 60), purging the resource's VictoriaMetrics series and Loki streams with it. The shortest window is 300 s (LIVENESS_FLOOR_SECSincrates/fleeger-shared/src/retention.rs:delete_immediately, or a shorterkeep_for), well under the default 1800 s ceiling. Whichever comes first, the ceiling or the sweep that reaps the row, ends a silent resource's alert; with neither (the ceiling disabled with0or not above the rule'sstale_secs, and the reaper off or the windowkeep_forever) it keeps firing:- The ceiling first (always while the reaper is off or the window is
keep_forever): it retires the alert with its[CLEARED] … has not reported for longer than the configured ceilingnotice; reaping the row later changes nothing for alerting. - The reaper first: the resource drops out of ingestion's list and the
next cycle clears its firing alert as vanished,
[CLEARED] … is no longer reported by ingestion … This is NOT a recovery, whether its location is live or dark (a dark site's suppressed alert included). A resource whose rule window (theresource_unreachablerule'sstale_secs) is longer than its retention window can be reaped before it ever fires.
Either way, once its row is gone nothing fires for that resource again while it stays silent, and when it reports again ingestion records it anew: a resource whose alert was already cleared gets no "back online".
location_stateis not reaped, solocation_unreachableis unaffected.Deleting a registered resource in Fleeger (a Proxmox node, a Kubernetes cluster, an iDRAC host, a check), or a full checkpoint rollback removing one , does not wait for the ceiling: the gateway has ingestion drop the state of everything collected through it, its discovered guests, nodes and pods included, so none of them fires, and an alert already firing for one is cleared as vanished ("no longer monitored"), never as recovered. That needs an agent that stamps its messages with the registration's inventory id (rc.3) and ingestion-svc reachable during the delete or rollback; otherwise the rows age out through the ceiling as described above. A resource a rollback recreates under its old id is no longer held back by its delete's 5-minute tombstone: a check reports again on its next run, while a recreated Proxmox node, Kubernetes cluster or iDRAC host comes back without its credential and reports nothing until one is attached again. Deleting a discovered resource on its own (a VM, a pod) clears nothing: the agent keeps collecting it through its registered parent.
One alert per resource (unless it is grouped, see below): location, resource id and, for a Proxmox or Kubernetes resource, its
cluster, so two clusters'vm-100alert apart; the alert text (see Naming a cluster label) and labels name the cluster. A resource's alert raised while an older agent reported it without a cluster is carried over to the labeled resource once the upgraded agent reports it (ingestion then marks the unlabeled row superseded and leaves it out of the state list), rather than cleared as no longer monitored and raised again. With two clusters in a location this bridge cannot tell which cluster the shared alert belonged to: it moves to whichever cluster's resource is checked first.From 20 resources of one location firing at once, one grouped alert covers them (
RESOURCE_GROUP_THRESHOLDinservices/alerting-svc/src/system_monitor.rs). Counted are the resources that would newly fire this cycle: stale at a live location and within the ceiling, exactly as for one alone, without an alert of their own and not muted on their own (below). The agent chooses its resource ids, so one that invents ids and stops reporting them while its location stays live raised one alert and one notification for every invented resource, up toingestion.max_resources_per_location(10,000 by default) at once; a real mass outage at a live site paged as often. Under 20, each resource alerts on its own exactly as before, and a single outage is unchanged. The count is per cycle: an agent that lets its invented ids go silent a few at a time (up to 19 per cycle) still pages for each of them, as before.- Identity: one grouped alert per location under the
resource_unreachablerule, fingerprintresource_unreachable#group|<location>, which no resource's fingerprint can equal. Its labels arelocation(so a location mute silences it and offboarding the location clears it) andresources, the count; it has noresource_id. - Muted resources are never grouped: a resource with a live mute of
its own (matched as the engine matches it,
locationandclusterscope included) keeps its own alert, muted and so silent, and does not count towards the 20. A member muted while the group fires leaves it the same way. A resource mute therefore keeps its promise however many resources are down around it. - Text: the count, the first 10 resources in lexicographic order of
their text and how many more, and how long the most recent of them has
been silent, e.g.
25 resources at 'Site A' unreachable — no metrics for 2m or longer: 'vm-00' ('proxmox'), … and 15 more. Each id, source and cluster name is cut to 64 characters (…) and quoted with\and'escaped, so an id cannot pose as several entries. Its subject isthe group of resources at '<location>'; the text and the subject are rewritten each cycle they change (members, the location's name, or the silence growing by the minute), without paging. - Notifications: one when the group fires, then re-notified every
renotify_secslike any alert, with the group as it is that cycle (the alert's text is brought up to date before it is re-notified). A resource joining or leaving the group pages nobody. - Alerts already firing when 20 others go down keep their own alert (re-notified, and recovered or retired, as before) and are not counted; the group's text says how many more alert on their own. So forming a group neither clears an alert nor pages twice for one resource (but see Rolling update below: while two pods run, a page can come both ways). Nineteen alerting on their own and one more going down form no group.
- Sticky membership: a member leaves the group only by reporting again, going silent past the ceiling, leaving the state list or being muted. No member ever gets an alert of its own because the group shrank, so a real outage's recovery tail, or invented ids aging out one by one, pages nobody and nothing flaps around 20. A resource that goes down while the group fires joins it only while the group, counting it, has at least 20 members (20 or more going down at once always join); otherwise it is a new outage and alerts and pages on its own, so a single failure next to a shrunken group is not hidden in it. After an alerting-svc restart the monitor does not know the members, and holds every resource of the location that goes down without an alert of its own in the firing group.
- Clearing: only once the group has no member left. The notice is
[RESOLVED] … is back onlineonly when every resource it covered while it fired reports again;[CLEARED] … has not reported for longer than the configured ceilingwhen every one went pastresource_stale_ceiling_secs, and[CLEARED] … no longer reported by ingestionwhen every one left the state list, as each one's own alert would have been. Otherwise it is[CLEARED] … has no resource left that is unreachable without an alert of its own … This is NOT a recovery, saying how many were back online, past the ceiling, no longer reported, or are still unreachable but muted (and alert on their own). The monitor keeps what the group covered in memory: after an alerting-svc restart a group is never called a recovery, and its notice says the monitor knows only what it covered since the restart, or nothing of them. A group cleared by someone else while it fired (resolved by hand, or by the other pod during a rolling update) and formed again is a new group: its notice counts only what the new one covered. A clear by hand is not a way to silence a mass outage: the next cycle, the members still down form a new group if they are 20 or more, and otherwise each alerts and pages on its own. Mute the location or its resources instead. - Rolling update: for the moment two alerting-svc pods run, both run the monitor. Firing and clearing are deduplicated by fingerprint, so the same alert is not raised twice, but the two can disagree: the new pod starts without the group's members and treats it as after a restart, so one pod may fire a resource on its own that the other holds in the group, or clear the group with the unknown-members notice; a page can come both ways for that moment.
- Ceiling and the backstop: a resource past the ceiling neither fires nor counts, as before; one that leaves the group that way gets no notice of its own (the group's text drops it, and its clear notice counts it). A resource let through by the backstop counts like any stale one.
- Dark site: a location gone dark suppresses the group like its resources' alerts: a firing group is left exactly as it is, members and all, and none is raised there. Members going past the ceiling while the site is dark do not clear it; once the site is back they are judged as above. A location whose members all left the state list has its group cleared as no longer reported.
The monitor samples each resource's age once per cycle and tests the ceiling before the window, so a slow cycle could carry an age from under
heartbeat_stale_secsto past the ceiling between two samples, and the resource was retired without ever firing. alerting-svc therefore refuses to start whenresource_stale_ceiling_secsis on and smaller thanheartbeat_stale_secsplus twoeval_interval_secs, the same margin the heartbeat ceiling gets. The margin covers ordinary cycles, but not a cycle held up for longer than two intervals (a hung upstream holds one for the wholehttp_timeout_secs) nor aresource_unreachablewindow set just under the ceiling with the rule's ownstale_secs. For those, the monitor remembers when it last sampled the resource list (on the monotonic clock, so an NTP step does not move it). A resource past the ceiling that was not yet stale at that sample, judged by the rule's current window, has never been reported, so it is judged once more: if its location is up it fires, and the next cycle retires it as no longer monitored as usual. If its location is dark it is suppressed, like any stale resource there, and kept waiting, since the same hang that skipped its sample can make every location look dark; it fires once its location is up again, and leaves the queue if it reports again or drops out of ingestion's list. At most 10,000 resources wait; past that, one is retired unseen as before, with a warning. A resource already stale at the previous sample, or long gone, is retired without firing as before, and so is every resource on the first cycle after alerting-svc starts, after the rule is re-enabled, or after cycles failed for longer than the ceiling: then there is no usable previous sample, and every resource that went past the ceiling in that time would otherwise fire once and clear.- The ceiling first (always while the reaper is off or the window is
The system monitor reuses the same engine (dedup, mute, re-notify, dispatch) as
the two evaluators, so built-in alerts behave identically to user rules. Metrics
are keyed by a location's stable id, so the monitor also pulls inventory-svc's
id → name map each cycle and writes the friendly name into the alert text
(falling back to the id if inventory is unreachable).
A third built-in, check_down, is seeded by migration 0004_check_down and
covers uptime / synthetic monitoring (see
Uptime / synthetic monitoring). A fourth worker, the check
monitor, drives it from the probe metrics: each cycle it reads
probe_success{source="synthetic"} from VictoriaMetrics and, for every failing
check, fires — but only after a few consecutive failed cycles
(check_fail_confirmations, default 2) so a single blip never pages. A count
whose wait ((count - 1) x eval_interval_secs) passes ~10 years is refused at
startup, since check_down would never fire. Each
vantage point (location + check) votes once, with its newest sample in the
lookback window: the probe's reason label puts a passing and a failing probe in
different series, and the older one must not count as a second vantage.
Each check has its own window: twice its probe interval plus
3 minutes, never less than 5 minutes nor more than 64 days. That leaves room
for one lost probe and for the lateness a check at the default minute always
had: the agent probes a due check in its next collection cycle, so a location
polling less often than every 3 minutes (poll_interval_secs) can still miss
it. Each cycle the monitor asks inventory-svc how the enabled checks are
probed (GET /internal/checks/probes: each one's interval, home location and
vantages), reads every check from VictoriaMetrics over 5 minutes and the
checks with a longer window again over theirs (grouped into windows of
5 minutes doubled, so a handful of queries, each naming only its checks), and
drops every sample older than its own check's window, by VictoriaMetrics'
clock, or from a location that no longer probes the check.
So a check probed every 10 minutes is read over 23, while a check probed every
minute that stops reporting is absent after 5, however long other checks'
windows are. A check probed less often than once a month reads as absent
between its probes. A longer window applies at once; a shorter one (the
interval shortened) only after it has been asked for one whole shorter window,
by when the check has been probed at its new interval; shortened again before
that, it waits a whole window of the newest interval from then. A check inventory no longer lists
(deleted, or its probe disabled) gets the shortest window at once. If
inventory does not answer, the windows stand; nor are they shrunk by an answer
listing no check while inventory lists no location (wiped or mid-restore), nor
by a 404 once it has answered or in the first 5 minutes after the monitor
starts (a rollout may be under way).
Until inventory has answered once since the monitor started, absence counts as
in an unsighted cycle (below). An inventory-svc older than alerting-svc
has no such endpoint: every check then gets 5 minutes, as before, with a
warning in the log, so upgrade inventory-svc with or before alerting-svc.
If inventory still has not answered 5 minutes after the start, that is logged
too. VictoriaMetrics reduces each series to its newest sample and the time it
was written (last_over_time and tlast_over_time in one query, never from
its rollup cache), so the answer is one value per series however long the
window, though VictoriaMetrics still scans the window for the checks read over
it. It resolves
(with the "back online" all-clear) when the check passes again. A check whose
whole location has gone unreachable is suppressed — that's
location_unreachable's job — so a dark site pages once, not once per check.
The alert is one per check, though, and the location it names is only a
representative failing vantage point: when that location is dark but other,
live locations probe the check, they judge it. Passing there resolves it
as back online, the notice naming where it passed and that the named location
is dark (Seen passing from 'dc2'; 'homelab' is dark and not reporting.);
failing there keeps it — by the same majority as any alert, so a check that
fired with 2 of 3 vantages failing is back online once one of the failing two
is dark and only 1 of the 2 live ones fails; reported from no live location —
including any single-vantage check there — it is left alone, as before, and the
dark cycles never count as absence (nor is inventory asked about it). The
alert's labels keep naming the dark location: a re-fire does not rewrite them,
and the subject of a multi-vantage alert names no location anyway. A check that
stops reporting from every live location — deleted, or its location offboarded
(dropped from ingestion's location list, not merely dark) — is cleared after
the same number of consecutive absent cycles with a [CLEARED] notice saying
it is no longer reported from any live location, not "back online": nothing
passed, and failing results coming back fire it again. A cycle in which
the alert's location is dark does not count as absent, and restarts the count.
So does a blind cycle: locations are live (their heartbeats arrive)
but VictoriaMetrics holds no recent probe_success sample from any of them —
none within the shortest window of a check probed from a live location, since a
daily check's sample stays in its window for two days after the path broke —
because its data was lost or restored, ingestion is not writing to it, or the
metrics stream is down while heartbeats still flow. A check reported nowhere
then may only be unseen, and so it may for its own window after the samples
return, while that refills (each check reappears with its next probe): 5
minutes for a check at the default minute, two days for one probed daily
(the agent currently probes a check with an interval over an hour about
hourly). The
same holds for each check's first window after the monitor starts (a restart
may follow a blind spell) or the rule is re-enabled (nothing watched for one
meanwhile), and for every check until inventory has told the
monitor how they are probed. In such an unsighted cycle a missing check
counts as absent only when inventory-svc (GET /resources/{id}) shows it
gone: deleted (a 404, trusted only while inventory's /locations answers with
some locations, not while it is empty or failing), its probe disabled, or
neither its home location nor any of its vantages live — a check whose only
location was offboarded. Any other alert, at a live or an offboarded location,
is held, its count restarted, and so is every one when inventory cannot be
asked (its /locations failing that cycle, no check is looked up; lookups run
eight at a time with a 5-second timeout each, and an alert without a
resource_id is never looked up). Nothing fires in a blind cycle (it has no
failing sample either). One probe from any live location, for any check, makes
a cycle sighted, so a single check whose samples stopped while others still
report is cleared as before. A sample lingering from a dark or offboarded
location does not count, and with no live location at all the dark and gone
rules apply instead. The monitor logs a warning once per blind spell (a note
after a start or a re-enable of the rule), only when it holds an alert (an install with no check reports
nothing at all, which is not a fault), and notes when the samples return and
when absence counts again. While ingestion's location list is empty (wiped or
mid-restore) a missing location says nothing and its checks are left alone —
except a location the offboard endpoint recorded (below). A check whose alert
names an offboarded location but which passes from another live location is
resolved as back online, and the notice names the location it was seen passing
from.
Offboarding a location (POST /internal/locations/{id}/offboard)
clears its other alerts itself, but not its check_down ones: one is
per check, and the location it names is only a representative failing vantage
point, so clearing it would close an alert for a check still failing
elsewhere (and fire it again next cycle) or call a recovery an offboard. The
endpoint instead records the location in offboarded_location (migration
0018_offboarded_location), and the check monitor reads it every cycle,
before ingestion's list: a recorded location that is not live reads as gone
even while the list is empty, as it is on a single-location hub after the
offboard, so the rules above apply — failing from another live location keeps
the alert, passing there resolves it with the note, reported from none clears
it as silent. A recorded location reporting again counts as onboarded anew
only once ingestion's state for it spans more than two stale windows
(first_seen to last_seen, both ingestion's clock): messages published
before the offboard can be delivered after the purge (a JetStream backlog)
and re-create the state with a fresh last_seen, but such a backlog drains in
one burst. Until then the location is dark (its samples are no evidence, its
alerts are left alone unless another, live location sees the check pass, the
note then saying it was offboarded) and, once silent again, gone. Onboarded anew, it is
judged as any other, and the monitor drops its record then, or once no firing
check_down alert names it and two stale windows have passed since the
offboard (so a backlog delivered within them, even for a check not yet
confirmed at the offboard, still meets the record). The rule assumes ingestion
keeps up with its stream: one minutes behind at the offboard can deliver a
backlog long enough to count as a re-onboard, which running the offboard again
fixes. While the check_down rule is disabled the monitor does not run, so the
endpoint clears its alerts like any other's. Persisted, unlike the heartbeat
monitor's in-memory last-seen map, so a restart between the offboard and the
clear does not leave the alert firing. The endpoint's cleared counts only the
alerts it cleared. All of these are location ids: ingestion's list, the
probes' and the alerts' location label and the offboard path name a
location by the id its agent reports under (the alert text shows inventory's
name for it). The
failing reason (timeout, status_mismatch, tls_error, …) rides into the
alert text and labels.
A fifth built-in, workload_degraded (migration 0007_workload_alerts),
alerts on Kubernetes workload availability rather than individual pods. It's an ordinary threshold rule over
k8s_workload_desired - k8s_workload_available (> 0) with a 10-minute for_secs
window, so a healthy rolling update's brief dip self-resolves and only a stuck
rollout fires. It ships disabled — enable it and attach channels — and, like
every built-in, its enabled, severity, channels and description are
editable; as a threshold rule its threshold and for_secs are too.
Another built-in, nats_storage_high — "NATS storage almost full" (migration
0012_nats_storage_alert) — watches the NATS account's JetStream
budget. A full account refuses every agent publish and stops metric and log
ingestion platform-wide, so it ships enabled (severity warning). It is a
threshold rule over
100 * last_over_time(fleeger_nats_account_storage_used_bytes{…}[7d])
/ ( (last_over_time(fleeger_nats_account_storage_limit_bytes{…}[7d]) > 0)
and (tlast_over_time(fleeger_nats_account_storage_limit_bytes{…}[7d])
> tlast_over_time(fleeger_nats_account_storage_used_bytes{…}[7d]) - 15))
(each {…} is {location="meta",source="fleeger"}; migration
0014_nats_sampler_deadman gave it this form)
with comparator = gte, threshold = 80 (percent) and for_secs = 300, so a
backlog that builds during an ingestion restart and drains right after does not
page; the account fills over days, so five minutes of dwell costs nothing. An
account without a storage limit of its own writes no limit gauge, so the query
returns no series and the rule stays silent. ingestion-svc samples the gauges
every 30s and writes them to VictoriaMetrics directly, not through NATS, so the
signal survives a full account. The query reads each gauge's last sample in the
past 7 days: while ingestion-svc writes, that is the current one, so a drained
account resolves the alert at once; if ingestion-svc (or its sampler) stops, the
value holds at the last known fill, so a firing alert keeps firing instead of
resolving as if the account had been drained, and after 7 days of silence it
resolves. A drain writes a fresh sample at once, so an account drained just
before ingestion-svc stopped does not keep reading as full. To clear a held
alert, bring ingestion-svc back (its next sample replaces the held one), or
disable the rule (enabled = false), which retires it; re-enable it once
ingestion-svc samples again. The limit counts only when the latest sample wrote
it too, so an account whose limit is removed while ingestion-svc runs goes
silent at the next sample, not a week later. A held value is the fill before
ingestion-svc stopped: an account filling while it is down shows only in the
dead-man below. A time-to-full projection was deliberately not built. The stream
gauges (fleeger_nats_stream_bytes / _max_bytes) feed the two stream rules
below. What to do when it fires:
the NATS storage panel on the Services page (see
NATS & JetStream capacity).
The ingest streams have caps of their own, and at a cap the broker deletes the
oldest messages instead of refusing the publish, so data is lost without any
error. Migration 0013_nats_stream_alerts seeds two more built-ins, both
enabled, one alert per stream (every series keeps its stream label, and
the evaluator fingerprints a series by its labels, so metrics and logs fire
and resolve separately):
nats_stream_near_cap— "NATS stream near its cap",warning: the stream's fill against the nearest of its caps, in percent,100 * max by (location, source, service, stream) ( label_replace(fleeger_nats_stream_bytes{…} / (fleeger_nats_stream_max_bytes{…} > 0), "cap", "bytes", "", "") or label_replace(fleeger_nats_stream_messages{…} / (fleeger_nats_stream_max_messages{…} > 0), "cap", "messages", "", "") or label_replace(fleeger_nats_stream_oldest_age_seconds{…} / (fleeger_nats_stream_max_age_seconds{…} > 0), "cap", "age", "", ""))(each
{…}is{location="meta",source="fleeger"}), withcomparator = gte,threshold = 90andfor_secs = 300. A cap the stream does not have has no series, so a stream without caps (the olderLimitsshape) is silent; the account-levelnats_storage_highcovers that one.nats_stream_discarding— "NATS stream discarding unforwarded data",critical:increase(fleeger_nats_stream_discarded_unforwarded_messages_total{location="meta",source="fleeger"}[15m])withcomparator = gt,threshold = 0andfor_secs = 0, since every message counted is lost data. It resolves 15 minutes after the last loss. The counter counts only messages that left the stream without being acked, never acked ones nor an operator's drain; how, and its limits, are in NATS & JetStream capacity.series_bound_exceeded— "Metric series bound exceeded: a location makes too many new series",warning, migration0022_series_bound_alert:sum by (series_location) (increase(fleeger_ingestion_new_series_dropped_total{location="meta",source="fleeger"}[15m]))withcomparator = gt,threshold = 0andfor_secs = 0, one alert per location (summed over ingestion-svc's replicas). It fires when ingestion-svc dropped points of new series of a location past its hourly bound ([ingestion] max_new_series_per_location_hour), about a minute after the first drop (the sampler writes the location's counter first as 0), and resolves 15 minutes after the last. The location's existing series and every other location are still written. Causes: an agent inventing resource ids, metric names or label values, a site larger than the bound, or an ingestion-svc restart (or an agent upgrade that renames its series), after which every series of a location counts as new. ingestion-svc's storage sampler writes the counter, so while the sampler fails (nats_sampler_stale) this alert is silent.vm_series_limit_reached— "VictoriaMetrics series limit reached: samples dropped hub-wide",warning, migration0022_series_bound_alert:sum(increase({__name__=~"vm_hourly_series_limit_rows_dropped_total|vm_daily_series_limit_rows_dropped_total"}[15m]))withcomparator = gt,threshold = 0andfor_secs = 0, one alert for the hub. VictoriaMetrics stores its own metrics (-selfScrapeInterval=30s) and exposes each counter only while its limit (-storage.maxHourlySeries,-storage.maxDailySeries) is set; a limit of0is no series. Once a limit is reached every series not yet written in that hour or day of VictoriaMetrics' uptime loses its samples, existing series included, for every location and the hub's own: raise the limit above the hub-wide active series plus their churn, or find the source of the new series (Observability & self-monitoring).
Their threshold and for_secs are editable like the other threshold
built-ins. If ingestion-svc stops writing the gauges, both go stale and
resolve (holding the last sample would not help: a stream fills only while
ingestion is not forwarding, which is after that sample); nats_sampler_stale
below fires instead, and the loss of such an outage is counted when ingestion-svc
restarts and fires the second rule then. (ingestion-svc writes each stream's
counter as 0 in its first sample after starting and the count from the next,
30s later: VictoriaMetrics' increase takes a series' first sample as its
baseline when it is large against the next change, so a counter starting at the
outage's loss would read 0.)
A hub without Loki has no logs stream: ingestion-svc deletes it, so neither
rule has a logs series there (see
NATS & JetStream capacity).
nats_stream_near_cap does not say which cap is near; GET /api/nats/jetstream
or the fleeger_nats_stream_* gauges against their _max_* show it.
All three rules read gauges only ingestion-svc writes, so they are blind while
it is down, which is when the account and the streams fill. Migration
0014_nats_sampler_deadman seeds a dead-man's switch for them, in the
shape of backup_stale: nats_sampler_stale — "NATS storage sampler
silent", warning, enabled — the seconds since ingestion-svc last wrote the
account usage gauge (the one gauge every sample writes, with or without a
limit),
time() - max(tlast_over_time(fleeger_nats_account_storage_used_bytes{location="meta",source="fleeger"}[7d]) or vector(0))
with comparator = gt, threshold = 300 (seconds) and for_secs = 300, so it
fires about 10–11 minutes after the last sample (VictoriaMetrics' default 30s
search latency offset hides the newest half-minute of data, plus up to one
evaluation tick) and resolves once a new sample is visible. A gauge
missing from the whole 7-day window (never written, or silent for longer) reads
as epoch 0 and keeps it firing: silence never means "all clear". The dwell also
keeps a fresh install quiet while ingestion-svc writes its first sample. When it
fires, check that ingestion-svc runs and that its log has no JetStream storage sampling failed warning (it cannot reach NATS or VictoriaMetrics, or JetStream
is off for the account, which the alert below reports first). Its threshold
and for_secs are editable like the other threshold built-ins.
NATS can be up, with JetStream on at the server, while JetStream is off for
the SPOKES account every agent publishes into: when the account's grant
(--js-disk-storage in deploy/nats/gen-auth.sh, loaded by the nats-push job)
exceeds the server's max_file_store, the server refuses to enable it for that
account, logs Error configuring jetstream for account … (10047) once and
carries on. The NATS readiness probe (/healthz?js-enabled-only=true)
checks only the server, so every pod stays Ready while every agent publish is
refused. ingestion-svc's sampler reads the account as a SPOKES user, so it sees
this: when the read is answered "JetStream not enabled for account" (10039), and
for no other failure, it writes fleeger_nats_account_jetstream_enabled 0 to
VictoriaMetrics (every successful read writes 1). Migration
0020_nats_account_jetstream_alert seeds nats_account_jetstream_disabled
— "NATS: JetStream disabled for the agents' account — ingestion halted",
critical, enabled — over
last_over_time(fleeger_nats_account_jetstream_enabled{location="meta",source="fleeger"}[7d])
with comparator = eq, threshold = 0 and for_secs = 0: the server's answer
is its configured state, not a transient, so it fires about 30–90 s after the
account loses JetStream (up to one sample, VictoriaMetrics' 30 s search latency
offset and one evaluation tick) and resolves at the first sample that reads the
account again. Like nats_storage_high it holds the last sample, so it keeps
firing if ingestion-svc stops while the account is disabled. A hub whose
ingestion-svc never wrote the gauge has no series, and the rule stays silent.
A disabled account has no storage to sample, so nats_sampler_stale fires as
well about ten minutes later; it stays the backstop for every other sampler
failure. To fix it, raise max_file_store to at least the grant (on Helm,
natsServer.persistence.size, of which it is 75%; the chart refuses a smaller
volume) and restart NATS, or re-sign SPOKES with a smaller grant and re-run the
push job. Its threshold and for_secs are editable like the other threshold
built-ins.
A log batch Loki refuses in a way that may be its own (a 500 whose body names
no transient cause, or a 5xx other than 502 to 504 and 520 to 524) 10 times,
with Loki accepting other batches in between, is dropped by ingestion-svc; a
Loki outage, a timeout, any 429, a 502 to 504, a 520 to 524 (a proxy in front
of Loki) or a 500 naming an unavailable ring or ingester never drops one (see Observability & self-monitoring). Each drop
counts in fleeger_ingestion_log_batches_dropped_total{reason="loki_refused"},
which the storage sampler writes to VictoriaMetrics from 0 after each start,
one series per replica (pod). Migration
0021_log_batches_dropped_alert seeds log_batches_dropped — "Log
batches dropped: Loki refused them", warning, enabled — over
sum without (pod) (increase(fleeger_ingestion_log_batches_dropped_total{location="meta",source="fleeger",reason="loki_refused"}[15m]))
with comparator = gt, threshold = 0 and for_secs = 0, the shape of
nats_stream_discarding, one alert however many replicas drop: every batch counted is log lines Loki never stored,
so it fires at the next evaluation after the sample that counts the drop, and
resolves 15 minutes after the last one. The batch's subject, stream sequence
and Loki's last answer are in ingestion-svc's ERROR line dropping log batch Loki refused. A hub whose ingestion-svc predates this has no series, and the
rule stays silent. Its threshold and for_secs are editable like the other
threshold built-ins.
A latency SLO needs no new machinery: it's an ordinary threshold rule over
the probe latency, e.g.
quantile_over_time(0.95, probe_duration_seconds{phase="total"}[10m]) with
comparator = gt. (A one-click template for it lands with the status UI.)
Naming a cluster label
Alert summaries and subjects (threshold alerts and resource_unreachable)
show a series' cluster label the way the UI does, never as the
raw label:
cluster:<name>shows as the name,k8s-cluster:<uid>as the first 8 characters of the UID.- A label that names an inventory registration (
registration:<id>, a cluster the agent could not read, orresource:<id>, a Proxmox node in no cluster) shows as that resource's name, else its address, from inventory-svc'sGET /locations/{id}/resourcesfor the alert's location. Only that location's Proxmox and Kubernetes registrations count. Without a name, a registration readsregistration <first 8 characters of its id>and a node in no cluster its id. - A name that two such registrations of the location show, or that a
cluster:<name>label seen there shows, gets the start of the id, as the UI'sclusterNamerdoes:pve (registration abcd1234). Names are sanitized and cut to 64 characters before that. - A name is not unique across locations the way the raw label was, so a
threshold alert on a series with a
resource_idthat is named with one also names its location:cpu on vm-100 (pve) at 'Site A'. (A series without one already lists itslocationlabel.)resource_unreachablealways names the location.
Looking a name up never waits on inventory-svc. alerting-svc answers from a cache and reads a location's names in the background: on its first lookup, and every 5 minutes. For ids the last read lacked it reads once after 30 s, then after 1, 2, 4 and 5 minutes while no read finds any of them, however many are missing. One read per location runs at a time; it gives up after 5 s and keeps the names it had. A name read for the first time therefore shows from the next cycle on. The system monitor looks up every labeled resource each cycle, so names are normally read while a resource still reports, before any alert needs them. Every age a system-monitor cycle judges is taken when it reads the state lists, so a slow call later in the cycle does not age a subject past its window.
Only the text changes: the fingerprint, the alert's cluster label and mutes
keep the raw label. The text is written when an alert is raised, so a firing
alert keeps its summary and its recovery notice its subject; only its
re-notifications use the current name.
The alert lifecycle (dedup, mute, re-notify)
All firing/resolving flows through one engine so the two evaluators behave identically:
- Fingerprint — a stable key per
(rule, subject)(rule id + the series' sorted labels, or rule id + location). Dedup is by fingerprint: a condition that stays breached keeps onealertrow rather than spamming a new one each cycle. A threshold series'reasonlabel is left out: it says what state the series is in, not which subject it is (probe_tls_cert_validis0{reason=…}while invalid and a bare1once fixed), so a changed or dropped reason is the same alert. The alert keeps the reason it was raised with until it resolves: in its stored labels, and in its text when the series is named by its labels (… on check_id c1, location homelab, reason self-signed; aresource_idname leaves it out). A later change of reason updates neither; the resolve notice names neither (see Resolve). An alert raised by an older release is keyed with its reason; its rule's next evaluation moves it to the new key in place, keeping its start and its notification stamp, so the upgrade neither pages nor clears it. Should two rows of one subject be firing under different reasons at the upgrade, the older is kept and the other retired without a notice. - Subject — a threshold alert's notification (and its summary in the
alert list) names the series by its
resource_id. A series without one is named by its own labels, compactly (stream metrics), leaving out__name__,source,service, andlocationwhen it is the hub's ownmeta; any other location is kept. A series left with nothing readsthe hubwhen it is hub-wide (location="meta", e.g.nats_storage_high) andthe aggregateotherwise (a label-less result such asbackup_stale's). The name depends only on the series' own labels, never on what else the query returned. Aclusterlabel is never shown raw; it reads as the UI shows it (see Naming a cluster label). Label values have control characters (newlines included) replaced by spaces and are cut to 64 characters, the whole name to 200, with an ellipsis. When the query computes its value (a ratio,increase(),max by,desired - available), so the series has no metric name, the text leads with the value:value 95.000 >= 90 on stream metrics,value 2.000 > 0 on k3s. The name is display text only; the fingerprint is the raw labels, less a threshold series'reason. - Fire — first breach creates a
firingalert and notifies. While it stays firing it only re-notifies afterrenotify_secs, so a persistent root cause doesn't storm the channels. - Resolve — when the condition clears, the alert flips to
resolvedand a one-off "resolved" notice is sent. A threshold alert whose series the query still returns, back on the right side of the threshold, recovers ([RESOLVED] … recovered.). One whose series the query no longer returns is asked about, all of a rule's in one instant query per cycle (more only past 8 KiB of query): does VictoriaMetrics still hold a current sample of it? If so, the query left it out because it is healthy (up == 0,rate(x[5m]) > 5), and it recovers. If not (an offline agent, a deleted workload, a check set to accept invalid certificates, which then stops reportingprobe_tls_cert_valid), nothing was measured, so it is cleared as[CLEARED] … is no longer reported … This is NOT a recovery, as the system monitor clears a subject that vanished. Both leave the alertresolved.- Which series is asked about. When the alert's series has a metric name
(a plain selector, a comparison filter such as
up == 0,last_over_time), exactly that series: that metric with exactly its labels, so a check that stops sendingprobe_tls_cert_validbut goes on sendingprobe_successis still cleared, not recovered. Its current sample is also judged by the rule's threshold: still breaching, the alert is held, not resolved, for one cycle. That covers a sample written after the rule's query ran, which the query returns next cycle, so the alert goes on firing as usual. Breaching again the next cycle, the raw sample is not what the query judges (a filter on something else,x > 90 and on(host) maintenance == 0,unless,topk,offset, a rollup that keeps the name such asmax_over_time, orkeep_metric_names): the series is present, so the alert recovers, a cycle late. The hold is kept in memory, so a restart in between can hold an alert one cycle more. A query that computes its value (rate,increase, arithmetic,max by) has no name to ask by; then any series carrying all of its labels counts, whatever its metric, and counts as healthy: no raw sample is the rule's value, so only presence is asked. That is weaker: another metric of the same subject keeps it current, so such a rule can recover a subject one of whose metrics stopped. Among the built-in rulescert_expiringandcert_invalidask by their exact series;workload_degraded,nats_storage_high,nats_stream_near_capandnats_stream_discardingby their labels; the dead-man rules (backup_stale,nats_sampler_stale) always return their one series. Any label name is asked about, escaped as MetricsQL needs (service.name,app.kubernetes.io/name, a control character as\uXXXX). - An alert with no labels to ask by (a label-less aggregate such as
count(probe_success == 0) > 3orsum(x > 5) > 0, or one whose only label is areason) recovers when its query returns nothing, as every alert did in older releases: no subject of it can stop being reported, and the rule's own result is the whole measurement, so an empty result is the aggregate back under the threshold. Nothing is asked of VictoriaMetrics. - Current means within the lookback VictoriaMetrics applies to an
instant query. For a rule that reads its metric at an instant (a plain
selector,
up == 0,max by, arithmetic), that is the window its own query just used: a longer one would find the last sample of a series that just stopped and call it a recovery, a shorter one would miss a healthy series pushed less often. A rule that names a range of its own is still asked at the lookback, not over its range, so the two windows differ.nats_stream_discarding(increase(…[15m])) andnats_storage_high(last_over_time(…[7d])) go on returning a series that stopped until it leaves their range (15 minutes, 7 days); only then is it asked about, and both ask by labels, so any series with the same labels reported within the lookback recovers it. A rule asking by exact series with such a range (last_over_time(x[1h]) > 5) over a metric pushed less often than the lookback finds its healthy series, which the query filtered out, not current, and clears the alert as no longer reported instead of recovering it. - If VictoriaMetrics does not answer, nothing is resolved on a guess: the alerts that query asked about stay firing and are asked about again next cycle (logged once, not every cycle); when a rule needs several queries, the alerts of those that answered are resolved on their answers. The warning clears once that rule's queries all answer (or nothing is left to ask), whatever other rules do.
- The notice names the subject without the
reasonit was raised with (probe_tls_cert_valid on check_id c1, location homelab recovered.): that reason no longer holds. The stored alert keeps it.
- Which series is asked about. When the alert's series has a metric name
(a plain selector, a comparison filter such as
- Mute / maintenance windows — a rule can be snoozed (
mute_until), and amuterow can snooze a rule, a resource, or a whole location for a window. A window can start now (a plain snooze) or be scheduled ahead (starts_in_minutes) and carry a humanlabel— a maintenance window for planned work. The engine honorsstarts_at ≤ now < until. While muted the alert is still recorded (visible in the UI) but no notification is sent. A "resolved" notice is never muted — the all-clear always gets through. (Recurring windows are a later addition.) A muted resource is never part of a groupedresource_unreachablealert (above), so its mute holds even in a mass outage; a location mute silences that location's group. Aresourcemute can also carry alocationand acluster, compared with the alert's labels of those names: each one, when set, must match, so muting one cluster'svm-100leaves another's alone. Left out, it matches the id in any location or cluster, which is how every mute stored before rc.3 keeps working. A set scope never matches an alert lacking that label: only Proxmox and Kubernetes resources reported by rc.3 agents carrycluster. Scope a mute only for those, whose ids can repeat across clusters or locations. Check ids are unique, and a check alert'slocationis the location id of one failing vantage, which can change from cycle to cycle, so a location-scoped check mute can stop matching. The target, and a scope when set, are compared exactly, so the API refuses one that is blank or has leading or trailing whitespace (a pasted id with a stray space) instead of listing a mute that silences nothing. Migration0019deletes such rows stored before, and its ASSERTs refuse new ones written directly (an import skips ASSERTs). A blanklocationorclusterscope, which only a write past the API could store, did match the resource's alerts lacking that label, so its deletion can unmute them. A mute is deleted by the alerting reaper a day after its window ends, whatever the retention policy says:GET /muteslists live and upcoming windows only, so nothing reads an expired one.
Notification channels
Channels are pluggable behind a single Channel abstraction; this build ships
ntfy, Discord, Matrix, and email (SMTP). Adding a provider is a new variant
plus its send — the rest of the system is unaware.
Channel config (webhook URLs, SMTP passwords, Matrix tokens) is encrypted at
rest with the same service-held AES-256-GCM key as the inventory credentials
(FLEEGER_CREDENTIAL_KEY); only ciphertext is stored, and it's decrypted only at
dispatch. Nor does it reach the logs: a failed delivery or channel test logs the
provider and what went wrong (discord request failed: error sending request),
never the request URL, since a webhook URL or ntfy topic is the credential
itself and alerting-svc's logs are readable through /api/logs with
resource.view. A rule lists the channel(s) it notifies.
A rule's or a channel's name is 1 to 100 characters (not bytes), on update as on create: an empty or longer new name is refused with a 400 before anything is stored. A name sent back unchanged is not checked, so a row saved with an over-long name before that check can still have its other fields edited.
A rule name, a rule's scope_location, a channel name, and every string in a
channel's config (URLs, topic, room id, tokens, SMTP host, user, password and
addresses), may not contain a control character (a line break, tab, NUL, DEL or
a C1 code; non-ASCII is fine): saving one is refused with a 400 that names
the field but never repeats the value, since it is usually a credential. A scope is checked as it is stored, after its padding is trimmed, so a
trailing line break is still just trimmed off. A rule's query may span lines,
so it keeps line breaks and tabs (\n, \r, \t) but refuses any other
control character the same way. On an update only a new name, scope or
query is checked, so a client that sends back the value it read is not refused
over it. The rule description may still hold line breaks.
Rows saved before those checks are not changed or migrated; they are checked only when that name, scope, query or config is edited. Until then the runtime copes with them: an ntfy notification's title travels in an HTTP header, so a control character in it (a line break in a rule name, say) is sent as a space; non-ASCII is kept. A request the stored config makes impossible to build (a token with a line break) fails the channel test with a 400, and a rule's delivery reports the channel as unusable rather than retrying it every cycle.
Editing a channel
The API never returns a channel's config, not even masked: GET /channels
and GET /channels/{id} give its id, name, kind, enabled, created_at and
updated_at only. So an edit cannot send back what it read, and
PUT /channels/{id} takes a partial config: an object whose
fields are set over the stored config, a field sent as null is removed, and
a field left out keeps its stored value. A rename, or a new topic or SMTP
recipient, does not need the webhook URL, token or password again. The
result must still be a valid config of the channel's kind (a required field
removed is a 400, and nothing is written); the kind itself is fixed at
create. Only the fields sent are checked for control characters; the
destination of the merged config is vetted as on create. A merged config
that does not parse is refused with the kind of error and the fields sent,
never the parser's text, which could quote a decrypted stored value.
A stored config that can no longer be decrypted (its key is gone) has
nothing to keep, so the PUT must then carry a whole config, which replaces
it; a partial one is refused with a 400 saying so.
A config key the channel's kind does not read (a misspelt pasword, say) is
refused with a 400 naming the key, never its value, on create and in a PUT,
rather than stored unused, as for a rule's fields. A PUT may still send
such a key as null, which removes one stored before this check.
PUT /channels/{id} takes the optional expected_updated_at a rule's PUT
does: the channel's updated_at as read, sent back exactly. A channel
written since is refused with 409 STALE, before any other check, and
nothing is written; a deleted one is a 404. Without the token (or with
null) the PUT applies regardless, as a rule's does, but its write is still
conditional: a channel written between the PUT's own read and its write is
read again and the PUT made again over it, so a partial config is never
merged over a stale copy. Every write of the channel's row spends the token:
an edit, a rekey (which re-encrypts the config), and a rule create or
update that routes a rule to the channel (which writes the channel's row, see
above).
Alerts → Channels has an Edit button per channel. The form opens with
the channel's name and kind (locked), and every config field empty, since none
is shown back: an empty field keeps its current value, so only the fields
typed into are sent, and a save with nothing changed closes the form
without a write. An optional field (the ntfy server and token, the SMTP
port) can be removed with its checkbox. On STALE the form reloads the
channel, takes its new name unless you changed the name too (then it says so),
keeps the config fields you typed in, and asks you to save again, as the rule
form does.
The audit row of an edit (notification_channel.update) names the
config's fields as merged and those the edit changes, never a value, and
records the expected_updated_at sent. Those are reckoned against the
handler's read; the completed row's result names the config fields the
write actually changed, against the row it was applied to.
Where a channel may deliver
Whoever may create alert rules may create channels, and alerting-svc can reach the hub's own services, some of which have no authentication of their own (VictoriaMetrics' admin API deletes every series on one POST). So a channel may not point at the hub itself. Refused:
- Loopback (
127.0.0.0/8,::1, and0.0.0.0/::, which reach loopback on connect). From inside the hub, "this machine" is the hub. - The hub's own services, as alerting-svc's config names them: the
[victoriametrics],[ingestion],[inventory],[nats]and[database]URLs. Refused by host name, and by every address that name resolves to, on any port, sohttp://victoriametrics:8428, the container's IP and another name for the same address are all refused. - The hosts, addresses and CIDR ranges in
[alerting] hub_services, a list in the same form as the core agent'sFLEEGER_HUB_SERVICES. Names are refused like the service URLs above, addresses and ranges by every address a destination resolves to. It is where Loki goes: Loki's delete API takes its parameters from the URL and has no key, so a webhook aimed at it would delete every log. The compose stack sets["loki"]; the chart sets the host ofloki.urlplusalerting.hubServices. In Kubernetes a name resolves to its Service's ClusterIP, not to the pod IP behind it, so list the pod and service CIDRs too (alerting.hubServicesin the chart) to refuse the pod IPs of VictoriaMetrics and Loki; leave out any range a notification target of yours lives in. An entry that is no host, address or range stops alerting-svc from starting. - Link-local and cloud-metadata hosts, as before.
VictoriaMetrics' delete API also needs a key now, which only ingestion-svc holds, so even a channel the guard lets through cannot delete series.
LAN and public hosts stay allowed: a self-hosted ntfy, Gotify or SMTP relay on the LAN works as before.
Services run directly on a host (no container network, the service URLs
set to localhost or 127.0.0.1): the guard refuses loopback, but not the
host's own LAN addresses, which it does not look up. A service that
listens on all interfaces (VictoriaMetrics' default :8428, a SurrealDB or
NATS bound to 0.0.0.0) is then reachable from alerting-svc at
http://<host-LAN-IP>:<port>. List those addresses in [alerting] hub_services, for example hub_services = ["192.168.1.10", "fd00::10"], or
bind the services to 127.0.0.1. A notification target on the same host is
then refused as well, as on loopback. The compose and Kubernetes layouts are
not affected: there VictoriaMetrics' port is not published and NATS listens on
127.0.0.1.
The check runs when a channel is created or its config is edited (a 400
naming the reason), and again on every delivery, so a channel stored before the
check existed fails its test with the same 400 and, attached to a rule, is
reported as unusable rather than retried. On delivery the address connected to
is the one judged: HTTP host names resolve through the guard itself, and SMTP
connects to the address the guard checked (with the certificate still verified
against the host name), so a name that changes its DNS answer between check and
connect gains nothing. Notification requests do not follow redirects: a webhook
that answers with a 3xx is reported as that status, so point the channel at the
final URL. An HTTP proxy from the environment (HTTPS_PROXY and friends) is the
operator's and is not judged; when one is set, the proxy resolves the
destination, and only the name check and the pre-send lookup apply.
Services alerting-svc's config does not name, such as Loki, are not covered by this check; keep them off networks alerting-svc can reach, or behind authentication.
Cycle and delivery tunables
These [alerting] keys are range-checked when the service starts, and an
out-of-range value fails the pod immediately rather than starting in a broken
state. Only start range-checks them: db migrate and db unlock still parse
the whole config, so a value that is not a number at all (or does not fit one)
blocks them too, but a merely out-of-range one does not.
| Key | Default | Allowed | Why the bound |
|---|---|---|---|
eval_interval_secs |
30 |
1 or more; while heartbeat_alert_ceiling_secs or resource_stale_ceiling_secs is on, at most half of (that ceiling − heartbeat_stale_secs) |
It is the pause between monitor cycles. At 0 there is none: all four monitors re-query VictoriaMetrics, SurrealDB, ingestion-svc and inventory-svc back to back, hundreds of times a second. The upper bound keeps two cycles between the default window and each retirement ceiling. |
renotify_secs |
3600 |
0–8000000000000 |
0 is allowed and re-notifies on every cycle. The window is added to a datetime in the database, whose datetimes end in the year 262142; a value a little past the maximum (~250,000 years, kept a few thousand years short of that edge) overflows it and makes the re-notify check error for every still-firing alert. Any value up to it already works as "effectively never". |
check_fail_confirmations |
2 |
any count whose wait, (count − 1) × eval_interval_secs (the interval taken as at most 315360000), is at most 315360000 s (~10 years) |
check_down fires only after that many consecutive failed cycles. A count whose wait passes ~10 years would never fire, so it is refused rather than silently disabling the rule; to stop check_down, disable the rule. |
http_timeout_secs |
30 |
1 or more |
A zero timeout is a deadline already passed, not "no timeout": every request slower than about a millisecond fails. |
http_connect_timeout_secs |
10 |
1 or more |
Same, for the TCP handshake: every webhook to an off-host endpoint would fail. |
smtp_timeout_secs |
120 |
1 or more |
Same, for the whole SMTP conversation: every email notification would fail. |
A 0 timeout mostly works against a stub on loopback, which answers faster than
the timer ticks — that is how it slips through a local test. There is no upper
bound on the timeouts, nor on eval_interval_secs with both retirement ceilings
off (see the table for the bound while either is on): nothing overflows, a long
timeout only stretches a cycle whose upstream hangs, and an interval longer than
the pod lives already means "one cycle, then none" — no cut-off would separate
working values from broken ones. The advice to keep http_timeout_secs well
under the cycle is advice, not a check: a cycle that outlasts the interval simply
starts its pause later.