// Benutzung · Monitoring & Alerting
Uptime / synthetic monitoring
Fleeger can actively probe an endpoint and turn the result into metrics, alongside the metrics it scrapes from Proxmox/Kubernetes/iDRAC. The first probe type is an HTTP(S) healthcheck — "is this URL reachable and does it return what I expect?". This is the foundation; alerting, a status UI, more probe types, Kubernetes auto-discovery, multi-vantage-point probing and reporting build on top.
Probe types
The first and richest type is the HTTP(S) healthcheck (http_check). Three
lighter, dependency-free types cover non-web services:
| Kind | address |
Passes when |
|---|---|---|
http_check |
an http(s) URL | status + optional body assertions hold |
tcp_check |
host:port |
a TCP socket opens within the timeout |
tls_check |
host:port (defaults 443) |
the TLS handshake completes (also reports cert expiry/validity) |
dns_check |
a bare hostname | the name resolves (optionally to an expected answer, via check_expected_body) |
The address is checked against the kind on create, whenever it is sent on an
update, and whenever an update changes the kind without sending one: the
stored address must then pass the new kind's check, or the update is refused
with 400 and nothing changes. So a tls_check on example.com
cannot simply become an http_check; send the new kind's address with it. A
checkpoint rollback applies the same check to a kind or address it changes
(see Config revisions, checkpoints & rollback).
The check's page links its address only when it is an http: or https: URL;
any other address is shown as text.
All four share the check_* cadence fields and emit the same probe_success
(+reason) and probe_duration_seconds{phase="total"} metrics, so alerting and
the UI treat them uniformly. (ICMP ping and gRPC health are a follow-up — they
need raw-socket privileges and a gRPC client respectively.)
The check model
A check is a first-class resource (kind http_check / tcp_check /
tls_check / dns_check) under a Location —
not a separate store. That means it inherits tags, an optional encrypted
credential, config history/checkpoints and per-location RBAC exactly like a
proxmox_node or k8s_cluster. What sets it apart is that the spoke-agent
probes it rather than scraping an API.
The target URL lives in the resource's existing address field (validated as a
plain http(s) URL with no embedded credentials, and screened against
link-local/cloud-metadata hosts — the same SSRF guard as every other target
address). The probe parameters live in check_* fields (all optional, with
agent-side defaults):
| Field | Meaning | Default |
|---|---|---|
check_method |
HTTP method to probe with (GET/HEAD/…) |
GET |
check_interval_secs |
How often to probe (≥ 5) | 60 |
check_timeout_secs |
Per-probe timeout (≥ 1, must be < interval) |
10 |
check_expected_status |
Exact code (200) or class (2xx) |
2xx |
check_expected_body |
Optional substring the body must contain (a dns_check's expected answer) |
none |
check_follow_redirects |
Whether the prober follows redirects | false |
check_enabled |
Actively probed? (null = enabled) |
enabled |
check_* fields are only accepted for the check kinds (http_check,
tcp_check, tls_check, dns_check); supplying them on any other kind is
rejected. Not every check kind reads every field: check_interval_secs,
check_timeout_secs and check_enabled apply to all four;
check_expected_body to http_check and, as the expected answer, dns_check;
check_method, check_expected_status and check_follow_redirects to
http_check alone. A check kind refuses a value for a field it does not read
with a 400 naming the field, on create and on update: a tcp, tls or dns check
refuses check_method, check_expected_status and check_follow_redirects, and a tcp or tls check check_expected_body. There,
null, or a blank method, expected status or body, sets nothing and is
accepted. A dns_check stores its expected answer trimmed, and a blank one as
none. A stored value for a field the check's kind does not read is dropped on
any PUT to the check: one that changes its kind (an http_check changed to
tcp_check drops its method, expected status, body and redirect flag; a
dns_check changed to tcp_check its expected answer), and any other edit of
a check that holds such a value from an earlier release or a checkpoint
rollback. Such a value sent back unchanged is accepted and dropped too; a
different value is refused. check_expected_body means something else to each
kind that reads it (a substring of the response to an http_check, an exact
answer to a dns_check), so a change between the two drops it as well: left
out or sent back unchanged it is gone, and a new value is taken as sent.
The resource form empties the box on that change. A kind that is not a check
reads none of a check's settings, so a check changed to one (an http_check
turned into the proxmox_node it was probing) drops all of them the same way:
its check_* parameters, vantages, SLO target, and status-page entry, label
and component. The form sends no check field for such a kind, and the change
goes through; a value sent with it that differs from the stored one is refused
as above, label and component included. On PUT /api/resources/{id},
a null returns a parameter with a default to that default and clears
check_expected_body; a field left out is untouched. The effective
probe cadence is also bounded below by the agent's own poll interval — a check
cannot probe faster than the collection cycle.
Credentials
An http_check may reference an encrypted credential (like any resource) to
carry HTTP auth. The agent's prober recognizes:
username(+ optionalpassword) → HTTP Basic authtoken/bearer_token→Authorization: Bearer …header_name+header_value→ a single custom header
The secret is applied to the request and never logged or emitted as a metric label.
The prober
The prober is a spoke-agent collector (collectors/http.rs), so a check is
probed from the agent at its Location — reusing that Location's network
vantage point (a later issue adds multi-vantage-point probing). Each cycle the
agent probes every enabled check that is due per its own interval, and
publishes one synthetic-source MetricMessage per
probe over NATS. A down target is a valid measurement (probe_success = 0),
not a collection failure, so it never trips the resource's collect-error state.
Metrics
All points carry the standard location / source="synthetic" /
resource_id="<check id>" labels, plus:
| Metric | Labels | Meaning |
|---|---|---|
probe_success |
reason |
1 if all assertions passed, else 0 |
probe_duration_seconds |
phase |
latency by phase: total, first_byte |
probe_http_status_code |
— | the HTTP status returned (when a response came back) |
probe_response_size_bytes |
— | response body size (when read) |
probe_tls_cert_expiry_seconds |
— | seconds until the served TLS cert expires (negative once expired); HTTPS only |
probe_tls_cert_valid |
reason |
1 if the cert is fully valid, else 0 with a reason; HTTPS only |
The reason label on probe_success is a bounded set so cardinality stays
small: ok on success, otherwise timeout, connection_refused, dns_error,
tls_error, status_mismatch, body_mismatch, request_error, or
blocked_target: a check the bundled core agent did not probe because its
target is one of the hub's own services (see
Agent modes: remote spoke vs. bundled core). It both
lets alerting match probe_success == 0 and explains why a check is down.
Phases. This first cut emits
totalandfirst_byte(time to response headers). Finer phases (dns/connect/tls) and the latency waterfall are a follow-up, together with the status UI.
Example
// POST /api/locations/<loc>/resources
{
"kind": "http_check",
"address": "https://example.com/healthz",
"check_interval_secs": 30,
"check_timeout_secs": 5,
"check_expected_status": "2xx",
"check_expected_body": "OK"
}
Then query VictoriaMetrics through the gateway, e.g. probe_success{source="synthetic"}
or probe_duration_seconds{phase="total"}.
UI
The admin UI has an Uptime section:
- Uptime dashboard (
/uptime) — a global up/down/unknown/disabled summary and per-location tiles for every check, each showing current status, latency and last-checked age (plus an "alerting" marker whencheck_downis firing). - Check detail (
/uptime/<id>) — current status + failing reason, an up/down availability timeline, a latency chart (total & time-to-first-byte), rolling uptime % for the selected range, and the failure reasons seen.
Checks themselves are created and edited like any resource: on a location
page, add a resource of kind http_check and fill in the probe fields (method,
interval, timeout, expected status/body, follow-redirects, enabled). An empty
Interval or Timeout box means the default above (shown as the box's
placeholder): a new check is created without one, and emptying the box on an
edit returns a stored value to the default.
Kubernetes auto-discovery
The agent already discovers Ingresses; for each it now derives a candidate
URL (scheme from TLS, the first host, and an optional
fleeger.io/uptime-path annotation) and notes whether it opted in via the
fleeger.io/uptime: "true" annotation. The hub stores a candidate URL only
when a create of an http_check would accept it (an http(s) URL without
credentials and not on a link-local or cloud-metadata host), and stores
null in its place otherwise. The Uptime page shows a "Discovered
candidates" inbox listing ingresses not yet covered by a check, offering only
a candidate URL that is an http(s) URL; annotation-opted-in ones are flagged
recommended. One click promotes a candidate to an
active http_check, tagged with its ingress provenance
(source:ingress:<ns>/<name>) and probed with the default parameters above,
which it stores no value for. (Full GitOps auto-materialization — the agent
creating and reconciling checks from the annotation without a click, and
deactivating them when the ingress disappears — is a follow-up; today promotion
is one click.)
TLS certificate monitoring
For an HTTPS check the agent opens a second, dedicated TLS handshake to grab
the served leaf certificate (reqwest doesn't expose it). A capturing rustls
verifier records the cert and accepts unconditionally — so even an expired,
self-signed or hostname-mismatched cert is inspected — while the verifier the
HTTP probe itself uses (rustls-platform-verifier, over the agent host's trust
store) decides trust, so the probe and the certificate verdict agree. An
internal CA installed on the agent host is trusted, and a CA removed from it is
not, from the next check on; a host with no CA bundle at all trusts nothing,
reports every certificate invalid (untrusted_chain or self_signed when
nothing else is wrong with it) and logs a warning once. The leaf is
parsed (validity window, SANs, issuer) to categorize why it's invalid:
expired, not_yet_valid, hostname_mismatch, self_signed, or
untrusted_chain. Only public certificate metadata is read, never any private
key material.
This drives probe_tls_cert_expiry_seconds and probe_tls_cert_valid, plus two
built-in threshold alerts: "TLS certificate expiring" (default < 14 days) and
"TLS certificate invalid". The check-detail page shows days-to-expiry and
validity. (The cert-manager cross-check — comparing a discovered Certificate
CRD against the served cert — is a follow-up.)
A check with Don't verify the TLS certificate (http_check) or Don't
judge the certificate's validity (tls_check) ticked (allow_invalid_certs)
reports the expiry but no validity: there is no probe_tls_cert_valid series,
so "TLS certificate invalid" never fires for it, while "TLS certificate
expiring" still does. The check-detail page then shows the days to expiry (or
expired) marked not checked, and warns under 14 days as for any other check.
Reporting (uptime %, SLO, incidents)
The check-detail page reports uptime % over fixed windows (24h / 7d / 30d /
90d), computed with avg_over_time subqueries that collapse the reason and
location labels to "up if any vantage is up". A per-check SLO target
(check_slo_target, percent) drives an error-budget figure: over 30 days,
budget used = actual downtime ÷ allowed downtime (100 − target)%. An incident
timeline is derived from the up/down series (1→0 / 0→1 transitions) with each
down period's start, duration and whether it's ongoing. A one-click Export
downloads a JSON report (windows + incidents). (Excluding maintenance-window
downtime from these figures is a follow-up.)
Multi-vantage-point probing
A check can be probed from more than its home location: check_vantages lists
extra location id keys (each must name a location; deleting a location takes it
out of every check's list), and inventory hands the check to those
locations' agents too (each publishes under its own location label, giving per-vantage up/down +
latency). The check-detail page shows a per-vantage breakdown. Alerting then
uses a majority quorum — check_down fires only when a majority of a check's
live vantages fail — so a single failing vantage among several is a partial
outage that doesn't page, while a single-vantage check still fires on its own
failure. Each vantage votes with its newest probe result only, whatever its
reason label, so a check that just went down (or flaps between up and down)
isn't outvoted by its own earlier passing probe. A vantage whose whole
location is dark is excluded from the quorum (that's location_unreachable's
job). (An authenticated check's credential is
location-scoped, so it only carries auth from its home location — noted for a
follow-up.)
Public status page
An optional, unauthenticated status page (/status) shows a curated view of
the checks explicitly opted in (check_public, default off — nothing is
public until you turn it on). It exposes only a public label, a component
group, the current status, and 24h uptime — never ids, addresses, tags,
or any PromQL. A check with no label (never set, or cleared) is shown under its
resource name, or as check when it has none. The projection is served by a single gateway endpoint
(GET /api/public/status) that lives on the gateway's public (no-auth) router,
fans out internally to inventory + VictoriaMetrics, strips everything sensitive,
and caches for 30s to bound anonymous load. If inventory-svc cannot be read, the endpoint answers
503 and the page says the status is temporarily unavailable, rather than
showing an empty list; that answer is cached for 10s. There is no write path from the
public page. (Manual incident notes / status posts are a follow-up.)
Alerting
The built-in check_down alert (always on, undeletable — pick channels to
be notified) fires when a check fails for a few consecutive cycles and clears
when it recovers. A whole-location outage is suppressed in favor of
location_unreachable. For planned work, schedule a maintenance window
(a mute with a start offset + label) so the check doesn't page. Latency SLOs are
plain threshold rules over probe_duration_seconds. See
Alerting for the full lifecycle.