Zum Inhalt springen
Produkte

// Benutzung · Monitoring & Alerting

Uptime / synthetic monitoring

Fleeger can actively probe an endpoint and turn the result into metrics, alongside the metrics it scrapes from Proxmox/Kubernetes/iDRAC. The first probe type is an HTTP(S) healthcheck — "is this URL reachable and does it return what I expect?". This is the foundation; alerting, a status UI, more probe types, Kubernetes auto-discovery, multi-vantage-point probing and reporting build on top.

Probe types

The first and richest type is the HTTP(S) healthcheck (http_check). Three lighter, dependency-free types cover non-web services:

Kind address Passes when
http_check an http(s) URL status + optional body assertions hold
tcp_check host:port a TCP socket opens within the timeout
tls_check host:port (defaults 443) the TLS handshake completes (also reports cert expiry/validity)
dns_check a bare hostname the name resolves (optionally to an expected answer, via check_expected_body)

The address is checked against the kind on create, whenever it is sent on an update, and whenever an update changes the kind without sending one: the stored address must then pass the new kind's check, or the update is refused with 400 and nothing changes. So a tls_check on example.com cannot simply become an http_check; send the new kind's address with it. A checkpoint rollback applies the same check to a kind or address it changes (see Config revisions, checkpoints & rollback). The check's page links its address only when it is an http: or https: URL; any other address is shown as text.

All four share the check_* cadence fields and emit the same probe_success (+reason) and probe_duration_seconds{phase="total"} metrics, so alerting and the UI treat them uniformly. (ICMP ping and gRPC health are a follow-up — they need raw-socket privileges and a gRPC client respectively.)

The check model

A check is a first-class resource (kind http_check / tcp_check / tls_check / dns_check) under a Location — not a separate store. That means it inherits tags, an optional encrypted credential, config history/checkpoints and per-location RBAC exactly like a proxmox_node or k8s_cluster. What sets it apart is that the spoke-agent probes it rather than scraping an API.

The target URL lives in the resource's existing address field (validated as a plain http(s) URL with no embedded credentials, and screened against link-local/cloud-metadata hosts — the same SSRF guard as every other target address). The probe parameters live in check_* fields (all optional, with agent-side defaults):

Field Meaning Default
check_method HTTP method to probe with (GET/HEAD/…) GET
check_interval_secs How often to probe (≥ 5) 60
check_timeout_secs Per-probe timeout (≥ 1, must be < interval) 10
check_expected_status Exact code (200) or class (2xx) 2xx
check_expected_body Optional substring the body must contain (a dns_check's expected answer) none
check_follow_redirects Whether the prober follows redirects false
check_enabled Actively probed? (null = enabled) enabled

check_* fields are only accepted for the check kinds (http_check, tcp_check, tls_check, dns_check); supplying them on any other kind is rejected. Not every check kind reads every field: check_interval_secs, check_timeout_secs and check_enabled apply to all four; check_expected_body to http_check and, as the expected answer, dns_check; check_method, check_expected_status and check_follow_redirects to http_check alone. A check kind refuses a value for a field it does not read with a 400 naming the field, on create and on update: a tcp, tls or dns check refuses check_method, check_expected_status and check_follow_redirects, and a tcp or tls check check_expected_body. There, null, or a blank method, expected status or body, sets nothing and is accepted. A dns_check stores its expected answer trimmed, and a blank one as none. A stored value for a field the check's kind does not read is dropped on any PUT to the check: one that changes its kind (an http_check changed to tcp_check drops its method, expected status, body and redirect flag; a dns_check changed to tcp_check its expected answer), and any other edit of a check that holds such a value from an earlier release or a checkpoint rollback. Such a value sent back unchanged is accepted and dropped too; a different value is refused. check_expected_body means something else to each kind that reads it (a substring of the response to an http_check, an exact answer to a dns_check), so a change between the two drops it as well: left out or sent back unchanged it is gone, and a new value is taken as sent. The resource form empties the box on that change. A kind that is not a check reads none of a check's settings, so a check changed to one (an http_check turned into the proxmox_node it was probing) drops all of them the same way: its check_* parameters, vantages, SLO target, and status-page entry, label and component. The form sends no check field for such a kind, and the change goes through; a value sent with it that differs from the stored one is refused as above, label and component included. On PUT /api/resources/{id}, a null returns a parameter with a default to that default and clears check_expected_body; a field left out is untouched. The effective probe cadence is also bounded below by the agent's own poll interval — a check cannot probe faster than the collection cycle.

Credentials

An http_check may reference an encrypted credential (like any resource) to carry HTTP auth. The agent's prober recognizes:

  • username (+ optional password) → HTTP Basic auth
  • token / bearer_token → Authorization: Bearer …
  • header_name + header_value → a single custom header

The secret is applied to the request and never logged or emitted as a metric label.

The prober

The prober is a spoke-agent collector (collectors/http.rs), so a check is probed from the agent at its Location — reusing that Location's network vantage point (a later issue adds multi-vantage-point probing). Each cycle the agent probes every enabled check that is due per its own interval, and publishes one synthetic-source MetricMessage per probe over NATS. A down target is a valid measurement (probe_success = 0), not a collection failure, so it never trips the resource's collect-error state.

Metrics

All points carry the standard location / source="synthetic" / resource_id="<check id>" labels, plus:

Metric Labels Meaning
probe_success reason 1 if all assertions passed, else 0
probe_duration_seconds phase latency by phase: total, first_byte
probe_http_status_code — the HTTP status returned (when a response came back)
probe_response_size_bytes — response body size (when read)
probe_tls_cert_expiry_seconds — seconds until the served TLS cert expires (negative once expired); HTTPS only
probe_tls_cert_valid reason 1 if the cert is fully valid, else 0 with a reason; HTTPS only

The reason label on probe_success is a bounded set so cardinality stays small: ok on success, otherwise timeout, connection_refused, dns_error, tls_error, status_mismatch, body_mismatch, request_error, or blocked_target: a check the bundled core agent did not probe because its target is one of the hub's own services (see Agent modes: remote spoke vs. bundled core). It both lets alerting match probe_success == 0 and explains why a check is down.

Phases. This first cut emits total and first_byte (time to response headers). Finer phases (dns/connect/tls) and the latency waterfall are a follow-up, together with the status UI.

Example

// POST /api/locations/<loc>/resources
{
  "kind": "http_check",
  "address": "https://example.com/healthz",
  "check_interval_secs": 30,
  "check_timeout_secs": 5,
  "check_expected_status": "2xx",
  "check_expected_body": "OK"
}

Then query VictoriaMetrics through the gateway, e.g. probe_success{source="synthetic"} or probe_duration_seconds{phase="total"}.

UI

The admin UI has an Uptime section:

  • Uptime dashboard (/uptime) — a global up/down/unknown/disabled summary and per-location tiles for every check, each showing current status, latency and last-checked age (plus an "alerting" marker when check_down is firing).
  • Check detail (/uptime/<id>) — current status + failing reason, an up/down availability timeline, a latency chart (total & time-to-first-byte), rolling uptime % for the selected range, and the failure reasons seen.

Checks themselves are created and edited like any resource: on a location page, add a resource of kind http_check and fill in the probe fields (method, interval, timeout, expected status/body, follow-redirects, enabled). An empty Interval or Timeout box means the default above (shown as the box's placeholder): a new check is created without one, and emptying the box on an edit returns a stored value to the default.

Kubernetes auto-discovery

The agent already discovers Ingresses; for each it now derives a candidate URL (scheme from TLS, the first host, and an optional fleeger.io/uptime-path annotation) and notes whether it opted in via the fleeger.io/uptime: "true" annotation. The hub stores a candidate URL only when a create of an http_check would accept it (an http(s) URL without credentials and not on a link-local or cloud-metadata host), and stores null in its place otherwise. The Uptime page shows a "Discovered candidates" inbox listing ingresses not yet covered by a check, offering only a candidate URL that is an http(s) URL; annotation-opted-in ones are flagged recommended. One click promotes a candidate to an active http_check, tagged with its ingress provenance (source:ingress:<ns>/<name>) and probed with the default parameters above, which it stores no value for. (Full GitOps auto-materialization — the agent creating and reconciling checks from the annotation without a click, and deactivating them when the ingress disappears — is a follow-up; today promotion is one click.)

TLS certificate monitoring

For an HTTPS check the agent opens a second, dedicated TLS handshake to grab the served leaf certificate (reqwest doesn't expose it). A capturing rustls verifier records the cert and accepts unconditionally — so even an expired, self-signed or hostname-mismatched cert is inspected — while the verifier the HTTP probe itself uses (rustls-platform-verifier, over the agent host's trust store) decides trust, so the probe and the certificate verdict agree. An internal CA installed on the agent host is trusted, and a CA removed from it is not, from the next check on; a host with no CA bundle at all trusts nothing, reports every certificate invalid (untrusted_chain or self_signed when nothing else is wrong with it) and logs a warning once. The leaf is parsed (validity window, SANs, issuer) to categorize why it's invalid: expired, not_yet_valid, hostname_mismatch, self_signed, or untrusted_chain. Only public certificate metadata is read, never any private key material.

This drives probe_tls_cert_expiry_seconds and probe_tls_cert_valid, plus two built-in threshold alerts: "TLS certificate expiring" (default < 14 days) and "TLS certificate invalid". The check-detail page shows days-to-expiry and validity. (The cert-manager cross-check — comparing a discovered Certificate CRD against the served cert — is a follow-up.)

A check with Don't verify the TLS certificate (http_check) or Don't judge the certificate's validity (tls_check) ticked (allow_invalid_certs) reports the expiry but no validity: there is no probe_tls_cert_valid series, so "TLS certificate invalid" never fires for it, while "TLS certificate expiring" still does. The check-detail page then shows the days to expiry (or expired) marked not checked, and warns under 14 days as for any other check.

Reporting (uptime %, SLO, incidents)

The check-detail page reports uptime % over fixed windows (24h / 7d / 30d / 90d), computed with avg_over_time subqueries that collapse the reason and location labels to "up if any vantage is up". A per-check SLO target (check_slo_target, percent) drives an error-budget figure: over 30 days, budget used = actual downtime ÷ allowed downtime (100 − target)%. An incident timeline is derived from the up/down series (1→0 / 0→1 transitions) with each down period's start, duration and whether it's ongoing. A one-click Export downloads a JSON report (windows + incidents). (Excluding maintenance-window downtime from these figures is a follow-up.)

Multi-vantage-point probing

A check can be probed from more than its home location: check_vantages lists extra location id keys (each must name a location; deleting a location takes it out of every check's list), and inventory hands the check to those locations' agents too (each publishes under its own location label, giving per-vantage up/down + latency). The check-detail page shows a per-vantage breakdown. Alerting then uses a majority quorum — check_down fires only when a majority of a check's live vantages fail — so a single failing vantage among several is a partial outage that doesn't page, while a single-vantage check still fires on its own failure. Each vantage votes with its newest probe result only, whatever its reason label, so a check that just went down (or flaps between up and down) isn't outvoted by its own earlier passing probe. A vantage whose whole location is dark is excluded from the quorum (that's location_unreachable's job). (An authenticated check's credential is location-scoped, so it only carries auth from its home location — noted for a follow-up.)

Public status page

An optional, unauthenticated status page (/status) shows a curated view of the checks explicitly opted in (check_public, default off — nothing is public until you turn it on). It exposes only a public label, a component group, the current status, and 24h uptime — never ids, addresses, tags, or any PromQL. A check with no label (never set, or cleared) is shown under its resource name, or as check when it has none. The projection is served by a single gateway endpoint (GET /api/public/status) that lives on the gateway's public (no-auth) router, fans out internally to inventory + VictoriaMetrics, strips everything sensitive, and caches for 30s to bound anonymous load. If inventory-svc cannot be read, the endpoint answers 503 and the page says the status is temporarily unavailable, rather than showing an empty list; that answer is cached for 10s. There is no write path from the public page. (Manual incident notes / status posts are a follow-up.)

Alerting

The built-in check_down alert (always on, undeletable — pick channels to be notified) fires when a check fails for a few consecutive cycles and clears when it recovers. A whole-location outage is suppressed in favor of location_unreachable. For planned work, schedule a maintenance window (a mute with a start offset + label) so the check doesn't page. Latency SLOs are plain threshold rules over probe_duration_seconds. See Alerting for the full lifecycle.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.