Zum Inhalt springen
Produkte

// Betrieb

Offboarding an agent

The Services page's Agents tab lists every location an agent has reported metrics for, with its reachability, resource count, and how long since it last reported. Expanding a row shows its per-source resource breakdown and live NATS connection detail. Offboarding is how you retire an agent or clean up a stale/unknown one.

What offboarding does

Offboarding an agent:

  • clears its reachability/uptime state, so it disappears from the dashboard and the Services → Agents tab,
  • permanently deletes all of that location's metric history from VictoriaMetrics,
  • deletes all of that location's logs from Loki, when the hub has Loki configured, and
  • revokes its NATS credentials — the broker rejects that agent from then on, so it cannot reconnect or publish even if it's still running, and
  • clears that location's firing alerts (location unreachable, its resources', and any other alert labelled with the location). Each is closed with a [CLEARED] notice saying its location was offboarded, not "back online" — nothing was checked. Without this, on a hub with a single location the alerts would stay firing for good. The alerts are cleared only once the state and metrics purge has succeeded: if it fails, the offboard fails, the alerts stay firing, and running it again clears them. alerting-svc's heartbeat monitor forgets the location at the same time, so a heartbeat rule does not raise its alert again. That memory is per process, which is right for the one alerting-svc replica every shipped deployment runs (Compose, the Helm chart and deploy/k8s); with more, only the replica that answered would forget it. A heartbeat cycle already under way re-checks the location before raising its alert. One race remains: a cycle of the built-in unreachable monitors that read ingestion-svc's state just before the purge can still raise one of the location's alerts again after the clear; running the offboard again clears it.
  • hands the location's uptime-check alerts (check_down) to the check monitor instead of clearing them. Such an alert is one per check, and the location it names is only one of the places the check failed from: the check may still be failing from another location, or passing there. The offboard records the location as offboarded in alerting-svc's database (it survives a restart, and holds on a single-location hub, where ingestion-svc's location list is empty afterwards), and the check monitor judges the alert on its next cycles: still failing from another live location, it stays firing (no [CLEARED] followed by a new [FIRING]); passing from one, it is resolved as back online, the notice naming where it passed; reported from no live location, it is cleared after check_fail_confirmations cycles as no longer reported from any live location, not "back online". The record is dropped once no alert names the location, or once the location has been onboarded anew — reporting again for longer than two heartbeat stale windows (heartbeat_stale_secs), whose checks are then judged as any other location's. A shorter return does not count: messages the agent published before the offboard can still be delivered after the purge (a JetStream backlog, or the agent publishing until the revoke lands), and ingestion then lists the location again for a while. Until then the location reads as dark — its results are not evidence and its alerts wait — and once it falls silent it is gone again. The record is kept for at least those two stale windows after the offboard, so a backlog arriving within them, after the alert was cleared or before a failing check was confirmed, still meets it. This assumes ingestion-svc keeps up with its stream: if it is minutes behind when you offboard, the backlog can last long enough to count as a re-onboard; running the offboard again fixes it. While the check_down rule is disabled the check monitor does not run, and the offboard clears those alerts too.

The log delete goes through Loki's compactor delete API, as the retention reaper's does, so it needs the compactor's deletion enabled (deletion_mode: filter-and-delete, as the shipped Loki configs have it). It covers all of the location's log history, however long Loki's own retention_period keeps it, up to Loki's own now, so a hub whose clock runs ahead of Loki's still gets it accepted. Loki removes the lines when its compactor next runs, not at once. If Loki refuses the delete (for example because deletion is not enabled), the offboard still completes and ingestion-svc logs a warning (offboard: Loki refused to delete the logs of '<location>'); those logs then stay until Loki's own retention drops them. If Loki cannot be reached, fails (5xx), or answers with its rate limit or timeout (429, 408), the offboard fails and the location's state is kept, so you can run it again. Should something in front of Loki (a proxy or gateway rule) answer every log delete with 429, offboards and location deletes keep failing until that rule lets the deletes through, and the retention reaper keeps retrying the log purge of every dead resource (and its metric purge with it) on each sweep.

This is destructive and can't be undone — the past metrics and logs are gone and the old credentials are dead. To bring the location back, re-onboard it (Onboard issues a fresh token, which the agent exchanges for new credentials); the revoked ones stay revoked. See re-onboarding a location.

Offboard an agent

  1. Open Services and switch to the Agents tab.
  2. Find the agent (a row with an amber status dot — agent down / never seen — is usually the one you want to remove) and expand it.
  3. Click Offboard and confirm.

The agent's credentials are revoked immediately, so it can't reconnect — a still-running agent will start getting authorization errors and stops reporting for good (no need to also stop it by hand). An agent that restarts later, for example after a reboot, fails to connect and is restarted in a loop (Docker with --restart unless-stopped, or a crash-looping pod), so remove it and its credentials as described below even if you do not bring the location back.

Re-onboarding a location

An agent that finds credentials on its volume uses them and ignores any registration token. After offboarding, or after the NATS credentials were regenerated (gen-auth.sh), those credentials no longer work, so an agent started on the old volume with a new token keeps failing to connect. Remove the old credentials, then onboard with a new token from Onboard:

  • Docker: docker rm -f fleeger-agent and docker volume rm fleeger-agent-data, then run the Docker command from Onboard (or the onboarding guide).
  • Helm: helm uninstall fleeger-agent -n <namespace> deletes the release's credential volume along with it (PVC fleeger-agent-data for a release named fleeger-agent, <release>-fleeger-agent-data otherwise). Then install again with the new token: the Kubernetes command from Onboard replaces the fleeger-agent-token Secret, which helm uninstall leaves in place. A helm upgrade with only a new token keeps the old volume and does not work.
  • Plain manifests (deploy/k8s/agent/): kubectl delete -f fleeger-agent.yaml deletes the fleeger-agent-data PVC with the agent; then replace the token Secret and kubectl apply -f fleeger-agent.yaml again.

If offboarding fails

auth-svc revokes the credentials by connecting to the NATS broker over the same address it hands agents (nats.publicUrl in the Helm chart, [nats] public_url in auth-svc's config). When that is a wss:// URL through an ingress, a load balancer or a tunnel that the edge does not trust, the edge answers that connection with a 503 ("Fleeger is behind a proxy the edge does not trust…"). Offboarding (or deleting the location) still succeeds, because auth-svc has recorded the revocation, but the agent stays admitted: it can keep connecting and publishing. auth-svc logs the failed push at WARN (... marked revoked, but the push failed; the reconcile will push it), and its background reconcile keeps trying to push it, failing the same way.

The reconcile uses that address on every pass, so with this misconfiguration it has been failing since auth-svc started, before anyone offboards anything: its first few failures are logged at WARN, later ones at ERROR. Repeated reconcile errors in auth-svc's log are the early sign.

Fix the trust setting. The next reconcile pass, within about half a minute, pushes the revocation and disconnects the agent; offboarding again does no harm.

The edge's own log shows the refused connection's peer address, which is the address to trust.

Unknown agents

An agent card marked unknown (with a raw id instead of a friendly name) is reporting under a location that no longer exists in inventory — a leftover from a deleted location or a test publish. These are safe to offboard.

Deleting a location cleans up automatically

You don't have to offboard separately when removing a location: deleting a location automatically offboards its agent, so a removed location never leaves a phantom behind.

The delete first asks inventory-svc whether it would refuse it. If it would, the delete answers with inventory-svc's refusal as is and nothing is revoked: the location keeps a working agent. If inventory-svc cannot answer (it is down), nothing is revoked either and the error says so: retry the delete.

Then the delete revokes the agent's NATS credentials and expires the location's unused registration tokens. If that fails (auth-svc or its database is down), nothing is revoked, the location is not deleted, and the delete answers with an error saying so: retry the delete once auth-svc is back. NATS being down does not block it: the revocation is recorded and auth-svc pushes it to the broker as soon as it can (within about 10 seconds of NATS coming back). Then, still before the location is deleted, its state is cleared, its metrics and logs purged, and its firing alerts cleared (only once the purge has succeeded). If either fails, the location is not deleted and the error says so: retry the delete once ingestion-svc (or alerting-svc) is back. Each step is idempotent, so the retry redoes whatever is left. Only then is the location deleted in inventory-svc. They come first because nothing could retry them once the location is gone: no later request names it. With an inventory-svc older than the delete check, which might still refuse the delete, they run after it instead, and a failure is logged by the gateway as an error saying to offboard the location's agent (DELETE /api/agents/<location>) to finish them.

A delete of an id inventory-svc does not know (already deleted, or never created) is not refused by the check, so it still revokes, purges and clears whatever is left under that id (state, metrics, logs, alerts) before inventory-svc answers 404.

If the revoke succeeds but inventory-svc then fails the delete, the location stays, with its agent already revoked and its state, metrics and alerts already cleared, and the error says so: retry the delete. To keep the location after all, onboard it again with a new registration token; the agent cannot get its credentials back by retrying its old one.

Onboarding (the reverse)

For a registered location, Onboard mints a one-time registration token and shows the helm install command to bring an agent up — see Onboarding a new location.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.