// Betrieb
Offboarding an agent
The Services page's Agents tab lists every location an agent has reported metrics for, with its reachability, resource count, and how long since it last reported. Expanding a row shows its per-source resource breakdown and live NATS connection detail. Offboarding is how you retire an agent or clean up a stale/unknown one.
What offboarding does
Offboarding an agent:
- clears its reachability/uptime state, so it disappears from the dashboard and the Services → Agents tab,
- permanently deletes all of that location's metric history from VictoriaMetrics,
- deletes all of that location's logs from Loki, when the hub has Loki configured, and
- revokes its NATS credentials — the broker rejects that agent from then on, so it cannot reconnect or publish even if it's still running, and
- clears that location's firing alerts (location unreachable, its
resources', and any other alert labelled with the location). Each is
closed with a
[CLEARED]notice saying its location was offboarded, not "back online" — nothing was checked. Without this, on a hub with a single location the alerts would stay firing for good. The alerts are cleared only once the state and metrics purge has succeeded: if it fails, the offboard fails, the alerts stay firing, and running it again clears them. alerting-svc's heartbeat monitor forgets the location at the same time, so a heartbeat rule does not raise its alert again. That memory is per process, which is right for the one alerting-svc replica every shipped deployment runs (Compose, the Helm chart anddeploy/k8s); with more, only the replica that answered would forget it. A heartbeat cycle already under way re-checks the location before raising its alert. One race remains: a cycle of the built-in unreachable monitors that read ingestion-svc's state just before the purge can still raise one of the location's alerts again after the clear; running the offboard again clears it. - hands the location's uptime-check alerts (
check_down) to the check monitor instead of clearing them. Such an alert is one per check, and the location it names is only one of the places the check failed from: the check may still be failing from another location, or passing there. The offboard records the location as offboarded in alerting-svc's database (it survives a restart, and holds on a single-location hub, where ingestion-svc's location list is empty afterwards), and the check monitor judges the alert on its next cycles: still failing from another live location, it stays firing (no[CLEARED]followed by a new[FIRING]); passing from one, it is resolved as back online, the notice naming where it passed; reported from no live location, it is cleared aftercheck_fail_confirmationscycles as no longer reported from any live location, not "back online". The record is dropped once no alert names the location, or once the location has been onboarded anew — reporting again for longer than two heartbeat stale windows (heartbeat_stale_secs), whose checks are then judged as any other location's. A shorter return does not count: messages the agent published before the offboard can still be delivered after the purge (a JetStream backlog, or the agent publishing until the revoke lands), and ingestion then lists the location again for a while. Until then the location reads as dark — its results are not evidence and its alerts wait — and once it falls silent it is gone again. The record is kept for at least those two stale windows after the offboard, so a backlog arriving within them, after the alert was cleared or before a failing check was confirmed, still meets it. This assumes ingestion-svc keeps up with its stream: if it is minutes behind when you offboard, the backlog can last long enough to count as a re-onboard; running the offboard again fixes it. While thecheck_downrule is disabled the check monitor does not run, and the offboard clears those alerts too.
The log delete goes through Loki's compactor delete API, as the retention
reaper's does, so it needs the compactor's deletion enabled (deletion_mode: filter-and-delete, as the shipped Loki configs have it). It covers all of the
location's log history, however long Loki's own retention_period keeps it,
up to Loki's own now, so a hub whose clock runs ahead of Loki's still gets it
accepted.
Loki removes the lines when its compactor next runs, not at once. If Loki
refuses the delete (for example because deletion is not enabled), the offboard
still completes and ingestion-svc logs a warning (offboard: Loki refused to delete the logs of '<location>'); those logs then stay until Loki's own
retention drops them. If Loki cannot be reached, fails (5xx), or answers with
its rate limit or timeout (429, 408), the offboard fails and the
location's state is kept, so you can run it again. Should something in front
of Loki (a proxy or gateway rule) answer every log delete with 429, offboards
and location deletes keep failing until that rule lets the deletes through,
and the retention reaper keeps retrying the log purge of every dead resource
(and its metric purge with it) on each sweep.
This is destructive and can't be undone — the past metrics and logs are gone and the old credentials are dead. To bring the location back, re-onboard it (Onboard issues a fresh token, which the agent exchanges for new credentials); the revoked ones stay revoked. See re-onboarding a location.
Offboard an agent
- Open Services and switch to the Agents tab.
- Find the agent (a row with an amber status dot — agent down / never seen — is usually the one you want to remove) and expand it.
- Click Offboard and confirm.
The agent's credentials are revoked immediately, so it can't reconnect — a
still-running agent will start getting authorization errors and stops reporting
for good (no need to also stop it by hand). An agent that restarts later, for
example after a reboot, fails to connect and is restarted in a loop (Docker
with --restart unless-stopped, or a crash-looping pod), so remove it and its
credentials as described below even if you do not
bring the location back.
Re-onboarding a location
An agent that finds credentials on its volume uses them and ignores any
registration token. After offboarding, or after the NATS credentials were
regenerated (gen-auth.sh), those credentials no longer work, so an agent
started on the old volume with a new token keeps failing to connect. Remove the
old credentials, then onboard with a new token from Onboard:
- Docker:
docker rm -f fleeger-agentanddocker volume rm fleeger-agent-data, then run the Docker command from Onboard (or the onboarding guide). - Helm:
helm uninstall fleeger-agent -n <namespace>deletes the release's credential volume along with it (PVCfleeger-agent-datafor a release namedfleeger-agent,<release>-fleeger-agent-dataotherwise). Then install again with the new token: the Kubernetes command from Onboard replaces thefleeger-agent-tokenSecret, whichhelm uninstallleaves in place. Ahelm upgradewith only a new token keeps the old volume and does not work. - Plain manifests (
deploy/k8s/agent/):kubectl delete -f fleeger-agent.yamldeletes thefleeger-agent-dataPVC with the agent; then replace the token Secret andkubectl apply -f fleeger-agent.yamlagain.
If offboarding fails
auth-svc revokes the credentials by connecting to the NATS broker over the
same address it hands agents (nats.publicUrl in the Helm chart, [nats] public_url in auth-svc's config). When that is a wss:// URL through an
ingress, a load balancer or a tunnel that the edge does not trust, the edge
answers that connection with a 503 ("Fleeger is behind a proxy the edge does
not trust…"). Offboarding (or deleting the location) still succeeds, because
auth-svc has recorded the revocation, but the agent stays admitted: it can
keep connecting and publishing. auth-svc logs the failed push at WARN
(... marked revoked, but the push failed; the reconcile will push it), and its
background reconcile keeps trying to push it, failing the same way.
The reconcile uses that address on every pass, so with this misconfiguration it has been failing since auth-svc started, before anyone offboards anything: its first few failures are logged at WARN, later ones at ERROR. Repeated reconcile errors in auth-svc's log are the early sign.
Fix the trust setting. The next reconcile pass, within about half a minute, pushes the revocation and disconnects the agent; offboarding again does no harm.
- Helm:
edge.trustedProxies(andedge.realIpHeader), see the Helm chart reference's Edge trusted proxies. - Raw manifests: the
set_real_ip_fromblock, see Telling the edge who the client is. - Docker Compose: Behind a reverse proxy.
The edge's own log shows the refused connection's peer address, which is the address to trust.
Unknown agents
An agent card marked unknown (with a raw id instead of a friendly name) is reporting under a location that no longer exists in inventory — a leftover from a deleted location or a test publish. These are safe to offboard.
Deleting a location cleans up automatically
You don't have to offboard separately when removing a location: deleting a location automatically offboards its agent, so a removed location never leaves a phantom behind.
The delete first asks inventory-svc whether it would refuse it. If it would, the delete answers with inventory-svc's refusal as is and nothing is revoked: the location keeps a working agent. If inventory-svc cannot answer (it is down), nothing is revoked either and the error says so: retry the delete.
Then the delete revokes the agent's NATS credentials and expires the
location's unused registration tokens. If that fails (auth-svc or its database
is down), nothing is revoked, the location is not deleted, and the delete
answers with an error saying so: retry the delete once auth-svc is back. NATS
being down does not block it: the revocation is recorded and auth-svc pushes it
to the broker as soon as it can (within about 10 seconds of NATS coming back).
Then, still before the location is deleted, its state is cleared, its
metrics and logs purged, and its firing alerts cleared (only once the purge
has succeeded). If either fails, the location is not deleted and the
error says so: retry the delete once ingestion-svc (or alerting-svc) is back.
Each step is idempotent, so the retry redoes whatever is left. Only then is
the location deleted in inventory-svc. They come first because nothing could
retry them once the location is gone: no later request names it. With an
inventory-svc older than the delete check, which might still refuse the
delete, they run after it instead, and a failure is logged by the gateway as
an error saying to offboard the location's agent (DELETE /api/agents/<location>) to finish them.
A delete of an id inventory-svc does not know (already deleted, or never created) is not refused by the check, so it still revokes, purges and clears whatever is left under that id (state, metrics, logs, alerts) before inventory-svc answers 404.
If the revoke succeeds but inventory-svc then fails the delete, the location stays, with its agent already revoked and its state, metrics and alerts already cleared, and the error says so: retry the delete. To keep the location after all, onboard it again with a new registration token; the agent cannot get its credentials back by retrying its old one.
Onboarding (the reverse)
For a registered location, Onboard mints a one-time registration token and
shows the helm install command to bring an agent up — see
Onboarding a new location.