// Troubleshooting
Troubleshooting
Common problems when running Fleeger and how to resolve them. Each section comes from the guide it belongs to, linked at its end.
A location is unreachable
Work the path from the location outward:
- Is the spoke-agent running at that location? (in dev, the host process / its config). A stopped agent is the most common cause.
- Can the agent reach
inventory-svcto list its resources, and the targets (Proxmox / iDRAC / k8s API)? Check the agent log. - Can it reach NATS, and are its credentials scoped to this location? A mis-scoped credential is broker-denied (see the NATS layout in the metrics pipeline).
- Is
ingestion-svcup and consuming? If other locations are fresh, the hub is fine and the problem is at the location.
From: Checking whether a location is reachable
Offboarding succeeds but the agent stays connected
auth-svc revokes the credentials by connecting to the NATS broker over the
same address it hands agents (nats.publicUrl in the Helm chart, [nats] public_url in auth-svc's config). When that is a wss:// URL through an
ingress, a load balancer or a tunnel that the edge does not trust, the edge
answers that connection with a 503 ("Fleeger is behind a proxy the edge does
not trust…"). Offboarding (or deleting the location) still succeeds, because
auth-svc has recorded the revocation, but the agent stays admitted: it can
keep connecting and publishing. auth-svc logs the failed push at WARN
(... marked revoked, but the push failed; the reconcile will push it), and its
background reconcile keeps trying to push it, failing the same way.
The reconcile uses that address on every pass, so with this misconfiguration it has been failing since auth-svc started, before anyone offboards anything: its first few failures are logged at WARN, later ones at ERROR. Repeated reconcile errors in auth-svc's log are the early sign.
Fix the trust setting. The next reconcile pass, within about half a minute, pushes the revocation and disconnects the agent; offboarding again does no harm.
- Helm:
edge.trustedProxies(andedge.realIpHeader), see the Helm chart reference's Edge trusted proxies. - Raw manifests: the
set_real_ip_fromblock, see Telling the edge who the client is. - Docker Compose: Behind a reverse proxy.
The edge's own log shows the refused connection's peer address, which is the address to trust.
From: Offboarding an agent
The audit log cannot be read
- "The audit log is unavailable right now": inventory-svc, which keeps the log, is not answering, or its audit database is not migrated yet. The hub refuses audited actions in that state too. Try again once inventory-svc is up.
- "You are not permitted to read the audit log": your roles no longer
hold
audit.viewglobally. - "The filters were refused": the reason follows, for example a time the hub could not read.
From: Reading the audit log
A service will not start: the database lock is held
If the service crashes or is forcibly terminated while a migration is in progress, the database lock might remain held, preventing subsequent starts or migrations.
To forcibly release the lock, use the unlock command:
fleeger db unlock --forceEach service has the command (<service> db unlock --force; inventory-svc's
releases its own database's lock and the audit database's). The unlock is
journalled in the hub-wide audit log, written
before the lock is released; when the audit log cannot take the row (the audit
database not migrated yet, or SurrealDB unreachable), the command refuses and
releases nothing. Only when that is the case, and you are sure, add
--without-audit: the lock is then released after a warning on stderr, and
the row is tried once more afterwards (best effort; when that fails too, the
warning prints the row as its only record). This is how the audit database's
own lock is released when it got stuck before its first migration committed:
inventory-svc db unlock --force --without-audit, then
inventory-svc db migrate.
[!WARNING] Only use this command if you are certain that no other instance is currently running a migration. Forcing an unlock while a migration is actually in progress can lead to database corruption.
Known issues in 1.0.0-rc.5
- A v2 archive on rc.4 reads as "not a Fleeger backup archive". rc.4 cannot tell a newer format from a file that is not a backup (an archive over 256 MiB is refused for its size first). rc.5 and later say made by a newer Fleeger; restore it there for any future format.
- An archive holds at most 64 GiB of plaintext (the limit of AES-GCM for
one value; the base64
.fbkfile is about 85 GiB at that point). A larger backup, in practice one with a very large metrics history, breaks off there, and inventory-svc logsencrypt backup. - A store answer that ends cleanly but short (the store stops early without an error) is taken as the whole part; the archive cannot tell it from a complete one. Check a restored platform as in the restore drill.
- Safari, iOS and plain HTTP still collect a download in the tab's memory , which may not fit a large backup with metrics.
- A download fails if the browser stops the download worker, or a Fleeger update retires it, mid-download; the page says so. Start it again.
- Logs that rc.4 did not purge stay in Loki: those of locations offboarded
before the upgrade, and of resources reaped outside rc.4's window or while
the hub's clock ran ahead of Loki's. Loki's own
retention_perioddrops them. - Restoring a backup from an older release** still imports that release's schema definitions and migration stamps, as the rc.4 notes describe. rc.5 added no migration, so a backup taken on rc.4 brings back rc.5's own schema.
- Unchanged from rc.4: the issues still open from the
rc.4 notes, including
resource_countafter a restore.