Zum Inhalt springen
Produkte

// Troubleshooting

Troubleshooting

Common problems when running Fleeger and how to resolve them. Each section comes from the guide it belongs to, linked at its end.

A location is unreachable

Work the path from the location outward:

  1. Is the spoke-agent running at that location? (in dev, the host process / its config). A stopped agent is the most common cause.
  2. Can the agent reach inventory-svc to list its resources, and the targets (Proxmox / iDRAC / k8s API)? Check the agent log.
  3. Can it reach NATS, and are its credentials scoped to this location? A mis-scoped credential is broker-denied (see the NATS layout in the metrics pipeline).
  4. Is ingestion-svc up and consuming? If other locations are fresh, the hub is fine and the problem is at the location.

From: Checking whether a location is reachable

Offboarding succeeds but the agent stays connected

auth-svc revokes the credentials by connecting to the NATS broker over the same address it hands agents (nats.publicUrl in the Helm chart, [nats] public_url in auth-svc's config). When that is a wss:// URL through an ingress, a load balancer or a tunnel that the edge does not trust, the edge answers that connection with a 503 ("Fleeger is behind a proxy the edge does not trust…"). Offboarding (or deleting the location) still succeeds, because auth-svc has recorded the revocation, but the agent stays admitted: it can keep connecting and publishing. auth-svc logs the failed push at WARN (... marked revoked, but the push failed; the reconcile will push it), and its background reconcile keeps trying to push it, failing the same way.

The reconcile uses that address on every pass, so with this misconfiguration it has been failing since auth-svc started, before anyone offboards anything: its first few failures are logged at WARN, later ones at ERROR. Repeated reconcile errors in auth-svc's log are the early sign.

Fix the trust setting. The next reconcile pass, within about half a minute, pushes the revocation and disconnects the agent; offboarding again does no harm.

The edge's own log shows the refused connection's peer address, which is the address to trust.

From: Offboarding an agent

The audit log cannot be read

  • "The audit log is unavailable right now": inventory-svc, which keeps the log, is not answering, or its audit database is not migrated yet. The hub refuses audited actions in that state too. Try again once inventory-svc is up.
  • "You are not permitted to read the audit log": your roles no longer hold audit.view globally.
  • "The filters were refused": the reason follows, for example a time the hub could not read.

From: Reading the audit log

A service will not start: the database lock is held

If the service crashes or is forcibly terminated while a migration is in progress, the database lock might remain held, preventing subsequent starts or migrations.

To forcibly release the lock, use the unlock command:

fleeger db unlock --force

Each service has the command (<service> db unlock --force; inventory-svc's releases its own database's lock and the audit database's). The unlock is journalled in the hub-wide audit log, written before the lock is released; when the audit log cannot take the row (the audit database not migrated yet, or SurrealDB unreachable), the command refuses and releases nothing. Only when that is the case, and you are sure, add --without-audit: the lock is then released after a warning on stderr, and the row is tried once more afterwards (best effort; when that fails too, the warning prints the row as its only record). This is how the audit database's own lock is released when it got stuck before its first migration committed: inventory-svc db unlock --force --without-audit, then inventory-svc db migrate.

[!WARNING] Only use this command if you are certain that no other instance is currently running a migration. Forcing an unlock while a migration is actually in progress can lead to database corruption.

Known issues in 1.0.0-rc.5

  • A v2 archive on rc.4 reads as "not a Fleeger backup archive". rc.4 cannot tell a newer format from a file that is not a backup (an archive over 256 MiB is refused for its size first). rc.5 and later say made by a newer Fleeger; restore it there for any future format.
  • An archive holds at most 64 GiB of plaintext (the limit of AES-GCM for one value; the base64 .fbk file is about 85 GiB at that point). A larger backup, in practice one with a very large metrics history, breaks off there, and inventory-svc logs encrypt backup.
  • A store answer that ends cleanly but short (the store stops early without an error) is taken as the whole part; the archive cannot tell it from a complete one. Check a restored platform as in the restore drill.
  • Safari, iOS and plain HTTP still collect a download in the tab's memory , which may not fit a large backup with metrics.
  • A download fails if the browser stops the download worker, or a Fleeger update retires it, mid-download; the page says so. Start it again.
  • Logs that rc.4 did not purge stay in Loki: those of locations offboarded before the upgrade, and of resources reaped outside rc.4's window or while the hub's clock ran ahead of Loki's. Loki's own retention_period drops them.
  • Restoring a backup from an older release** still imports that release's schema definitions and migration stamps, as the rc.4 notes describe. rc.5 added no migration, so a backup taken on rc.4 brings back rc.5's own schema.
  • Unchanged from rc.4: the issues still open from the rc.4 notes, including resource_count after a restore.

From: Upgrading a running Fleeger hub to 1.0.0-rc.5

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.