Zum Inhalt springen
Produkte

// Upgrades

Upgrading a running Fleeger hub to 1.0.0-rc.3

From: 1.0.0-rc1, or 1.0.0-rc.2 (rc.2 was published without upgrade notes, so everything since rc1 is covered here). To: 1.0.0-rc.3.

Each step is tagged [required] or [if …]. ## Summary

Area Change Manual work
IaC removed The IaC feature is gone: the Terraform tab, plan review and approval, config-repo editing, Terraform state custody (SurrealDB and S3), IaC webhooks, the git/secret credential kinds in the UI, the iac_managed flag and the control.approve permission. None.
Migrations renumbered The IaC migrations were deleted and the rest renumbered: inventory 0000–0031, control 0000–0001, as rc.2 shipped them. rc.3 adds inventory 0032–0037, alerting 0012–0015, auth 0011–0012 and ingestion 0002–0003 on top; control stays at 0001. keystone only compares file names against the newest applied one, so a database created before the IaC removal cannot be upgraded by simply starting rc.3: it would silently skip every future migration. The only such installation was migrated by hand on 2026-10-08. None for databases created after the IaC removal or already migrated.
SurrealDB server The server must be v3.2.4. The backup image now ships the 3.2.4 CLI; the services' client library has been 3.2.4 since rc1. [if] still on 3.1.x: upgrade the server first (§3).
TLS connections rc1 panicked on any TLS connection made through async-nats or the SurrealDB client (both rustls backends were compiled in, neither installed). Fixed. [if] you worked around it (e.g. kept a plaintext public_url): you can switch to TLS now.
Images CI now publishes every image as <semver>-rc.<n> (1.0.0-rc.2, 1.0.0-rc.3, …) plus latest-rc. The tag format changed from rc1's 1.0.0-rc1. Images no longer contain Terraform, git or openssh-client. Pull the published tag instead of building; pin it, never latest-rc.
Inventory requests inventory-svc used to answer agents' and control-svc's NATS requests strictly one at a time, so one slow or stuck request stalled every location. It now takes them one at a time per location, locations in parallel, and gives up on a request 60 s after it arrived (10 minutes for discovery and hardware reports), with an ERROR log line naming the subject. control-svc's resource lookups and the core agent's location list no longer queue at all. The deadline includes the time a request waits for its turn, and that changes what a location with very large discovery reports sees. A location's discovery reports take turns in a line of their own, so they do not hold up its other requests. Where a location's reports take more than 10 minutes per collection cycle, the ones at the end are cut short every cycle and their discovered objects are no longer refreshed (and are removed if discovery retention is enabled). None. A stalled location now shows in inventory-svc's log (missed its … deadline).
Proxmox guests of two clusters Discovered Proxmox VMs and CTs are keyed on their cluster's name as well as their VMID (migration 0032). Before, two clusters in one location that both had VM 100 shared one resource, whose name and parent flipped with every collection: one of the two guests was never listed, and a control action through it reached whichever cluster had reported last. The agent reads the cluster's name from /cluster/status (Sys.Audit, which PVEAuditor has); a node in no cluster keys its guests on its own registration. An existing resource keeps its id, checks and alerts: it goes to the cluster it was last reported through, and the other cluster's guest gets a new resource. A node that joins a cluster, or a renamed cluster, keeps its guests' resources the same way. If the agent cannot read /cluster/status, the node's resource shows that as its collect error and its guests are not updated. [if] two clusters in one location shared guests: check that each shared resource's checks and alerts now sit on the cluster you meant, and move any that do not. [if] two clusters in one location have the same name: rename one, or their guests still share resources. Upgrade the agents: an rc1 or rc.2 agent keeps the old key.
Kubernetes objects of two clusters Discovered k8s nodes and objects are keyed on their cluster as well as their name, as Proxmox guests are. Before, two k8s clusters in one location shared one resource for every same-named node or object (every default and kube-system namespace, for one), whose status and parent flipped with every collection, and one cluster's object was never listed. The agent identifies a cluster by the UID of its kube-system namespace, taken from the namespace list it already reads (list on namespaces; a get of kube-system only when the list fails), so the key survives agent restarts and kubeconfig or address changes. An existing resource keeps its id, checks and alerts: it goes to the cluster it was last reported through, and the other cluster's object gets a new resource. One limitation: if one cluster is registered twice in a location, the first upgraded collection can create new resources for the registration that reports first, and the old resources, with their checks and alerts, are left behind (and are removed if discovery retention is enabled). The same holds for a Proxmox cluster registered through two of its nodes. If the agent cannot read kube-system's UID, the cluster's resource shows that as its collect error, and its objects are no longer refreshed (and are removed if discovery retention is enabled). Metric series, alerts and pod logs are keyed on the cluster as well since see "Metric series of two clusters". [if] two k8s clusters in one location shared objects: check that each shared resource's checks and alerts now sit on the cluster you meant. [if] one cluster is registered twice in a location: remove the duplicate registration before upgrading, or re-attach its objects' checks and alerts to the new resources afterwards. [if] you run the agent with a narrowed ClusterRole: keep list on namespaces, and get on the kube-system namespace, or every Kubernetes action that names its cluster is refused (the shipped roles have both). Upgrade the agents: an rc1 or rc.2 agent keeps the old key.
Kubernetes actions of two clusters The location page sent every Kubernetes action (scale, restart, delete pod, cordon, reissue) to the location's first registered k8s cluster, so with two clusters, deleting cluster B's default/nginx deleted cluster A's. Each action now goes to the registration its object belongs to: a discovered object's parent, or for a pod the registration whose discovered objects carry the pod's cluster label (the one whose objects of that cluster were discovered most recently; the resource list gains read-only discovery_scope and last_discovered fields for that). It also names the cluster it is meant for (k8s-cluster:<uid>, a new cluster param of the k8s command, in the audit log), and the rc.3 agent refuses it, changing nothing, when the registration's kubeconfig reaches a different cluster or cannot read kube-system. An action whose cluster cannot be told is not sent and the page says why. That includes, even with a single k8s cluster, a pod action while no discovered object carries the pod's cluster yet: the first cycles after the upgrade, with an inventory-svc older than rc.3, or while discovery fails. With more than one k8s cluster, it also includes every pod action until the agent is upgraded, since older agents send pods without a cluster label. An older agent ignores the cluster param and acts on whatever the kubeconfig reaches. [if] more than one k8s cluster in a location: upgrade the agents, both to act on pods and for the agent's cluster check.
Metric series of two clusters Proxmox and Kubernetes metric series and pod logs carry a cluster label, the discovery scope of the "Proxmox guests of two clusters" and "Kubernetes objects of two clusters" rows (cluster:<name>, resource:<id> for a Proxmox node in no cluster, k8s-cluster:<uid>), and the hub tells resources apart by it. Before, two clusters in one location wrote one series for their VM 100, node pve1 or pod default/nginx: panels mixed the two, one going dark was hidden by the other's last-seen, they shared one alert, and the retention reaper purging one deleted the other's data. Resource ids (vm-<vmid>, node-<name>, pod-<ns>-<name>, workload-<ns>-<name>) are unchanged. ingestion-svc keys resource_state on the cluster too (migration 0002); the resource_unreachable alert, threshold-alert labels, Loki streams, the reaper's purges and the dashboard include it. If the agent cannot read the cluster in a cycle, it keeps the last one it read through the same address (and kubeconfig), across restarts too: the memory is a file in FLEEGER_STATE_DIR, by default the creds file's directory. The shipped core-agent deployments now set it: a new named volume core-agent-state in compose, and an emptyDir in the chart and the raw manifests, which survives container restarts but not a pod rescheduled to another node. A rotated kubeconfig token counts as a new address. With nothing remembered (the first cycles after this upgrade, an edited registration or token, a rescheduled core-agent pod), a cycle that cannot read the cluster sends none of that Proxmox node's or Kubernetes cluster's metrics and pod logs (its collect status says why; held-back pod logs are lost for good), for at most 2 cycles in a row (60 s at the default interval): the 3rd settles on registration:<inventory id>, so an rc.3 agent never sends Proxmox or Kubernetes series unlabeled and a cluster it cannot scope is still monitored apart from a scoped one. A Proxmox node that joins a cluster, or a registration that settled on registration: and later reads its cluster, moves every one of its series (each VM, node and pod) to a new identity: the old ones fire resource_unreachable like a removed resource's until the stale ceiling retires them, or, with the ceiling at 0, until the retention reaper deletes their rows. Across the upgrade: existing series are not relabelled; they stop growing, so a graph spanning the upgrade shows such a resource as two lines, the old one ending, and for the VictoriaMetrics lookback window (5 minutes by default) the dashboard tables list each such VM, node and pod twice, once without a cluster. The first labeled message for a resource marks its unlabeled resource_state row superseded: it is no longer listed, so it does not go stale and fire resource_unreachable, and an alert firing on it is carried over to the labeled resource, not cleared and raised again. The row is kept for the retention reaper, which purges it with the unlabeled series and log streams once the retention policy's window for the resource passes. With the reaper disabled the row stays, hidden, for good, which is harmless, and --retentionPeriod drops the series. A threshold alert firing on an unlabeled series is carried over to its labeled twin the same way, and the unlabeled series is ignored while its twin shows in query results, so nothing fires twice; if the twin drops out of the results first (a rule whose query filters out non-breaching series), the old series can fire or recover once under the old identity. With two clusters sharing a resource id, the bridge cannot tell whose the shared pre-upgrade alert was: it moves to whichever cluster's resource reports or breaches first, and a resource that was down across the upgrade gets no resource_unreachable until it has reported once with its cluster. Pod logs shipped before the upgrade have no cluster and no longer show under the pod. If the cluster cannot be read on the first rc.3 start and the poll interval is 45 s or more, the 2 held-back cycles can outlast heartbeat_stale_secs (90 s by default), so the unlabeled rows, not yet superseded by a labeled message, can fire resource_unreachable once before the first labeled message replaces them. A hub upgraded before its agents sees no change until they are upgraded; an older hub ignores the label. Alert summaries and notifications name the cluster rather than print the label: cluster:<name> as the name, k8s-cluster:<uid> as its first 8 characters, and registration:<id> or resource:<id> as that inventory resource's name or address; a threshold alert named that way also names its location, e.g. cpu on vm-100 (pve) at 'Site A'. Fingerprints, the alert's cluster label and mutes keep the raw label. The UI names them the same way. [if] a custom threshold rule or Explorer panel aggregates by resource_id (e.g. sum by (resource_id)): add cluster to its by clause, or it still merges two clusters. A resource mute without a cluster still covers every cluster's vm-100: see Resource mutes scoped to a cluster. Upgrade the agents: an rc1 or rc.2 agent keeps writing unlabeled series.
Resource mutes scoped to a cluster A resource mute (POST /api/mutes) takes an optional location and cluster, the alert's labels of those names, and then silences only the resource of that id in that location and cluster: when both are set, both must match. Before, it matched resource_id alone, so muting one cluster's vm-100 also muted the other's. GET /api/mutes lists both fields. A blank location or cluster, one with leading or trailing whitespace or over 255 characters, or either one on a rule or location mute, is refused with 400. Migration 0015 adds the fields. Across the upgrade: existing mutes have neither and keep matching the id in every location and cluster, as before, so nothing is unmuted. A mute scoped to a cluster never matches an alert without a cluster label. Only Proxmox and Kubernetes resources have one, and only from rc.3 agents, so a cluster-scoped mute on a check or iDRAC id never matches. The UI's quick mute mutes a rule and is unchanged. [if] you mute Proxmox or Kubernetes resources through the API and run two clusters or locations with the same resource ids: pass location and cluster on new resource mutes. Check ids are unique, so leave check mutes unscoped.
Alerts of a deleted resource Deleting a registered resource (a Proxmox node, a Kubernetes cluster, an iDRAC host, a check) now drops its metric-derived state, and that of the resources discovered under it, at once; deleting a discovered resource on its own clears nothing, as the agent keeps collecting it through its parent. The agent stamps every metric and pod-log message with the inventory id of the registered resource it was collected through (an envelope field, not a series or stream label: no series, alert or Loki stream changes identity). ingestion-svc stores it on resource_state (migration 0003), and after a successful delete the gateway calls its new DELETE /state/inventory/{id}, which removes those rows and ignores messages still carrying the id for 5 minutes, so an agent cycle in flight does not bring them back. Before, about 90 s after deleting a Proxmox node with N guests resource_unreachable fired N+1 times and stayed until the stale ceiling retired the alerts; now nothing fires, and an alert already firing for one of them is cleared as "no longer monitored". If ingestion-svc is unreachable during the delete, the delete still succeeds (the gateway logs a warning) and the gateway retries the call in the background for about 10 minutes, stopping if the resource exists again; only if every retry fails, or the gateway restarts meanwhile, does the state age out through the stale ceiling as before. A full checkpoint rollback does the same for each resource it removes, retrying a failed call the same way, and for each one it re-creates under its old id it calls the new DELETE /state/inventory/{id}/tombstone, so the resource is no longer held back by its delete's 5-minute tombstone (a recreated check reports on its next run; a recreated Proxmox node, Kubernetes cluster or iDRAC host comes back without its credential and reports nothing until one is attached again); the rollback's summary gains deleted_ids and recreated_ids for this. A rollback that fails changes nothing (it runs in one transaction), so its error carries no ids and the gateway acts only on a successful one's. A failed tombstone lift is not retried: the recreated resource's state then shows once the tombstone expires. An inventory-svc older than the gateway sends no ids, and a rollback then ages out as before. Across the upgrade: rows written by an agent older than rc.3, and resources deleted before the upgrade, carry no inventory id and age out as before; a row gets its id with the upgraded agent's next message. A hub upgraded before its agents behaves as before until they are upgraded; an older hub ignores the field. [if] you rely on deletes clearing alerts at once: upgrade the agents.
NATS permissions Unchanged since rc1. New gen-auth.sh import rebuilds an nsc store from a deployment's Secret, so fix-perms works when the store was never kept. [if] rc1's fix-perms step was never done: do it now (§4).
Discovery reports One discovered object that inventory-svc could not store used to end the whole discovery report: every object after it was left unrefreshed that cycle (and removed if discovery retention is enabled), and the agent only logged a warning. The hub now skips such an object, stores the rest, retries a write conflict once, and answers with how many objects it did not store and which failed first. An object of a kind discovery may not report is skipped the same way instead of refusing the report; a database that cannot be reached at all still ends the report. The agent shows a report with unstored objects, or one the hub answered with an error, as the Proxmox node's or Kubernetes cluster's collect error, e.g. 1 of 40 discovered objects were not stored (first: proxmox_vm '101' could not be stored); the database's reason is in inventory-svc's log. None. Upgrade the agents to see the failures: an older agent ignores the count.
Discovery reports off the collect cycle The agent waited for each discovery report's answer on the collect cycle, for at most async-nats's default 10 s, while the hub allows a report up to 10 minutes. A report the hub took longer over (a large location, or one queued behind the location's other requests) lost its answer, so the unstored-object count never reached the agent, and every such cycle sat for 10 s. The agent now queues discovery reports for a background task per location, which sends them one at a time once the cycle has made its last hub request, and waits up to the hub's deadline for each answer (hardware reports also wait that long now); a newer report for the same Proxmox node or Kubernetes cluster replaces one still waiting, and a deleted one's report is dropped. inventory-svc runs a location's discovery reports in a lane of their own, so they no longer hold up the location's resources, reveal and collect_status requests (they still take turns among themselves; hardware reports keep their place in the location's line). The collect cycle shows the outcome of the last answered report: a discovery failure appears in the collect error from the first cycle after the hub answers, and stays until a report with no failures, or a cycle with nothing to report, clears it; a report whose answer never arrives leaves it standing. A restarted agent shows no discovery failure until its first report is answered. None. Upgrade inventory-svc and the agents: with an older inventory-svc, a long discovery still holds up the location's other requests, though no longer the same cycle's.
Discovered objects with an empty name The first report of a discovered object with an empty name (which is its address, and may not be empty) stored nothing, yet the hub counted it as stored, so nobody was told. The hub now counts such an object as not stored, as it already did when the object existed before, and its Proxmox node or Kubernetes cluster shows a collect error, e.g. 1 of 40 discovered objects were not stored (first: proxmox_vm '101' could not be stored). None. A collect error may now appear on a node or cluster whose source reports an object without a name; give the object a name at the source. Only rc.3 agents show the error.
Discovered objects with a blank name A discovered object whose name was only whitespace (e.g. " ") passed the check that refuses an empty name, so it was stored with a blank address and showed as nameless in the UI and in series names. A resource's address may no longer be blank: such an object is now counted as not stored, as an empty-named one is, and the upgrade deletes the discovered objects already stored with a blank address, with their hardware inventory. The API already refused a blank address, so registered resources are unaffected. Metric series already carrying a blank name stay until metric retention removes them. None. Discovered objects with a blank name disappear on upgrade, and their node or cluster shows a collect error while the source still reports them; give the object a name at the source.
Resource edits and agent reports A resource edit (PUT /api/resources/{id}) or a checkpoint rollback restoring a resource in place read the resource, then wrote all of it back. A collect or hardware stamp, a discovered object's new status, or discovery adopting the object into its cluster's scope, committed in between, was undone until the next report, and another edit committed in between was lost. They now write only the fields an edit can change, and an edit whose fields changed meanwhile is made again from a fresh read: both edits apply, or the later one is refused (400) if it is not valid with the other. An edit that keeps losing that race answers 409 it kept changing during the update, new in rc.3. A location edit (PUT /api/locations/{id}) and a rollback's restore of a location had the same flaw: of two edits of different fields, the later one put back the earlier one's field as it had read it, and a rollback put back a local_collector set meanwhile. They now also write only the fields they set, and an edit that loses a write conflict is made again instead of answering 500. None.
Stale check fields Migration 0035 clears check_vantages and check_public on every resource that is not an uptime check. Only a resource that changed kind before the write path refused them could still hold them. No agent was served it through those vantages and the status page did not list it, but a credential set on it was refused (400 remove this check's extra vantage points before attaching a credential), and a caller with a location-scoped role needed the permission at each stale vantage to edit it. Such a resource now takes a credential and edits like any other. Changing it back to a check does not revive the cleared fields, as an edit of it already did not, and a checkpoint rollback no longer restores a stale check_public on such a resource either. None.
Location name taken Renaming a location to a name another location has (PUT /api/locations/{id}), or rolling back to a checkpoint whose location name another location has taken since, answered 500 An internal error occurred. Both now answer 409 location name '<name>' is already in use and change nothing. Creating a location with a taken name still answers 409, now with that message; before, its message was the database's, which named the other location's record id. A create that fails for any other database reason now answers 500, where every such failure used to be a 409. [if] a script reads a 500 on a location rename as a name clash, or parses the create's 409 message: match the 409 and the new message.
The meta location The hub's own meta location (self-monitoring) was stored without a creation time or a local_collector value, so reading it (GET /locations/meta), editing it and rolling it back to a checkpoint all failed with 500. The upgrade fills both in (local_collector false), and the meta location now reads, edits and rolls back like any other. None.
Rollbacks restore local_collector Rolling a location back to a checkpoint now also restores its local_collector flag; before, it kept the current value. A rollback past a change of that flag therefore moves collection between the hub's bundled core agent and the location's own agent. A checkpoint taken before the flag existed (migration 0014) does not record it and leaves it as it is. [if] you rely on a rollback keeping the current local_collector: set it again after rolling back.
Rollbacks restore allow_invalid_certs A checkpoint rollback that restores a resource in place now also restores its allow_invalid_certs flag, as recreating a deleted resource already did; before, it kept the current value. This can turn TLS certificate checking back off: a resource that skipped certificate verification at the checkpoint, and has verified it since, skips it again after the rollback. A checkpoint taken before the flag existed (migration 0011) does not record it and leaves it as it is. [if] you turned certificate checking on for a resource after a checkpoint you may roll back to: check the resource's TLS setting after rolling back, or take a new checkpoint.
The meta location is fixed The hub's own meta location (self-monitoring) is now editable except for its name and kind, and can no longer be deleted. An edit changing its name or kind is refused with 400 (one sending meta for either is accepted); its region, network config and local_collector edit as on any location. DELETE /api/locations/meta is refused with 409, before the gateway revokes any agent credentials for it: before, it deleted the location, and nothing recreated it. A checkpoint rollback of it leaves its name and kind and restores the rest. The location page shows its name and kind read-only and offers no delete for it. [if] a script edits the meta location: leave its name and kind out, or send them as meta.
No onboarding of meta auth-svc no longer mints agent credentials for the hub's own meta location, whose agent cannot be offboarded: POST /api/agent-tokens for meta is refused with 400, as _iac and _core are, and so is auth-svc mint-creds --location meta. A meta token minted before the upgrade and not yet used is refused at POST /api/bootstrap with 400 and mints nothing. The UI no longer offers Onboard for meta (on the location's Agents tab or the Services page's Agents tab). Only the exact id meta is refused. A meta agent registered before the upgrade keeps its credential: nothing is revoked automatically, and it cannot bootstrap again with a new token. [if] you onboarded an agent for meta: stop it, and revoke its credential on auth-svc directly, as Revoking a meta credential shows; the API no longer revokes it.
mint-creds checks the location id auth-svc mint-creds --location <id> now refuses a location id that POST /api/agent-tokens refuses: anything but 1 to 50 ASCII letters, digits, _ and -. It exits non-zero with mint failed: location must be a valid location id and mints nothing. Before, it minted a credential for such an id. A stored registration token for such an id, which the API never creates, is refused at POST /api/bootstrap with 400. None.
mint-creds prints only recorded credentials auth-svc mint-creds now prints a credential only after its user key is written to the revocation ledger. When it cannot connect or sign in to the database, or the database refuses the row, it exits non-zero with a message that the credential was not printed. Before, it logged a warning, or for a refused write nothing, and printed a credential that offboarding the location cannot revoke. [if] you minted any credential with mint-creds before rc.3 that is still in use: check its user key in the ledger and add the row if it is missing, as Recording a credential minted before rc.3 shows.
No offboarding of meta Offboarding the agent of the hub's own meta location (DELETE /api/agents/meta) is refused with 409 before anything is purged or revoked, as deleting the location is. Before, it was accepted: it purged the hub's own reachability state and self-monitoring metrics and revoked any meta agent credentials. The UI no longer offers an Offboard button for meta (on the Services page's Agents tab or the location's own Agents tab). None.
No offboarding of reserved pseudo-locations Offboarding (DELETE /api/agents/{id}) or deleting (DELETE /api/locations/{id}) one of the reserved pseudo-locations _core, _iac and _ingestion (fleeger_metrics::RESERVED_LOCATIONS) is refused with 400 before any upstream is called. This covers every spelling auth-svc already refuses to mint credentials for: other case, or one that sanitizes onto them such as .iac. Before, both were accepted (the location delete because inventory-svc's delete check passes a missing location): each ran a metrics purge and a credential revoke for a location no agent can have. None.
Revision history order Two edits of the same location or resource made at about the same time (or a credential attach and a detach) could be listed in a location's revision history in the opposite order from the one they were applied in, so the newest revision, which a checkpoint captures, could show a state the row no longer had. These edits now write their revision in the same database transaction as the change, so each location's and each resource's revisions follow the order that entity's changes were applied in (per entity, not across entities). An edit whose revision cannot be written is no longer applied: it fails with 500 and changes nothing, where before the change was applied and the request still answered 500. Revisions written before the upgrade keep their order. None.
Creates, deletes and credential updates journalled with the change Creating or deleting a location, resource or credential, and renaming a credential or rotating its secret, now write their revision in the same database transaction as the change, as edits already did. Before, the revision was written after the change committed: an edit of a just-created row could be listed as older than its create, two credential renames at about the same time could leave the older name as the newest revision, and a change whose revision could not be written was applied while the request answered 500. Such a change now fails with 500 and changes nothing. A credential rename together with a secret rotation (PUT /api/credentials/{id} with both) is now one write, applied together or not at all, and is retried when it loses a write conflict to a concurrent change of the credential (before, such a request could answer 500). A delete of a location that does not exist no longer writes a delete revision for it (it still answers 204). None.
Dead config [iac] in control-svc.toml, the chart's iac.* values, compose's IAC_GIT_SSH_KEY mount and any IaC git SSH Secret are ignored. The edge's /api/control/resources/{id}/state route is gone. Optional cleanup; nothing breaks if left.
Rekey and concurrent edits service rekey (inventory-svc and alerting-svc) now writes a re-encrypted secret only while the row still holds the ciphertext it read. Before, a credential rotated or a notification channel edited while a rekey ran was silently reverted to its previous secret, now under the new key, so a second run could not show it. A row edited or deleted meanwhile is left as it now is and counted in the completion line, which now reads rekey complete: N credentials re-encrypted, N already current, N edited or deleted meanwhile (left as they now are) in inventory-svc (was N re-encrypted, N already current) and the same with notification channels in alerting-svc. A row edited under the retired key (a replica not yet on the new key) is re-encrypted from the edit; one re-edited that way on every attempt stops the run with an error naming it. Also, a notification-channel PUT that changes only name or enabled no longer writes back the config it read, which could put a retired-key config back over a rekey's. [if] you edited a credential or notification channel while a rekey ran before rc.3: check it still holds the value you set. [if] a script parses the rekey completion line: match the new wording. [if] a rekey stops with edited under a retired key … run rekey again: run it again.
Credential key rotation alerting-svc has a rekey command too: service rekey re-encrypts every notification channel's secrets under the current FLEEGER_CREDENTIAL_KEY, as inventory-svc's does for credentials. Before, only inventory-svc could rekey, so after a rotation the alert-channel secrets stayed on the retired key, and dropping it made every channel created or last edited before the rotation undeliverable. The chart's key-change refusal and the rotation docs now name both. [if] you rotated the credential key and kept the old one as retired: run service rekey in alerting-svc, until a second run reports 0 ... re-encrypted, before you drop it. [if] you already dropped it: re-enter each notification channel's settings.
Backups after a key rotation Documentation only, no behaviour change. A backup holds the secrets as they were stored when it was taken, and service rekey re-encrypts only the live databases, so any backup taken before both rekeys were confirmed can hold old-key secrets, even one sealed under the new key. That includes alert-channel secrets after a rotation done before alerting-svc had a rekey. A UI .fbk from before the switch is also sealed as a whole under the old key. Restore already opens a .fbk with any retired key supplied in FLEEGER_CREDENTIAL_KEYS_RETIRED, but imports its secrets unchanged. The rotation and backup docs said only to keep the old key until the rekey was confirmed; they now say to keep it as long as any backup taken before both rekeys were confirmed is kept, and how to restore one. [if] you rotated the credential key and keep backups taken before both rekeys were confirmed: keep the old key off-box with them. To restore one, supply it again as retired to inventory-svc and alerting-svc, restore, then run service rekey in both (Backup & disaster recovery).
Restore and retired-key secrets A UI restore (POST /api/backup/restore) now, once its import is over, re-encrypts under the current FLEEGER_CREDENTIAL_KEY every credential in the inventory database still under a retired key (the whole table, as inventory-svc's service rekey would), within what is left of the [backup] timeout_secs budget, and counts the notification channels it brings back under a key other than the current one. Before, it imported them unchanged and said nothing, so they failed at their next use once the retired key was dropped. What is left (channels, which only alerting-svc's rekey re-encrypts, and credentials under a key inventory-svc does not hold) is reported with the rekey hint in the restore page, the response's new warning field and the inventory-svc log; the response also gains credentials_resealed and rekey_needed (credentials, notification_channels; null when that check could not run or finish, with credentials_resealed still counting what was re-encrypted). A restore that stops part way runs the same checks on the databases it did restore and adds the hint to its error. The restore itself still succeeds. The re-encryption uses the handling replica's primary key, so do not restore while replicas run different FLEEGER_CREDENTIAL_KEY values (mid-rotation). scripts/restore.sh is unchanged. [if] a restore shows a warning: keep the old key as retired and run service rekey in each service it names. [if] a script reads the restore response strictly: allow the new fields.
Control Resources that were iac_managed are now directly controllable, gated by control.execute like every other resource. Roles that held control.approve keep a dead entry. [if] you relied on the IaC flag to block direct power actions: review who holds control.execute.
Control actions on the wrong kind of resource POST /api/control/actions now refuses with a 400, and a rejected audit row, a k8s action on anything but a registered k8s cluster, a proxmox_vm action on anything but a Proxmox node and an idrac_power action on anything but an iDRAC host. Before, such an action was recorded as proposed, sent to the agent and failed there, usually as a misleading "credential reveal failed". The UI always sends the right kind. None.
Test control path The location page's Test control path button works. Since rc1 it always failed with resource '_preflight' not found, because control-svc looked up the resource of every action, and the button's ping names none. control-svc now sends a ping on the reserved id _preflight to the agent of the request's location without a lookup, and audits it as proposed and completed (or failed) like any action, so these pings now appear in the audit log. In a location with no reachable agent the button waits command_timeout_secs (20 s by default), then shows "agent did not reply before the timeout (location unreachable?)", and the audit list shows a failed row with the same text. That timeout is now a 504 with that message for every control action, where it was a 500 reading "An internal error occurred", so the location page's VM, iDRAC and Kubernetes actions name it too. A ping on a real resource, and every other action, still needs a resource of that location. [if] a script reads a 500 from POST /api/control/actions as an agent timeout: expect a 504 with code TIMEOUT.
Alert rule API On PUT /api/alert-rules/{id}, a JSON null for stale_secs, description or scope_location now clears the field; before, it was silently ignored. A field left out is still untouched. A blank or whitespace-only scope_location now also means no scope, on POST and PUT, and a scope is stored trimmed; before, either was stored as sent and made a heartbeat rule watch a location of that name, which never reports, so the rule never fired. Such a scope could come from an API client or from spaces typed into the form. Rules already stored that way are read the new way: a blank scope now watches every location, a padded one the location it names. The rule form's "Stale after" box can now be emptied to return a rule to the default window, and a typed 0 is refused instead of being treated as empty. A condition field the rule's kind is not evaluated from (for example query on a heartbeat rule, or stale_secs on check_down or on a threshold rule, the built-in backup_stale, cert_expiring, cert_invalid and workload_degraded included) is now refused with a 400 naming it, on POST and on a PUT that changes it; before, it was stored and never read. A scope_location on location_unreachable or resource_unreachable, which was ignored, is refused the same way. null and for_secs: 0 are still accepted. A threshold rule's for_secs above ~10 years is now refused with a 400, as stale_secs already was; such a rule could never fire. A negative for_secs, which was stored and shown but read as 0, is refused too. Creating a threshold rule without its query, comparator or threshold is refused as well. On the built-in threshold rules (backup_stale, cert_expiring, cert_invalid, workload_degraded), a threshold or for_secs sent on PUT now takes effect; before, it was ignored with a 200. A query or comparator that would change one of them is refused with a 400. A new name for any built-in rule is refused with a 400 too; before, it was ignored with a 200. [if] a script sends null for those fields meaning "unchanged": drop the key instead. Reload any open Fleeger tab after the upgrade: a page loaded before it sends null for every empty description, window and scope box on each rule save (and for a typed 0), which now clears those fields instead of being ignored. [if] a script created heartbeat rules with a blank or padded scope: they can now fire. List them with SELECT id, name, scope_location FROM alert_rule WHERE kind = 'heartbeat' AND scope_location != NONE AND (string::trim(scope_location) != scope_location OR string::trim(scope_location) = '') in the alerting database, and scope or disable any you do not want. [if] a script sends other-kind fields: drop them, or it now gets a 400. Rules that already hold such a value keep working and still accept edits that send it back unchanged; the value stays unused and harmless. null clears a leftover stale_secs or scope_location, and for_secs: 0 a leftover for_secs; a leftover query, comparator or threshold cannot be cleared through the API and can stay. To see them: SELECT id, name, kind FROM alert_rule WHERE (kind = 'threshold' AND (stale_secs != NONE OR string::trim(scope_location ?? '') != '')) OR (kind = 'heartbeat' AND (query != NONE OR comparator != NONE OR threshold != NONE OR for_secs != 0)) OR (kind IN ['location_unreachable', 'resource_unreachable'] AND string::trim(scope_location ?? '') != '') OR (kind = 'check_down' AND stale_secs != NONE) in the alerting database. [if] a threshold rule was given a for_secs past the new maximum: it never fired and still will not; edits that send the value back unchanged are accepted. Find such rules with SELECT id, name, for_secs FROM alert_rule WHERE kind = 'threshold' AND for_secs > 315360000 in the alerting database and lower the window or disable the rule. [if] a script sends a negative for_secs: send 0, which means the same; a rule that already holds one keeps working (it fires on the first breach) and can be set to 0. Find such rules with SELECT id, name, for_secs FROM alert_rule WHERE for_secs < 0 in the alerting database. [if] a script creates threshold rules without query, comparator or threshold: it now gets a 400; such a rule could never be evaluated. [if] a script sends full rule bodies to those four built-ins: the threshold and for_secs it sends are now applied (a for_secs of 0 clears workload_degraded's 10-minute window), and a query or comparator other than the stored one gets a 400. Send only the fields you mean to change. [if] a script sends a name to a built-in rule: send the stored name or none; any other gets a 400.
Threshold alert subjects Threshold alert text changed in two ways. (1) A series with no resource_id label is now named by its own labels instead of series: stream metrics, leaving out __name__, source, service and the hub's own location="meta" (any other location is kept). A series with nothing left reads the hub when hub-wide and the aggregate when it has no labels; label values are stripped of control characters and cut to 64 characters, the name to 200. (2) An alert whose query computes its value, so the series has no metric name, now leads with the value instead of a blank. This covers every such rule, including ones with a resource_id, whose name is kept. Built-ins, before → after: nats_storage_high >= 80 (value 85.000) on series → value 85.000 >= 80 on the hub; nats_stream_near_cap >= 90 (value 95.000) on series → value 95.000 >= 90 on stream metrics; nats_stream_discarding > 0 (value 12.000) on series → value 12.000 > 0 on stream metrics; backup_stale > 129600 (value …) on series → value … > 129600 on the aggregate; workload_degraded > 0 (value 2.000) on k3s → value 2.000 > 0 on k3s (its subject on k3s → k3s). Rules whose query keeps the metric name and has a resource_id read as before. Alert fingerprints are unchanged, so alerts firing across the upgrade are neither duplicated nor re-notified early. Such an alert keeps the old summary in the alert list, and its old subject, until it resolves: its re-notifications carry the new text, but its [RESOLVED] notice reads the old subject ( on series recovered.). [if] you match notification text in automation (a filter on on series or on a leading >=): match the new wording.
Heartbeat rule scope Narrowing a heartbeat rule's scope_location (from every location to one, or from one to another) now retires its firing alerts for the locations it no longer watches with a [CLEARED] … no longer watches <location> notice. Such alerts still firing from a re-scope before the upgrade are cleared the same way on the first heartbeat cycle after it, one notice each. Before, such an alert stayed up and was later resolved with a false "back online" as soon as that location reported anything, stale or not. [if] you route [RESOLVED] notices into automation: a re-scope now sends [CLEARED] for the dropped locations instead.
Alerts of a vanished subject A firing location_unreachable or resource_unreachable alert whose location or resource is no longer in ingestion-svc's state lists (offboarding clears them, and so can the retention reaper) was resolved with a false <subject> is back online.. It is now cleared: the notice is titled [CLEARED] <rule name> and reads <subject> is no longer reported by ingestion — … This is NOT a recovery: if it still exists, it may still be down. A subject still listed recovers as before. A cycle in which ingestion lists no location at all clears nothing. [if] you route [RESOLVED] or "back online" notices into automation: such alerts now send [CLEARED] instead.
Credential attach errors Attaching a credential from another location, by credential_id on POST /api/locations/{id}/resources or PUT /api/resources/{id}/credential, now answers 404 with credential '<id>' not found in this location, the same as an id that does not exist. Before, it answered 400 credential belongs to a different location, so a location-scoped user could tell whether a credential id existed in a location they cannot see. The message for an id that does not exist changed too, from credential '<id>' not found to credential '<id>' not found in this location. [if] a script checks for the 400, its message, or the old not-found message: match the 404 and the new message instead.
Inventory edit API On PUT /api/locations/{id}, a JSON null or a blank region now clears it (a blank used to be stored as "", null was ignored). POST /api/locations likewise stores a blank or whitespace-only region as none, where it stored it as sent. On PUT /api/resources/{id}, a null for name, check_expected_body, check_public_label or check_component now clears it, as a blank string always did; before, null was silently ignored, so emptying these boxes in the UI said "Saved." and changed nothing. A field left out is still untouched. A check whose label is cleared is shown on /status under its resource name, as one created without a label is. A null expected body no longer counts as a set check parameter, so a check whose only stored parameter is its body can change to a non-check kind in the same request; UI-created checks still cannot. The location and resource forms now send these fields only when they were changed. [if] a script sends null for those fields meaning "unchanged": drop the key instead. Reload any open Fleeger tab after the upgrade: a page loaded before it sends null for every empty one of these boxes on each save, which now clears a value set since the form was opened.
Inventory edit API: defaults On PUT /api/locations/{id}, a JSON null for network_config now clears it. On PUT /api/resources/{id}, a null for check_method, check_interval_secs, check_timeout_secs, check_expected_status, check_follow_redirects or check_enabled now returns that check parameter to its default; before, null was ignored with a 200. A field left out is still untouched. The UI sends null only for an emptied Interval or Timeout box. [if] a script sends null for those fields meaning "unchanged": drop the key instead.
Uptime check parameters check_method and check_expected_status on a tcp_check, tls_check or dns_check are now refused with a 400 naming the field, on POST /api/locations/{id}/resources and on a PUT /api/resources/{id} that sends one; before, the request answered 2xx and the value was dropped, as only an http_check reads them. null or a blank is still accepted. A PUT that changes an http_check to another check kind still drops its stored method and expected status, also when it sends them back unchanged; a different value is refused. The UI sends neither for these kinds. [if] a script sends either field for a tcp, tls or dns check: drop it.
Uptime check parameters: body and redirects check_follow_redirects on a tcp_check, tls_check or dns_check, and check_expected_body on a tcp_check or tls_check, are now refused with a 400 naming the field, on POST /api/locations/{id}/resources and on a PUT /api/resources/{id} that sends one; before, the request answered 2xx and stored a value no probe reads. null, or a blank body, is still accepted and stores nothing. A PUT that changes a check to a kind that does not read one of them now drops its stored value (an http_check changed to tcp_check used to keep its body and redirect flag), also when it is sent back unchanged; a different non-blank value is refused. Such a value stored by an earlier release is dropped on the check's next edit; a checkpoint rollback to a snapshot taken before the upgrade can bring one back, which is harmless (no probe reads it) and dropped again on the next edit. A dns_check now stores its expected answer trimmed, and a blank one as none, as the probe already read it. The UI sends neither field for a kind that does not read it. [if] a script sends either field for a check kind that does not read it: drop it.
Uptime check body on a kind change check_expected_body is a substring of the response to an http_check and an exact answer to a dns_check. A PUT /api/resources/{id} that changes a check between the two kinds now drops the stored value, also when it is sent back unchanged; a new value in the same request is stored, and a null clears it. Before, it was kept, so an http_check body such as OK became an expected answer no lookup returns, and the check failed with dns_error until it was edited. The resource form now empties the box when the kind changes between the two. [if] a script changes a check between http_check and dns_check and means to keep the value: send it again in a second request, or send the new kind's value in the same one.
Resource form: check interval and timeout An empty Interval or Timeout box on an uptime check now means the default (60 s and 10 s), shown as its placeholder. Emptying the box on an edit returns the check to the default, and an empty box on create stores no value; before, an emptied box was sent as 0 on create and on edit alike, and the save failed with check_interval_secs must be at least 5 (or the timeout equivalent). A check stored with no value shows an empty box instead of 60 or 10, and a new check left at the defaults is stored without a value rather than with an explicit 60/10: it probes at the same cadence. A check promoted from the Uptime page's discovered candidates is likewise stored without an interval, timeout or expected status. An edit now sends these two boxes only when they were changed, as it does the other clearable boxes. Nothing to do.
Per-location resource access A role granted for one location now reaches that location's resources by id: GET/PUT/DELETE /api/resources/{id}, its /hardware and its /credential. The gateway looks up the resource's location in inventory-svc and checks the role there; before, these routes needed the permission globally, so a location-scoped user could list a location's resources but not open, edit or delete one. A scoped user editing an uptime check without naming check_vantages needs the permission at every stored vantage too. A scoped user asking for a resource outside their locations, or one that does not exist, gets 403. Nothing to do.
Per-location credential access A role granted for one location now renames, rotates and deletes that location's saved credentials: PUT/DELETE /api/credentials/{id} are checked at the credential's location, with the same resource.edit/resource.delete as before; before, they needed the permission globally, so a location-scoped user could list and create a location's credentials but not change or remove one. A scoped user asking for a credential outside their locations, or one that does not exist, gets 403. Nothing to do.
Role assignments at a non-location A role assigned at the literal location "*" granted nothing on any permission check, yet the user read every location's inventory, state, metrics and logs, because the visible-locations list used "*" for "all locations" too. The gateway now decides "all locations" by asking whether the permission is held globally, and keeps only real location ids from the list; auth migration 0011 redefines the list function the same way. The IAM API (POST /api/users, POST /api/users/{name}/roles) now refuses with 400 an assignment location that is not a location id ([A-Za-z0-9_-]{1,50}, sent as is, not trimmed) or that is meta, _core, _iac or _ingestion (any case), "*" and "" included; for a global assignment leave location out or send null. An assignment with a key other than role and location is refused with 422, so a misspelt Location is no longer stored as a global grant. Across the upgrade: assignments already stored at such a location are not migrated: they grant nothing, a user whose only assignment is one of them now sees no location (403), and they stay listed and removable with POST /api/users/{name}/unassign. [if] you assigned a role at "*" (or another non-id) to mean "everywhere": assign it globally instead, and unassign the old one. To find them, in the auth database: SELECT in, out, location FROM roles WHERE location != NONE AND !fn::is_assignable_location(location). [if] a script assigns roles: send only role and location.
Heartbeat window A heartbeat rule's stale_secs must now be at most heartbeat_alert_ceiling_secs minus two eval_interval_secs (604740 s with the defaults) while that ceiling is on. A wider window used to switch off the retirement of silent locations for every heartbeat rule, with only a log warning. POST /api/alert-rules, a PUT that changes the window, and a PUT with enabled: true on a disabled rule are refused (400) beyond that. Rules already stored with a wider window keep working as before; alerting-svc names each of them in a warning at startup, except a window over ~10 years, which is evaluated at the default instead. [if] you use heartbeat windows of about a week or more: raise heartbeat_alert_ceiling_secs above the window plus two evaluation cycles, or scripts creating or re-enabling such rules get a 400.
Heartbeat ceiling margin alerting-svc now refuses to start when heartbeat_alert_ceiling_secs is on and smaller than heartbeat_stale_secs plus two eval_interval_secs. With the default window and interval that is 150 s, below the existing 300 s minimum, so it only matters for a longer window or interval; the shipped 604800 is far above both. Before, only ceiling > heartbeat_stale_secs was required, and a location could be dropped before its outage was reported. Each cycle also evaluates the heartbeat rules before dropping silent locations, so a location whose silence crosses the window and the ceiling in one slow cycle still fires. [if] you set the ceiling within two evaluation cycles of heartbeat_stale_secs, or run a very long eval_interval_secs with retirement on: raise the ceiling, lower the window or interval, or set the ceiling to 0.
Resource ceiling margin alerting-svc now also refuses to start when resource_stale_ceiling_secs is on and smaller than heartbeat_stale_secs plus two eval_interval_secs. With the default window and interval that is 150 s, below the existing 300 s minimum, so it only matters for a longer window or interval; the shipped 1800 is far above both. Before, only ceiling > heartbeat_stale_secs was required, and a resource whose age crossed the window and the ceiling in one slow cycle was retired as gone without resource_unreachable ever firing. Each cycle now also remembers when it last sampled the resource list, so a resource that was not yet stale then still fires once before it is retired, even after a cycle longer than two intervals or under a rule stale_secs close to the ceiling. It fires only while its location is up; while the location is dark it is suppressed and waits, and fires once the location is back. [if] you set resource_stale_ceiling_secs within two evaluation cycles of heartbeat_stale_secs, or run a very long eval_interval_secs with it on: raise the ceiling, lower the window or interval, or set the ceiling to 0.
Permission to create a resource Creating a resource in a location (POST /api/locations/{id}/resources) now needs resource.create and location.edit at that location. Before, it needed location.create, so a role holding resource.* could not create resources and a role holding location.create could. Each extra vantage of a new check now needs resource.create there too, where it needed location.create. Agent tokens, checkpoints and rollbacks still need location.create. Built-in roles are unaffected: admin holds *, operator and viewer create nothing. [if] a custom role creates resources through location.create: add resource.create and location.edit to it (globally or for the same locations), and resource.create at any location it adds as a vantage.
Checkpoint rollback A rollback now restores a check's extra vantage points only at locations that still exist, on a check kind, without a credential. Every vantage point it adds to a check (one the check doesn't hold now) needs the caller's rollback permission (location.create) there, or the whole rollback is refused (403) with nothing written. A rollback that would move an attached credential to another host is refused (400). gateway-svc tells inventory-svc where the caller holds the permission, in a header it strips from client requests. An older gateway-svc does not strip it, so the check holds only once gateway-svc runs rc.3: while gateway-svc is older than inventory-svc (during the upgrade, or after rolling back gateway-svc alone), a client can send that header itself, and a rollback is as unguarded as it was in rc1. [required] Upgrade gateway-svc first and let its rollout finish before inventory-svc's starts (a simultaneous rolling upgrade leaves old gateway pods in front of new inventory pods for a while), and never roll back gateway-svc alone to an older release.
Checkpoint rollback: all or nothing A rollback now writes its whole location in one transaction: it either commits everything or changes nothing. Before, a rollback failing partway left the location half rolled back. While it runs it holds a lease on the location (migration 0037, table rollback_lock, 120 s, renewed while it runs): any other write of that location (a location edit, a resource create, edit or delete, a credential create, attach, detach, rename, rotation or delete) answers 409 with code ROLLING_BACK, and so does a second rollback of it; the agent's discovery reports and per-resource status and hardware stamps of that location are skipped for that cycle, without an error. A rollback that ends without committing (it failed, had nothing to write, or its request was dropped) lifts the lease at once. Deleting the location is not held back: it wins, and the rollback answers 404 having changed nothing. GET /api/locations/{id}/rollback/lock (location.view) shows who holds the lease and until when, and POST /api/locations/{id}/rollback/unlock (the rollback's permission, location.create) lifts it; the location page shows both when a write gets that 409. Its Lift lock button shows for everyone holding location.create at that location, through a global role or one assigned for it; for that, GET /api/auth/me takes an optional ?location={id} and then answers the caller's permissions at that location. A rollback that keeps conflicting with other writes gives up with 409 after 5 attempts, having changed nothing. In the rare case its transaction's answer is lost (inventory-svc's from the database, or the gateway's from inventory-svc), it checks whether it committed where it can, and answers 500 with outcome_unknown: true if it cannot tell: check the location's history (running the rollback again is safe). The gateway reconciles no metric state for such a rollback, so if it did apply, its removed resources' state ages out through the stale ceiling and its recreated ones show no state until their tombstone expires. [if] you script writes to a location: retry a 409 ROLLING_BACK after the rollback finishes.
Deleting a resource deletes what was discovered under it Deleting a registered resource (a Proxmox node, a Kubernetes cluster) now deletes the resources discovered under it (VMs and CTs, Kubernetes objects) and their hardware inventory in the same transaction, and the confirmation says how many go with it. Before, they stayed listed with their last status until discovery retention removed them (never, by default). A credential attached to one survives as an orphan, as for any deleted resource. A full checkpoint rollback that removes a resource removes its discovered resources the same way. Registering the resource again brings them back as new resources with new ids. (Uptime checks are registered resources of their own, never discovered under another, so a delete takes none with it.) inventory-svc now also refuses a discovery report whose parent is not a registered resource of the reporting location, so an agent cycle that was running during the delete cannot put them back; a report committing at the same moment as the delete is serialised with it too. Deleting a location now deletes the location, its resources with their hardware inventory, and its credentials in one transaction, so a report committing as it does, with or without a parent, is serialised with it too. Discovered resources left behind by deletes before the upgrade are not removed, nor is the hardware inventory left behind by location deletes before the upgrade (it is never shown). The delete now also drops their alert state: see Alerts of a deleted resource. [if] you deleted registered resources before upgrading: delete the discovered resources left behind under them, or let discovery retention remove them.
SurrealDB password (compose) The compose stack passes the SurrealDB root user and password to the server through its environment instead of --user/--pass, so they no longer show in ps or /proc/<pid>/cmdline on the host. backup.sh and restore.sh (the backup image) likewise hand the password to surreal export/import in its environment. They still show in docker inspect and to root. The server uses them only when the datastore is first created, so an existing datastore keeps its root password, and changing SURREAL_PASS in .env still does not rotate it. docker compose exec surrealdb /surreal sql now logs in from that environment without --user/--pass. Helm and the raw manifests already used the environment. [required, compose] docker compose up -d recreates the surrealdb container once, so restart inventory-svc afterwards (§2). [recommended, compose] Rotate the root password, which local users could read until now (§2).
Raw manifests: proxy in front of the edge The edge now answers 503, saying what to fix, to every request that came through a proxy it does not trust: one carrying X-Forwarded-For or X-Real-IP whose client address it did not take, because set_real_ip_from in the fleeger-edge ConfigMap is unset or misses the proxy. Before, an Ingress without it silently made the login rate limit one bucket for every client and put the controller's address in the audit log. The Helm chart already refused that. A wrong CIDR is caught as well as a missing one. Requests without those headers (port-forward, NodePort, in-cluster clients) are unchanged; a client that sends one itself, e.g. through an office HTTP proxy to a NodePort, now gets the 503. [if] a proxy or Ingress is in front of the edge: set set_real_ip_from before rolling out (§1 step 5).
Compose: proxy in front of the edge The compose edge now does the same: it answers 503, saying what to fix, to every request that came through a proxy it does not trust, i.e. one carrying X-Forwarded-For or X-Real-IP whose client address it did not take, because set_real_ip_from in deploy/compose/edge.nginx.conf is unset or misses the proxy. Before, a reverse proxy in front of compose silently made the login rate limit one bucket for every client and put the proxy's address in the audit log. Requests without those headers (the stack reached directly on port 8080) are unchanged; a client that sends one itself gets the 503. [if, compose] a reverse proxy is in front of the edge: set set_real_ip_from in edge.nginx.conf before up -d (§2 step 5).
Helm: wrong or unseen proxy in front of the edge The chart's edge now also refuses at runtime, as the raw manifests and compose do: a request carrying X-Forwarded-For or X-Real-IP whose client address it did not take gets a 503 saying what to set. The render-time check is unchanged and still catches only an empty edge.trustedProxies with the chart's own ingress; the runtime check adds a CIDR that misses the controller, an HTTP proxy or LB from outside the chart, and a waived edge.preflightTrustedProxies. Before, those silently made the login rate limit one bucket for every client. Requests without those headers are unchanged. [if] a proxy is in front of the edge: check in the edge's log before upgrading that it already resolves client addresses, or everything through that proxy (UI, API, agents, revocations) gets the 503 (§5).
Internal redirects The clients that send the internal service token (gateway-svc, and alerting-svc's and ingestion-svc's reapers and monitors) no longer follow redirects, so a redirecting upstream cannot hand the token to another host; the 3xx is returned or reported as an error instead. No service redirects, so nothing changes in a normal install. [if] loki_url, victoriametrics_url or the NATS monitor URL points at a proxy that redirects (for example http to https): point it at the final URL, or the logs tab, the metrics views or the connections view show an error.
Bootstrap retry An agent can exchange its registration token again for 5 minutes after the first exchange. The retry gets fresh NATS credentials and the earlier ones for that token are revoked and pushed at once (the revocation reconcile repeats a failed push). A token mints at most 5 credentials. Whoever sees a used token within those 5 minutes can now also exchange it, and the agent loses its credentials, so treat the token as secret until the agent is up. Before, the hub burned the token before it answered, so an answer the agent never received (dropped connection, proxy cut-off, the agent's 30s deadline) lost the token and every restart was refused. A retry after an offboard of the location is refused. auth-svc migration 0012 adds a token_hash field to the agent credential ledger; it holds the token's SHA-256, which the token table already holds, and no secret. Nothing to do.
Failed bootstrap An exchange of a registration token that the hub could not record (writing the agent credential ledger failed, and the agent got a 500) no longer uses the token up: the hub now burns the token and records the credential in one transaction, after minting it. Before, the token was burned first, so the agent's restart was refused as an already-used token and the token was lost. The retry is a first exchange: it works until the token expires or the location is offboarded or deleted (which expires its unused tokens), and the 5-minute retry window starts when the hub records an exchange. Nothing to do.
Location delete revokes first Deleting a location now revokes its agent's NATS credentials, and expires its unused registration tokens, before inventory-svc deletes it. Before, the location was deleted first and a failed revoke was only logged, so the agent's credential stayed usable with nothing to retry it. A revoke fails only when auth-svc or its database does, and then nothing is revoked and the delete is refused with a 500 saying the location was not deleted. A failed push to NATS no longer fails a revoke (here or on the Agents page): it is logged, and the revocation reconcile pushes it. If the revoke succeeds and the delete then fails, the 500 says the agent is already revoked; retrying the delete completes it. [if] auth-svc is down: a location delete fails until it is back; retry it then.
Refused location delete A location delete is now checked with inventory-svc before the agent is revoked (a new internal GET /internal/locations/{id}/delete-check, which the gateway does not forward). A delete inventory-svc refuses (a 4xx) is answered with inventory-svc's status and message, and the agent keeps its NATS credentials; before, the agent was revoked first and the refusal reported as a 500 saying the agent was revoked. What you can see: when inventory-svc is down, a location delete now revokes nothing and is refused with a 500 saying so. While gateway-svc already runs this release and inventory-svc still runs rc.2, which has no check, deletes behave as on rc.2. None.
Error bodies A request body the services cannot read (malformed JSON, a missing JSON content type, a field of the wrong type) is now answered in the usual JSON error shape, {"error": true, "message": …, "code": "INVALID_BODY"}, with the same status as before (400, 413, 415 or 422); before, it was plain text. The message names the field, and the UI now shows it instead of "Request failed (422)". A query parameter or path segment the services cannot read (a mistyped limit, a path that is not UTF-8 once decoded) is answered the same way with code INVALID_REQUEST and its usual 400. [if] a script reads these error bodies as plain text: read the message field.
Gateway timeouts gateway-svc's calls to the services, VictoriaMetrics, Loki and the NATS monitor had no time limit, so an upstream that accepted a connection and never answered held the browser's request open indefinitely. They now give up after 60s (10s to connect), or after an hour for a backup download or restore (the edge's limit; one cut off is answered 504, and a restore cut off may be partly applied), set by a new optional [http] section in gateway-svc.toml (timeout_secs, connect_timeout_secs, backup_timeout_secs); 0 is refused at startup. [if] you raised control-svc's command_timeout_secs above 60, or a backup or log query takes longer than these limits: raise the matching [http] value in gateway-svc.toml.
Loki query timeout The bundled Loki set no query timeout, so a log query running past 30 s had its connection cut by Loki's default write timeout and reached the gateway as a dropped reply. Both shipped Loki configs (deploy/compose/loki/loki-config.yml, deploy/loki/loki-config.yml) now set limits_config.query_timeout: 30s and server.http_server_write_timeout: 40s, so Loki answers 504 "Request timed out" itself, and the UI says the query timed out. The chart deploys no Loki. [compose] Loki reads its config only at start: restart it after the pull (§2 step 8). [if] you run your own Loki: give it a query_timeout below gateway-svc's [http] timeout_secs (60 s) and a http_server_write_timeout above it.
Public status page errors When gateway-svc cannot read the checks from inventory-svc, GET /api/public/status now answers 503 Status is temporarily unavailable (code UNAVAILABLE), cached for 10 s, and the status page shows an error. Before, a refused read became an empty status page, cached for 30 s. alerting-svc's monitors likewise skip a cycle when ingestion-svc answers with an error status, instead of reading the error body as "nothing there", which could resolve every firing check alert. [if] something polls /api/public/status: expect a 503 while inventory-svc is unavailable.
NATS storage The NATS account's JetStream storage was invisible: when it filled, every agent publish was refused and nothing in Fleeger noticed. ingestion-svc now writes fleeger_nats_* gauges for the account (used / limit) and the metrics and logs streams to VictoriaMetrics every 30s; a new built-in alert, NATS storage almost full (nats_storage_high, warning, on by default), fires at 80% of the account limit for 5 minutes; and the Services page shows a NATS storage panel where an admin (iam.manage) can drain a stream down to messages newer than a chosen age. A drain that would delete messages not yet forwarded is refused unless explicitly confirmed, and every drain is recorded in the Retention page's sweep log. [if] your NATS account has no JetStream storage limit of its own, the alert stays silent (there is nothing to measure against); set one, as deploy/nats/gen-auth.sh does. Attach channels to the new alert to be notified.
NATS stream caps At a stream's cap (stream_max_bytes, stream_max_msgs, stream_max_age_secs) the broker deletes the oldest messages while ingestion is not forwarding them, and the agents' publishes still succeed, so the data was lost without a trace. ingestion-svc now counts the messages deleted before they were forwarded as fleeger_nats_stream_discarded_unforwarded_messages_total{stream} and logs a warning while it happens, and two new built-in alerts, on by default, fire per stream: NATS stream near its cap (nats_stream_near_cap, warning, 90% of the nearest cap for 5 minutes) and NATS stream discarding unforwarded data (nats_stream_discarding, critical, any such loss in 15 minutes). [if] your NATS creds came from deploy/nats/gen-auth.sh before this release: run ./gen-auth.sh fix-perms ingestion and restart ingestion-svc, which grants it the consumer advisories that report messages deleted while delivered but unacked (VictoriaMetrics or Loki down); until then those are not counted and ingestion-svc logs that the subscription was refused. Attach channels to the new alerts to be notified.
NATS sampler dead-man The three NATS alerts read gauges that only ingestion-svc writes, so when it (or its sampler) stopped, the series went stale and a firing NATS storage almost full resolved as if the account had been drained, while an account or stream filling during the outage went unseen. A new built-in alert, NATS storage sampler silent (nats_sampler_stale, warning, on by default), fires about 10–11 minutes after ingestion-svc's last NATS storage sample, and resolves once a new one arrives. nats_storage_high now holds its last reading while no sample arrives, for up to 7 days, so it stays firing instead of resolving; while samples arrive it reads as before, and a drain writes a sample at once. To clear a held alert, restore ingestion-svc or disable the rule. [if] you upgrade while ingestion-svc is down or cannot reach NATS: the new alert fires about 10–11 minutes after the upgrade. Attach channels to the new alert to be notified.
No logs stream without Loki A hub with ingestion-svc's [loki] unset kept the logs JetStream stream and its ingestion-logs consumer from when Loki was configured, so nothing read it: it filled to its cap and NATS stream near its cap and NATS stream discarding unforwarded data fired for logs for good. ingestion-svc now deletes the logs stream (and its consumer) while [loki] is unset, checked every 30s, and logs how many messages it dropped; the metrics stream is never touched. It leaves the stream alone while its consumer is active (another ingestion-svc with Loki, e.g. the old pod during the rollout; removed on a later pass), and when it is not in the shape ingestion-svc creates (other subjects, or parked by service stream migrate; logged once). Those two alerts resolve for logs. Agents' pod-log publishes are answered "no stream" and dropped; each agent logs that once. Setting [loki] again and restarting ingestion-svc recreates the stream. [if] ingestion-svc runs without [loki] and a logs stream still holds data (a Loki-off hub with the legacy logs stream): the first start of rc.3's ingestion-svc deletes it and its contents within seconds, once nothing consumes it. Those messages are not counted in fleeger_nats_stream_discarded_unforwarded_messages_total; the warning line naming the count is their only record. To keep them, before upgrading either point [loki] at a Loki for the upgrade, let it forward the backlog (in GET /api/nats/jetstream the logs stream's consumer shows num_pending and num_ack_pending 0) and unset it afterwards, or back the stream up with natscli as a SPOKES user that may subscribe to its own inbox and publish $JS.SNAPSHOT.ACK.>: nats --creds <that-user>.creds stream backup logs ./logs-backup (restore later with nats stream restore ./logs-backup on a hub with Loki). The shipped ingestion.creds is not enough: with it the backup stops after its first window and still exits 0 with a partial copy. [if] ingestion-svc runs without [loki] and agents still ship pod logs (FLEEGER_K8S_POD_LOGS): those logs are dropped; unset FLEEGER_K8S_POD_LOGS on the agents to stop collecting them. Agents older than this release log a warning per pod per cycle instead of once.
Backup time budget inventory-svc's calls to SurrealDB and VictoriaMetrics during a UI backup download or restore had no time limit, so a store that stopped answering held the operation until something in front of it gave up. They now share a budget of 3000s (10s per connection), set by a new optional [backup] section in inventory-svc.toml (timeout_secs, connect_timeout_secs); 0 or more than 86400 is refused at startup. A restore cut off by it reports which databases it restored and which one may be partly written. These calls also ignore HTTP_PROXY, HTTPS_PROXY and ALL_PROXY now, since they carry SurrealDB's root credentials. [if] a backup download with metrics, or a restore, takes longer than 50 minutes on your data: raise [backup] timeout_secs, and the edge's proxy_read_timeout on those routes (an hour) with it. [if] FLEEGER_VM_URL points at a VictoriaMetrics that inventory-svc can reach only through a proxy: a backup with metrics, or a restore of one, now fails to reach it; make it directly reachable.
Proxy variables Calls inside the deployment no longer go through a proxy set by HTTP_PROXY, HTTPS_PROXY or ALL_PROXY (in either case): gateway-svc's calls to the services, VictoriaMetrics, Loki and the NATS monitor, alerting-svc's calls to VictoriaMetrics, ingestion-svc's to VictoriaMetrics and Loki, and every call carrying the internal service token. Before, such a proxy also saw the internal token, users' session cookies and the password re-entered for a backup or restore. No shipped deployment sets one. alerting-svc's notifications (ntfy, Matrix, Discord, webhooks) still use it. [if] a service, VictoriaMetrics, Loki or the NATS monitor is reachable only through such a proxy: make it directly reachable.
SPOKES account push The chart's nats-push hook, the raw manifests' fleeger-nats-push Job and deploy/nats/push-accounts.sh no longer push the base SPOKES account over the one auth-svc manages. Before, every chart install or upgrade (and every re-run of the Job or script) replaced the account and re-admitted every offboarded agent until auth-svc's next reconcile pass, and for as long as auth-svc was down. auth-svc now tags each account JWT it pushes with the SHA-256 of the base it was built on, and all three run deploy/nats/push-spokes.sh, which looks the account up first and pushes nothing when it carries the tag of the same base. A changed base is still pushed. The upgrade to this release still pushes the base once, because the account auth-svc pushed before carries no tag yet: offboarded agents are accepted until auth-svc's first pass after it, as on every earlier upgrade. From the next offboard or reconcile push on, the hook leaves the account alone. [if] you keep a copy of push-accounts.sh outside the checkout: take push-spokes.sh next to it.
Docker agents restart The docker run commands the Onboard panel copies and the onboarding guides give now pass --restart unless-stopped, and keep the registration token off the command line (an --env-file written by printf; Kubernetes: a Secret and --set existingSecret). An agent started with the old command has no restart policy: it stays stopped after a failed first boot, a crash or a host reboot. Re-onboarding a Docker agent now means removing its container and the fleeger-agent-data volume first, since an agent ignores a new token while its old credentials are on the volume (Re-onboarding a location). [if] you run agents with Docker: docker update --restart unless-stopped fleeger-agent on each host.
Data left behind Existing git/secret credentials stay, listed but unused. Historical Terraform rows in the control audit log stay. None. Delete orphaned credentials from the UI if you like.
Local image builds Dockerfile.service names its cargo target cache by a new CACHE_ID build arg; deploy/build-and-push.sh and the compose stack pass one per checkout. Before, every checkout shared one cache, and an image built from one worktree could ship another worktree's code or miss its migrations, without an error. The docker builder prune --filter type=exec.cachemount step from the rc1 notes is no longer needed when building with deploy/build-and-push.sh or with compose run from inside the checkout. GitLab CI is not affected. [if] you build images locally: the first build per checkout starts with an empty cache, and the old shared cache is left unused; docker builder prune --filter type=exec.cachemount reclaims it once. [if] you build with plain docker build -f Dockerfile.service from several checkouts: pass --build-arg CACHE_ID=<one value per checkout>.

Do this first

  1. [required] Back up, as for rc1: all five SurrealDB databases (auth inventory ingestion alerting control, namespace fleeger) to the S3 backup set, e.g. a one-off run of the backup job: kubectl -n fleeger create job pre-rc3 --from=cronjob/fleeger-backup. This backup is for going back to rc.2; do not restore it onto rc.3 (see Known issues).

  2. [required] Check that inventory's newest applied migration is 0031_sweep_log_alerting_component or later, and control's 0001_control or later:

    SELECT VALUE migration FROM schema_migration ORDER BY applied_at DESC LIMIT 1;
    

    Anything else (in particular 0045_… / 0002_iac_audit_phase) is a database from before the IaC removal that was never migrated. Stop: these notes do not cover it.

1. Terraform / raw manifests (deploy/k8s/)

  1. [required] Do this first and, if needed, §3.
  2. [required] Copy the images into the registry your cluster pulls from, if that is not the GitLab registry, e.g. with docker buildx imagetools create --tag <yours>/<image>:1.0.0-rc.3 registry.thatsnasu.dev/netcode/fleeger/<image>:1.0.0-rc.3. The copy's top-level digest differs (it wraps the image in a new index); the image inside is byte-identical.
  3. [required] Set the image tag to 1.0.0-rc.3 for every Fleeger image, including frontend, backup and the core agent (spoke-agent).
  4. [if] your control-svc.toml has an [iac] section, remove it. It is ignored either way.
  5. [if] an Ingress or another proxy is in front of the edge: make sure the fleeger-edge ConfigMap you apply sets set_real_ip_from at server level to the proxy's addresses, as Fleeger on Kubernetes — plain manifests describes. Without it, the UI, the API and every agent connecting through wss://<host>/nats get a 503 once the new edge runs. So does auth-svc when its [nats] public_url is that wss:// URL, and then revoking or offboarding an agent fails, so the agent stays admitted. If you keep your own default.conf (edited in place or patched from an overlay), merge in rc.3's map … $fleeger_refuse block and the if ($fleeger_refuse) check; without them nothing is checked. If you set it with kubectl edit, re-applying the shipped 10-config.yaml wipes it: carry it over into the file. The edge reads its config only at start (it is mounted with subPath), so finish with kubectl -n fleeger rollout restart deploy/edge and check that a page through the Ingress loads; otherwise the check starts at the next unplanned restart. If it answers 503, set the CIDR (or re-apply the previous fleeger-edge ConfigMap) and restart the edge again; rollout undo does not bring back a ConfigMap.
  6. [required] Roll out, and check each init container's log: from rc.2, inventory applies 0032–0037, alerting 0012–0015, auth 0011–0012 and ingestion 0002–0003, and control logs Database schema is up to date (see Verification step 1).

2. Docker Compose (deploy/compose/)

  1. [required] Do this first.

  2. [required] In .env, set REGISTRY=registry.thatsnasu.dev/netcode/fleeger and TAG=1.0.0-rc.3 (or build from the rc.3 tag), then docker compose pull.

  3. [if] you mounted an IaC deploy key (IAC_GIT_SSH_KEY) or kept [iac] in config/control-svc.toml, remove them.

  4. [recommended] Rotate the SurrealDB root password: until now any local user on the host could read it with ps, and a new SURREAL_PASS in .env alone does not change it on an existing datastore. Generate a new one and set it while the old server still runs. Paste the current password at the prompt; both passwords stay off every command line (-e SURREAL_PASS with no value hands the container the variable from your shell, and the new one travels on stdin):

    printf 'Current root password: '; stty -echo; IFS= read -r OLD_PASS; stty echo; echo
    NEW_PASS=$(head -c 24 /dev/urandom | base64)
    SURREAL_PASS="$OLD_PASS" docker compose exec -T -e SURREAL_PASS surrealdb \
      /surreal sql --endpoint http://localhost:8000 --user root --auth-level root \
      <<< "DEFINE USER OVERWRITE root ON ROOT PASSWORD '$NEW_PASS' ROLES OWNER;" \
      && echo "$NEW_PASS"

    It prints the new password only if the change went through; a wrong current password prints an authentication error instead. Then put the new one in .env as SURREAL_PASS; step 6 restarts every service with it.

  5. [if] a reverse proxy is in front of the edge: uncomment and fill in the set_real_ip_from block at server level in rc.3's edge.nginx.conf, as Deploying Fleeger describes, before the next step. The rc.3 edge answers 503 to every request through an untrusted proxy, including agents on wss://<host>/nats and auth-svc when its [nats] public_url is that URL (then revoking or offboarding an agent fails, so the agent stays admitted). If you edited edge.nginx.conf in your checkout (rc1 step 7), keep your trust lines and take rc.3's map … $fleeger_refuse block and if ($fleeger_refuse) check with them; without those nothing is checked:

    git stash && git pull && git stash pop   # or check out the rc.3 tag instead of pull
    git diff deploy/compose/edge.nginx.conf  # your trust lines, plus rc.3's map and if

    Also publish the edge on loopback only when the proxy runs on the host (deployment shows the override): otherwise clients reaching port 8080 directly can arrive from the address you trust.

  6. [required] docker compose --profile core-agent up -d (drop the profile if you do not run the bundled core agent).

  7. [required] docker compose restart inventory-svc. The previous step recreates the surrealdb container (its command and environment changed), and inventory-svc can stay unusable for minutes after a database restart (see Known issues).

  8. [required] docker compose restart edge loki. Unless the pull in step 2 brought a newer nginx:1.27-alpine, up -d does not recreate the edge (its service definition did not change), and nginx reads edge.nginx.conf only at start. Loki likewise reads rc.3's loki/loki-config.yml (its query timeout) only at start. [if] a reverse proxy is in front: check that a page loads through it. If it answers 503 with "Fleeger is behind a proxy the edge does not trust", fix the address from step 5 (the 503s in docker compose logs --no-log-prefix edge show the one the edge sees) and restart the edge again.

3. SurrealDB server v3.2.4

[required] if the server is still on 3.1.x.

Tested on a copy of five production databases: v3.2.4 opens v3.1.5's RocksDB data in place (check_version passes; row counts, schema and queries are identical), and v3.1.5 can reopen the data afterwards, so rolling back is just the image tag. rc.2's backup image (3.1.5 CLI) exports from and restores into 3.2.4 without errors.

  1. Take a fresh backup.
  2. Change the server image to surrealdb/surrealdb:v3.2.4 and roll it out. The Deployment uses Recreate, so the database is briefly down.
  3. [required] kubectl -n fleeger rollout restart deploy/inventory-svc (compose: docker compose restart inventory-svc). See the database reconnect issue under Known issues: without it inventory-svc can stay unusable for minutes after any database restart, and the core agent's collection with it.

4. NATS hub permissions (if rc1's fix-perms was skipped)

rc1 narrowed four hub users (gateway, alerting, ingestion, core-agent), but the narrowing only takes effect once their .creds are reissued. To check, decode each .creds user JWT and compare it with user_perms in deploy/nats/gen-auth.sh.

  • With the nsc store at hand: ./gen-auth.sh fix-perms gateway alerting ingestion core-agent.
  • Without it (e.g. the Secret, or the Terraform variables that render it, are the only copy): rebuild one with gen-auth.sh import first. The steps are in Agent credentials & revocation.

Put the reissued .creds back where the deployment reads them, then restart only those four services. Agents do not re-onboard, and push-accounts.sh is not needed (and must never be run from an imported store).

5. Helm chart (charts/fleeger-hub)

Not exercised for this upgrade. The iac.* values are ignored; the chart no longer mounts an IaC git key or renders [iac]. §3 and §4 apply the same way.

[if] a proxy is in front of the edge: before the upgrade, check that the edge already sees real client addresses. In kubectl -n fleeger logs deploy/edge, the first field of each line is the address the edge resolved; if outside requests show the ingress controller's (or LB's) address there, edge.trustedProxies misses that hop and the rc.3 edge will answer everything through it with a 503: the UI, the API, agents on /nats and, with a wss:// nats.publicUrl, auth-svc's revocations. Set the CIDR covering that address first, as the Helm chart reference's "Edge trusted proxies" section describes. If the address is already covered, set edge.realIpHeader to the header the proxy sends. After the upgrade, check that a page loads through the proxy and that agents reconnect. The edge restarts by itself on the upgrade.

[if] a service mesh injects a sidecar into the edge pod: the edge then sees the sidecar's loopback address and answers proxied requests with the 503 above. Keep the edge out of the mesh with the new edge.podLabels (Istio) or edge.podAnnotations (Linkerd) values, as the Helm chart reference describes. Neither may set a key the chart sets itself.

Rollback

  • To rc.2: not the same schema any more. rc.3's migrations (inventory 0032–0037, alerting 0012–0015, auth 0011–0012, ingestion 0002–0003) stay applied: rc.2's init containers find nothing newer than their own newest migration and log Database schema is up to date, and the new fields are optional, so rc.2's services start. Switching the image tags back should therefore work, with the ingestion-svc step below. Not tested; for a clean rc.2, restore the pre-upgrade backup into empty databases under rc.2's images.
  • To rc1: not possible in place. rc1's inventory-svc writes iac_managed, which an rc.3 schema no longer has. Restore a backup taken under rc1 and deploy rc1's images.
  • ingestion-svc to rc.2: rc.2 ignores the cluster an rc.3 agent sends, writes unlabeled resource_state rows again, and does not read the labeled rows' cluster, so each labeled row goes stale and fires resource_unreachable until the stale ceiling retires it (30 minutes by default). Delete them right after the rollback, in the ingestion database: DELETE resource_state WHERE cluster != NONE;. The unlabeled rows rc.3 marked superseded are listed again by rc.2 and kept fresh by its writes.
  • SurrealDB server: set the image back to v3.1.5 (verified to reopen the data); ingestion-svc, inventory-svc and auth-svc recognise and retry its commit conflicts too, and inventory-svc names its failed COMMIT's reason rather than a generic "not executed".
  • Compose file: an rc.2 or rc1 compose.yaml puts the SurrealDB password back on the server's command line.
  • Compose edge: rolling back only the images keeps rc.3's edge.nginx.conf and its 503 for an untrusted proxy. Checking out an rc.2 or rc1 tree drops it again: an untrusted proxy is then served silently, with one shared login bucket.
  • Helm edge: the 503 for an untrusted proxy comes from the chart, not an image, so rolling back the images keeps it; fix edge.trustedProxies or edge.realIpHeader instead. helm rollback to an older chart drops it, and an untrusted proxy is then served silently again.

Verification

  1. Every init container logs Database schema is up to date or applies only migrations newer than the ones it had, and the newest applied migration (SELECT VALUE migration FROM schema_migration ORDER BY applied_at DESC LIMIT 1;) is 0037_rollback_lock in inventory, 0015_mute_cluster in alerting, 0012_nats_user_token in auth, 0003_resource_state_inventory_id in ingestion and 0001_control in control.
  2. The core agent logs core cycle complete: N location(s) with your number of local-collector locations, every ~16 s.
  3. Ingestion updates resource states: in the ingestion database, SELECT count() FROM resource_state WHERE last_seen > time::now() - 1m GROUP ALL; is non-zero.
  4. A control ping on any resource, and the location page's Test control path, return pong.
  5. No service and no NATS log line mentions a permission violation.
  6. [required] Take a fresh backup of all five databases (as in Do this first step 1), and from now on restore only backups taken under rc.3. Restoring a pre-rc.3 backup onto rc.3 undoes part of its migrations (see Known issues); keep the pre-upgrade backup only for going back to rc.2.

Known issues

  • Database reconnect: a database call on a connection that died without being closed (which is what a Recreate rollout of SurrealDB, an eviction or a node loss does) hangs for minutes, and while it does every database call of that service hangs. Restart inventory-svc after any SurrealDB restart. Seen on production on 2026-10-08: about 2 minutes without collection, and two location_unreachable alerts that resolved on their own. Left alone it recovered by itself after 3 to 16 minutes in tests. Since rc.3 (see the summary) a stuck inventory request no longer holds up other locations and is given up on at its deadline, but a discovery or hardware request caught by this holds its own location for up to 10 minutes, so the restart is still the rule.
  • Alerts can outlive a deleted resource. A delete, or a full checkpoint rollback that removes a resource, drops the resource's alert state (see the summary), but not when ingestion-svc cannot be reached at that moment, nor for rows an agent older than rc.3 wrote: about 90 s later resource_unreachable fires for it and each resource discovered under it, and the alerts stay until the stale ceiling (resource_stale_ceiling_secs, 1800 s by default) retires them. Expect them, and ignore them.
  • Restoring a pre-rc.3 backup onto rc.3 undoes part of the upgrade.** A SurrealDB backup carries the schema definitions and migration stamps of the release it was taken under. Both restore paths (the UI restore and scripts/restore.sh) import it into the live databases, whose rc.3 migrations stay marked applied, so the backup's older definitions replace rc.3's and the migrations that set them are never run again. Nothing reports it. For rc.3 that would bring back, for example, ingestion's resource_state_key unique index without cluster (ingestion 0002), inventory's resource_discovery index without discovery_scope (0032) and its address check that lets a blank address through (0034), and auth's visible-locations function from before 0011 (the gateway no longer relies on it). The restored rows also lack what rc.3's data migrations filled in, e.g. the meta location's creation time (0036). Take a fresh backup right after upgrading (Verification step 6), and do not restore a pre-rc.3 backup onto rc.3. To go back to such a backup, restore it under rc.2 (see Rollback).
  • Open and resolved issues. The compose promtail still mounts the Docker socket: the risk is accepted, and the mount is unchanged. Still open from rc1: trustedProxies values are not validated; NOTES.txt gives a bare --set; restoring a pre-rc.3 backup (above); old published frontend images; the NATS readiness probe; no config checksum on the raw NATS Deployment; a Loki-off hub's legacy logs stream (rc.3 deletes it, see "No logs stream without Loki"); no FLEEGER_CREDENTIAL_KEYS_RETIRED wiring in the raw manifests; no :? guard on the key in compose; and stream tunables unreachable through Helm. Resolved since rc1: base SPOKES pushes (see "SPOKES account push"), the NATS 3Gi upgrade recipe (now in the Helm chart docs), watching the NATS storage budget and stream caps, alerting-svc's rekey, TLS panics, docs prescribing a destructive gen-auth.sh, stale migration embedding, one slow inventory request stalling every location, the SurrealDB password on the server command line (compose only; Helm and the raw manifests never had it), and a proxy without a trusted CIDR on the raw manifests, on compose and in the Helm chart (at runtime: a wrong CIDR, an ingress outside the chart or a waived render-time check). The IaC findings went away with the feature.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.