// Upgrades
Upgrading a running Fleeger hub to 1.0.0-rc.4
From: 1.0.0-rc.3. To: 1.0.0-rc.4. For an older hub, upgrade to rc.3
first with the rc.3 notes.
Each step is tagged [required], [recommended] or [if …]. ## Summary
| Area | Change | Manual work |
|---|---|---|
| Upgrade order: gateway-svc first | Three request headers that only gateway-svc may set are new in rc.4: x-fleeger-credential-any-host (the caller holds resource.edit on all locations) and x-fleeger-local-collector-grant (the caller holds location.edit on all locations; also used for in-cluster Kubernetes registrations), as rc.3 added x-fleeger-rollback-locations. rc.4's gateway-svc drops a client's copy of each; rc.3's forwards them unchanged. While an rc.3 gateway-svc is in front of an rc.4 inventory-svc, any signed-in client can send them itself, and then attach any credential of a location at a host of its choosing (the agent sends the secret there), set local_collector, or register an in-cluster Kubernetes cluster at a core-agent location. In the same window the gateway sends neither header for real global editors, so they are treated as scoped. The acting user's name is now sent percent-encoded: with an rc.3 gateway-svc, rc.4's control-svc, inventory-svc and ingestion-svc answer 400 to every write, control action and stream drain of a user whose name is not ASCII; with an rc.3 service behind an rc.4 gateway-svc, such a name is recorded encoded (Jos%C3%A9). |
[required] Upgrade gateway-svc first and let its rollout finish, then inventory-svc, control-svc and ingestion-svc straight after; never roll back gateway-svc alone to rc.3 while inventory-svc runs rc.4 (§1). |
| Loki: ingestion limits | ingestion-svc now retries a log batch Loki answers with 429 or 408, as it already did for 5xx, instead of acking and dropping it, and pauses between pushes while Loki refuses them (0.5 s, doubling up to 15 s, reset by the first success). Loki's single-tenant defaults (4 MB/s, 6 MB burst, 4 MiB gRPC messages) are below one log message's 8 MiB stream limit, so a large batch would now be retried until the logs stream ages it out. Both shipped Loki configs (deploy/compose/loki/loki-config.yml, deploy/loki/loki-config.yml) set limits_config.ingestion_rate_mb: 8, ingestion_burst_size_mb: 16 and server.grpc_server_max_recv_msg_size/grpc_server_max_send_msg_size: 16777216. The chart deploys no Loki. |
[required, compose] Loki reads its config only at start, and up -d does not recreate it: docker compose restart loki (§2 step 6). [if] you run your own Loki (Helm loki.url, raw manifests): set the same three limits (§6). |
| Alerting: channel URLs in logs | A Discord, ntfy or Matrix delivery or channel test that failed in transport (DNS, connect, timeout, TLS) wrote the full request URL into alerting-svc's log, at warn from a rule's delivery and at error from a channel test. For Discord that URL is the channel's whole credential (the webhook token); for ntfy it holds the topic, which is the credential of an unprotected topic; for Matrix it holds the room id, and any user:password in the homeserver URL. ntfy and Matrix access tokens travel in a header and were not logged. alerting-svc's log is readable by anyone holding resource.view through the log viewer. rc.4 logs the error without the URL. Email channels were never affected. Lines written before the upgrade stay in Loki and in the container logs. |
[recommended] if such a delivery or test ever failed: rotate the Discord webhook, the ntfy topic, or Matrix credentials in the homeserver URL, and drop the old lines where your retention keeps them (§7). |
| Alerting: channel destinations | A notification channel may no longer deliver to loopback (127.0.0.0/8, ::1, and 0.0.0.0/::) or to the hub services named in alerting-svc's config ([victoriametrics], [ingestion], [inventory], [nats], [database]), by name or by any address the name resolves to, on any port. Creating a channel or editing its config so is refused with 400 naming the reason. The check runs again on every delivery, so a stored channel of that kind fails its test with the same 400 and, attached to a rule, is reported as unusable. HTTP channels connect only to the address that was checked, SMTP connects to the checked address and verifies TLS against the host name, and notification requests no longer follow redirects (a 3xx is reported as the error). LAN and public hosts are unaffected. An HTTP proxy from the environment is still used and is not checked. |
[if] a channel points at the hub host's loopback (e.g. a self-hosted ntfy on localhost): point it at the host's LAN address or name. [if] a webhook redirects: point it at the final URL. |
Backup: restore.sh keeps the VictoriaMetrics snapshot |
For a docker-exec backup set, scripts/restore.sh now saves the VictoriaMetrics snapshot tarball to RESTORE_DIR (default: the current directory) before it imports anything, and prints that path. Before, it printed a path in a temp directory that it had already deleted on exit, so the documented manual VictoriaMetrics restore could not be followed. If the directory is missing or not writable, it stops with nothing imported and says to set RESTORE_DIR. native sets are unchanged. |
[if] you restore a docker-exec set: run restore.sh from, or set RESTORE_DIR to, an existing writable directory with room for the snapshot (it can be several GB). Inside the backup image, / is not writable. See VictoriaMetrics restore. |
| Credentials: attach by id only where already used | Attaching an existing credential by id (POST /api/locations/{id}/resources with credential_id, PUT /api/resources/{id}/credential) is now allowed only at an endpoint where a resource using that credential already points. An endpoint is the URL's scheme, host and port with the scheme's default port filled in, so https://pve is https://pve:443 and http://pve:8006 is not https://pve:8006; bare host:port addresses compare as written. Anywhere else, and for a credential no resource uses (saved in the credential manager, or left over after a detach or delete), the attach is refused with 403 unless the caller holds resource.edit on all locations; everyone else enters the secret on the resource, which saves a new credential. Detaching, re-pointing and re-attaching no longer gets round it. Editing the address of a resource that has a credential from https to http, or a rollback doing so, is now refused like any other change of host or port. The resource form and the manager's Attach picker grey out credentials that cannot be attached at the address, and say why. |
[if] scoped users reuse a saved credential at another host, or attach credentials saved in the manager: they now enter the secret instead, or a user with resource.edit on all locations attaches it. |
Helm: image.tag defaults to the chart's appVersion |
In both charts image.tag now defaults to "", which means the chart's appVersion, the tag its release pipeline pushed; the backup CronJob's image follows the same rule. The install docs no longer pin the tag. Helm stores a tag set with --set image.tag=… and carries it into every later upgrade (plain, --reuse-values and --reset-then-reuse-values alike), so a pinned release keeps running its old images under the new chart. Checkout installs with locally built fleeger/*:local images now need --set image.tag=local; the UI's Onboard panel adds it to the Kubernetes command it hands out. The documented agent OCI install set image.registry, which the agent chart never read (it pulled fleeger/spoke-agent from Docker Hub); it now sets image.repository. CI no longer publishes a chart for a tag whose image jobs failed. |
[if] helm get values <release> shows an image.tag: clear it once on this upgrade, for fleeger-hub and fleeger-agent (§4). [if] you install from a checkout: add --set image.tag=local. [if] your agent install set image.registry: set image.repository=<registry>/spoke-agent instead. |
| Ingestion: message size limits | The metrics and logs JetStream streams now refuse a message over 1 MiB and 8 MiB respectively, and ingestion-svc pulls at most a stream's limit at a time, so one agent can no longer get ingestion-svc OOM-killed and crash-looping for every location. At startup ingestion-svc adds the limit to an existing stream that has none; a limit already set (nats stream edit <stream> --max-msg-size …) is kept and read again at runtime. A message stored before the limit and larger than it is acked and dropped with the warning dropping metric message over the stream's size limit, unparsed (or log message). If such a message holds up a consumer, ingestion-svc warns that the '<consumer>' consumer has been handed nothing for …, takes a larger probe batch and gets past it. rc.4 agents split a pod's log batch that would serialize over 4 MiB into several messages, and cut a single longer line, ending it in … [truncated by fleeger-agent]. |
[if] agents ship pod logs (FLEEGER_K8S_POD_LOGS): upgrade them, or a batch over 8 MiB from an rc.3 agent is refused by the stream. |
| Ingestion: resource state per location | Each location keeps at most max_resources_per_location ([ingestion] in ingestion-svc.toml; default 10000; 0 is refused at startup) rows in resource_state. Past it, a resource the location has no row for yet still has its metrics stored, but shows no reachability and raises no resource_unreachable; each such message logs not recording the state of a new resource: … and counts in fleeger_ingestion_resource_state_dropped_total{location}. Rows that exist keep updating, and deletes, offboards and the retention reaper make room. Migration 0004_resource_state_bound (ingestion) indexes resource_state.location and counts each location's rows in a new resource_count table, kept by an event on resource_state. GET /api/metrics/state/resources takes ?location= (repeatable) and is filtered in ingestion-svc; the gateway asks only for the locations the caller may see, and still filters an older ingestion-svc's answer. |
[if] fleeger_ingestion_resource_state_dropped_total grows for a real site (e.g. a Kubernetes location whose pod rows pile up with the reaper off): enable the retention reaper or raise max_resources_per_location. The chart does not expose the key. |
| Inventory: address checked on a kind change | A PUT /api/resources/{id} that changes a resource's kind without sending an address now checks the stored address against the new kind, as a create does, and refuses with 400 if it fails. Before, a tls_check address such as javascript:… could become an http_check address, which the uptime page rendered as a link. A checkpoint rollback that changes a resource's kind or address, or recreates one, is checked the same way and refused with 400 naming the resource, before anything is written. An edit that keeps the kind does not re-check the address. The uptime page now links an address only when it is an http: or https: URL. |
[if] a script changes a resource's kind without sending an address: send the new kind's address too. [if] a rollback is refused for an address: such a pair is already stored. To find them, in the inventory database: SELECT meta::id(id) AS id, meta::id(location_id) AS location, kind, address FROM resource WHERE discovered = false AND kind NOT IN ['tcp_check', 'tls_check', 'dns_check'] AND !string::starts_with(string::lowercase(string::trim(address)), 'http://') AND !string::starts_with(string::lowercase(string::trim(address)), 'https://'); and edit or delete them. |
| Inventory: requests per location | inventory-svc now holds at most 256 waiting requests per location lane (a location's discovery reports have a lane of their own). A request beyond that is answered at once with inventory-svc is busy: too many requests are waiting for this location; try again later and never runs; the agent sends it again on a later cycle. Before, one location could fill the responder's shared bound of 4096 and stall every other location, which an earlier fix was meant to prevent. The bound is the new optional [nats] max_pending_per_lane in inventory-svc.toml, 1 to 4096; anything else is refused at startup. A full lane logs one warning per burst. |
[if] a location legitimately sends more than 256 requests at once (e.g. hardware reports of more than 256 Redfish hosts): raise [nats] max_pending_per_lane. The chart does not expose the key. |
| Kubernetes: kubeconfigs that run commands | A kubeconfig must carry its credentials inline: token, username/password, or client-certificate-data/client-key-data, with certificate-authority-data. Saving one where any user sets exec or auth-provider, or a path in tokenFile, client-certificate or client-key, or any cluster sets certificate-authority, is refused with 400; so is YAML that does not parse. Before, the agent serving the location ran the exec command, and for a local_collector location that was the bundled core agent inside the hub. An rc.4 agent refuses a kubeconfig stored earlier: the cluster is no longer collected, its control actions fail, and its resource shows a collect error starting kubeconfig refused: that names the user or cluster and the field. Typical cases are EKS, GKE and AKS kubeconfigs (aws, gke-gcloud-auth-plugin, kubelogin). inventory-svc now also answers a credential reveal only to the location's own agent, or to the core agent for a local_collector location. |
[if] you register Kubernetes clusters: after the upgrade, look for kubeconfig refused: collect errors and replace each such credential with a ServiceAccount token kubeconfig (§5). Upgrade the agents: an rc.3 agent still runs a stored exec plugin. |
| Kubernetes: the core agent and the hub's cluster | At a location the bundled core agent collects (local_collector), a k8s_cluster's kubeconfig server and proxy-url may no longer be a hub service, loopback, link-local or, when the hub runs on Kubernetes, the hub cluster's own API (kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, KUBERNETES_SERVICE_HOST). Such a cluster is not collected and shows a collect error starting refused:, and its control actions are refused before anything is sent. The core agent connects to the address it checked; a kubeconfig with a proxy-url that passes the check is then refused by the Kubernetes client, which this build has no proxy support in. Registering a k8s_cluster without a credential (collected in-cluster, i.e. the hub's own cluster) at such a location, or making one by changing a resource's kind, detaching a cluster's credential or rolling back, now needs location.edit on all locations (403 otherwise). The core agent no longer collects in-cluster a registration whose credential has no kubeconfig; it shows refused: instead. Remote spokes are unchanged. |
[if] a k8s_cluster at a local_collector location shows a collect error starting refused:: give it a kubeconfig that reaches the cluster's real API address, or a credential with a kubeconfig. Existing in-cluster clusters keep working and stay editable; if one is removed, a user with location.edit on all locations re-registers it. |
Permissions: local_collector |
Setting or changing a location's local_collector (POST /api/locations, PUT /api/locations/{id}, or a checkpoint rollback restoring another value) now needs location.edit on all locations; without it the write is refused with 403. Sending the value the location already has still works and writes nothing, so flagged locations keep their flag. The bundled core agent no longer probes or collects from the hub's own services: an HTTP, TCP or TLS check whose host is a hub service name, resolves to a hub service's address, or is loopback, unspecified or link-local reports probe_success 0 with reason="blocked_target" and a collect error; a Proxmox or iDRAC registration at such an address is not collected and shows a collect error. DNS checks are not affected. The list is FLEEGER_HUB_SERVICES (compose .env) or coreAgent.hubServices (Helm): host names, IPs or CIDR ranges, by default the compose and chart service names (surrealdb, nats, victoriametrics, loki, promtail, the six *-svc, frontend, edge), plus the host of the agent's FLEEGER_NATS_URL; a list you set replaces the defaults. The core agent's checks connect only to the addresses they checked and no longer use HTTP(S)_PROXY. Remote spokes are unchanged. |
[if] scoped roles managed local_collector: grant location.edit globally to whoever should keep doing it. [if] the core agent checked a hub service on purpose: set FLEEGER_HUB_SERVICES / coreAgent.hubServices to a list without it. [if] the core agent's HTTP checks relied on a proxy: they now connect directly. [if, Helm] to refuse hub pods by pod IP too, add your pod and service CIDRs to coreAgent.hubServices, keeping the service names. |
| Permissions: rollback | A checkpoint rollback now needs the rights its changes need when made directly: resource.edit and location.edit at the location for a safe rollback, plus resource.create and resource.delete there for a full one. Only an explicit "mode": "safe" counts as safe: a request without mode needs the full set, although inventory-svc runs it as safe. Before, location.create at the location was enough for either mode. A vantage point the rollback adds at another location needs resource.create there (it needed location.create). Taking a checkpoint and lifting the rollback lock still need location.create. The Config history page offers Roll back, and Full reconcile, only to users holding those rights. |
[if] a custom role rolled back through location.create: add resource.edit and location.edit (and resource.create, resource.delete for full rollbacks) at the locations where it should keep rolling back, and resource.create at the vantage locations its checks use. [if] a script rolls back without mode: send "mode": "safe", or give it the full set. |
| Proxmox: container power actions | Power actions on LXC containers now work; every one failed before, because the agent always posted to the qemu endpoint. The agent first looks the guest up in /cluster/resources, which needs VM.Audit, and sends the action to the node listed there, so a guest that migrated since discovery is acted on where it runs. A guest that is not listed is refused with guest <vmid> is not listed in /cluster/resources; it does not exist, or the API token lacks VM.Audit on it (power actions need VM.Audit as well as VM.PowerMgmt). Reset on a container is refused (use reboot). |
[if] the control token has VM.PowerMgmt without VM.Audit: add VM.Audit, or actions on VMs now fail too. Upgrade the agents: an rc.3 agent still posts container actions to qemu. |
| Rollback: rows without history | A full rollback now removes a registered resource its checkpoint does not hold only if the resource has a revision and was created after the checkpoint; it leaves any other and lists it in the new kept_ids field of the rollback response. Before, it deleted rows that had no revision (rows from before revisions existed, and everything migrate-legacy imported), with their discovered resources and hardware inventory. Migration 0038_baseline_revisions (inventory) gives every location and registered resource without a revision a baseline create revision by system, dated to the row's created_at, so checkpoints taken from now on hold them. migrate-legacy now journals what it imports. |
[if] you script full rollbacks: read kept_ids. |
| Rollback: discovered resources | Discovered resources (Proxmox VMs and CTs, Kubernetes objects) are now fully out of config history and rollback. Editing one (name, tags) or setting or detaching its credential still applies, but writes no revision. A rollback skips entries for discovered resources that earlier releases wrote. Before, a full rollback after such an edit failed with 400 or 500 while the resource existed, or, once it was gone, recreated it as a registered resource with no parent that discovery never updates or removes. | [if] a full rollback left such a copy (a registered resource with discovered = false, no parent, and the discovered object's name as its address; the query in the "Inventory: address checked on a kind change" row lists it): delete it. |
Alerting: check_down counts each vantage once |
check_down now reduces each vantage point to its newest probe sample before counting. Before, the series of the previous reason label (e.g. reason="ok") stayed in the instant query's 5-minute lookback after a check failed, and counted as a passing vantage, so outages shorter than about 5 minutes, flapping checks, and multi-vantage checks were under-counted and never paged. |
None. Expect check_down to fire for short outages and flapping checks that were silent before. |
| Audit: user names that are not ASCII | gateway-svc sends the acting user to the services percent-encoded as UTF-8, and control-svc, inventory-svc and ingestion-svc decode it. A user named José is now audited and journalled as José instead of system, and an admin with such a name can download and restore backups (before: 401). A service now refuses with 400 an actor header it cannot decode, instead of attributing the request to system. ASCII names are unaffected. Rows already written as system stay so. |
None beyond the upgrade order in the first row. |
| Compose: restart policy | surrealdb, nats and victoriametrics in deploy/compose/compose.yaml now have restart: unless-stopped, as every other long-running service already had. Before, they stayed Exited after a Docker or host restart while the application services came back and restarted in a loop, until someone ran docker compose up -d. |
None beyond §2: its up -d recreates the three containers once with the policy. |
| Database: migrations | New since rc.3: inventory 0038_baseline_revisions and ingestion 0004_resource_state_bound. alerting stays at 0015_mute_cluster, auth at 0012_nats_user_token, control at 0001_control. Both run in the init containers (compose: the *-migrate one-shots). |
None. Check them under Verification step 1. |
| Database: reconnect | Every service now checks its SurrealDB connection every 5 s and opens a new one after two checks in a row get no answer within 5 s, logging Database connection presumed dead … and then Database connection replaced (keystone's DbHandle). A connection that died without being closed, as when SurrealDB comes back on a new pod IP after a Recreate rollout, no longer leaves inventory-svc (or another service) failing every request until it is restarted. Recovery takes about 20 s. A call already waiting on the dead connection still ends only at its caller's deadline (the gateway's [http] timeout_secs, or 60 s for an agent's inventory request). |
None. The rc.3 rule "restart inventory-svc after any SurrealDB restart" no longer applies once every service runs rc.4 (see Known issues). |
| Ingestion: envelope bounds | A metric or log message whose envelope fields (location, source, resource_id, cluster, inventory_id) are over 512 bytes, or that carries over 5,000 points or over 1,000 log streams, is now acked and dropped with a dropping metric message outside the envelope bounds: … (or log message) warning instead of being encoded. Before, one such message under 1 MiB could make ingestion-svc allocate gigabytes and crash-loop. |
None; agents never send such messages. The warning points at a misbehaving or compromised agent. |
Do this first
- [required] Back up all five SurrealDB databases (
auth inventory ingestion alerting control, namespacefleeger), e.g. a one-off run of the backup job:kubectl -n fleeger create job pre-rc4 --from=cronjob/fleeger-backup. This backup is for going back to rc.3; do not restore it onto rc.4 (see Known issues). - [required] Check that the hub runs rc.3's schema. The newest applied
migration
(
SELECT VALUE migration FROM schema_migration ORDER BY applied_at DESC LIMIT 1;) must be0037_rollback_lockininventory,0015_mute_clusterinalerting,0012_nats_user_tokeninauth,0003_resource_state_inventory_idiningestionand0001_controlincontrol. Anything older: upgrade to rc.3 first.
1. Upgrade gateway-svc first
[required] on every install path (see the first row of the summary).
- Bring gateway-svc to
1.0.0-rc.4on its own and wait until no rc.3 gateway pod is left: raw manifests and Helm in §3 and §4, compose in §2. A simultaneous rolling upgrade leaves old gateway pods in front of new inventory pods for a while. - Then upgrade the rest at once. Until control-svc, inventory-svc and
ingestion-svc run rc.4, a user whose name is not ASCII is recorded in
encoded form (
Jos%C3%A9); nothing else changes in that window. - Never roll back gateway-svc alone to rc.3 while inventory-svc runs rc.4, and roll inventory-svc back first when you roll back both.
2. Docker Compose (deploy/compose/)
- [required] Do this first.
- [required] In
.env, setTAG=1.0.0-rc.4(or build from the rc.4 tag), check out rc.4'sdeploy/compose/(it carries the newcompose.yamlandloki/loki-config.yml), thendocker compose pull. - [if] the core agent should probe a hub service, or should refuse more
than this stack's service names: set
FLEEGER_HUB_SERVICESin.envas.env.exampledescribes. A list you set replaces the defaults, so keep the service names in it. - [required]
docker compose up -d --no-deps gateway-svc(§1). - [required]
docker compose --profile core-agent up -d(drop the profile if you do not run the bundled core agent). It recreatessurrealdb,natsandvictoriametricsonce, because their restart policy changed, and runs the migrations. Unlike in rc.3, inventory-svc needs no restart afterwards: the rc.4 services reconnect by themselves. - [required]
docker compose restart loki, so Loki reads rc.4'sloki/loki-config.yml.edge.nginx.confdid not change, so the edge needs no restart.
3. Raw manifests (deploy/k8s/)
No manifest changed since rc.3.
- [required] Do this first.
- [required] Copy the images into the registry your cluster pulls from,
if that is not the GitLab registry, as for rc.3 (
docker buildx imagetools create --tag <yours>/<image>:1.0.0-rc.4 registry.thatsnasu.dev/netcode/fleeger/<image>:1.0.0-rc.4). - [required] Set gateway-svc's image to
1.0.0-rc.4, apply, and wait:kubectl -n fleeger rollout status deploy/gateway-svc(§1). - [required] Set the image tag to
1.0.0-rc.4for every other Fleeger image, includingfrontend,backupand the core agent (spoke-agent), and roll out. inventory's init container applies0038_baseline_revisionsand ingestion's0004_resource_state_bound; the others logDatabase schema is up to date. - [if] you run your own Loki: §6.
4. Helm chart (charts/fleeger-hub)
[required] Do this first.
[if] the release has a pinned tag.
helm get values fleeger -n fleegershows the values Helm stored; animage.tagthere keeps the old images running under the new chart. Clear it in step 4 with--set image.tag=. If you pass a values file with-f, settag: ""underimage:in it; deleting the key is not enough under--reset-then-reuse-values. If you install from a checkout with locally built images, set--set image.tag=localinstead.[required] gateway-svc first (§1). A Helm upgrade rolls every Deployment at once, so set gateway-svc's image beforehand and wait:
kubectl -n fleeger set image deploy/gateway-svc \ gateway-svc=<image.registry>/gateway-svc:1.0.0-rc.4 kubectl -n fleeger rollout status deploy/gateway-svcThe Helm upgrade in step 4 renders the same image, so it does not roll gateway-svc again.
[required] Upgrade the release:
helm upgrade fleeger oci://<registry>/charts/fleeger-hub --version 1.0.0-rc.4 \ -n fleeger --reset-then-reuse-values --set image.tag=Drop
--set image.tag=if no tag was stored. The init containers apply inventory0038and ingestion0004.[if] you run
fleeger-agentreleases: upgrade them the same way (--set image.tag=if a tag is stored), and if one setimage.registry, replace it with--set image.repository=<registry>/spoke-agent.[if]
coreAgent.enabled: the core agent now refuses this chart's service names by name and by ClusterIP. To refuse hub pods by pod IP too, setcoreAgent.hubServicesto the service names plus your pod and service CIDRs, if nothing the core agent monitors lives there.[if]
loki.urlis set: §6.
5. Kubernetes credentials that run commands
[if] you register Kubernetes clusters.
- After the upgrade, open each location with
k8s_clusterresources and look for a collect error startingkubeconfig refused:. It names the user or cluster and the field (exec,auth-provider,tokenFile,client-certificate,client-keyorcertificate-authority). - For each one, create a ServiceAccount in that cluster with the read
ClusterRole (plus write on the objects you act on, if you use control
actions), and build a kubeconfig that carries a token for it and the CA
inline (
certificate-authority-data):kubectl create token <sa>for a token with an expiry, or akubernetes.io/service-account-tokenSecret for a long-lived one. A file path in the kubeconfig is refused too: inline the certificate and key asclient-certificate-dataandclient-key-data. - Replace the cluster's credential with it (rotate the credential, or attach a new one). The cluster is collected again on the agent's next cycle. See Managing a Kubernetes cluster.
- Upgrade the remote agents. The hub refuses to store such a kubeconfig from now on, but an rc.3 agent still runs one stored earlier.
6. A Loki you run yourself
[if] loki.url (Helm) or loki_url and [loki] (raw manifests) point at a
Loki you run. Set at least:
server:
grpc_server_max_recv_msg_size: 16777216
grpc_server_max_send_msg_size: 16777216
limits_config:
ingestion_rate_mb: 8
ingestion_burst_size_mb: 16
and restart Loki. Without them, Loki answers a push over its burst with 429
and one over 4 MiB with 500 every time, and ingestion-svc now retries such a
batch until the logs stream ages it out (3 days by default), pausing up to
15 s between pushes, instead of dropping it as rc.3 did. Keep the query
timeout settings from the rc.3 notes.
7. Notification URLs that may have been logged
[recommended] if a Discord, ntfy or Matrix delivery or channel test ever
failed with a transport error. rc.3 logged the full request URL with
it, and alerting-svc's log is readable by every user with resource.view.
- Check whether it happened: in the Observability Explorer's log tab (or
Loki directly), query
{service="alerting-svc"} |= "for url ("over your whole Loki retention. Compose also keeps the lines indocker compose logs alerting-svc; Kubernetes in the pod logs. - For each channel that shows up, rotate the secret part of the URL: create a new Discord webhook and delete the old one; move to a new ntfy topic (unless the topic is protected by a token, which was not logged); for Matrix, change the password of any user named in the homeserver URL. Then re-enter the channel's settings and test it.
- Where your retention keeps them, delete the old lines, e.g. with Loki's
delete API for the
alerting-svcstream, or let retention expire them.
Rollback
- To rc.3: not tested. rc.4's migrations stay applied: rc.3's init
containers find nothing newer than their own newest migration and log
Database schema is up to date. The baseline revisions from0038are ordinary revisions to rc.3.0004's index,resource_counttable and event stay and keep counting; rc.3's ingestion-svc does not read them, and a later rc.4 finds the counts right. Switching the image tags back should therefore work. For a clean rc.3, restore the pre-upgrade backup into empty databases under rc.3's images. - gateway-svc: roll inventory-svc back before, or together with, gateway-svc, never gateway-svc alone (§1).
- What rc.3 brings back: a dead database connection again needs an
inventory-svc restart; rc.3 agents run
execkubeconfigs again, and rc.3's inventory-svc stores new ones; scoped users can again setlocal_collectorand attach credentials at any host; channels can again point at loopback or hub services. - Compose file: an rc.3
compose.yamldrops the restart policy ofsurrealdb,natsandvictoriametrics. - Helm:
helm rollbackto the rc.3 revision restores that revision's stored values, its tag pin included.
Verification
- Every init container logs
Database schema is up to dateor applies only newer migrations, and the newest applied migration (SELECT VALUE migration FROM schema_migration ORDER BY applied_at DESC LIMIT 1;) is0038_baseline_revisionsininventory,0004_resource_state_boundiningestion,0015_mute_clusterinalerting,0012_nats_user_tokeninauthand0001_controlincontrol. - In the
ingestiondatabase,SELECT meta::id(id) AS location, n FROM resource_count;lists each location with its row count, and for any one locationnequalsSELECT count() FROM resource_state WHERE location = '<id>' GROUP ALL;. - The core agent logs
core cycle complete: N location(s)with your number of local-collector locations, and no check of yours reportsreason="blocked_target"unless you meant it to. - No
k8s_clusterresource shows a collect error startingkubeconfig refused:orrefused:(§5). - Each notification channel's Test succeeds. A 400 naming loopback or a hub service means the channel must be re-pointed.
- A control
pingon any resource, and the location page's Test control path, returnpong. - A user whose name is not ASCII appears under that name in the control audit log after an action.
- [required] Take a fresh backup of all five databases (as in Do this first step 1), and from now on restore only backups taken under rc.4.
Known issues
- The database reconnect issue is fixed (summary, "Database: reconnect"): drop the rc.3 rule "restart inventory-svc after any SurrealDB restart" from your runbooks once every service runs rc.4. It still applies while any service runs rc.3, e.g. when SurrealDB is restarted before the services are upgraded.
- Restoring a backup from an older release onto rc.4** imports that
release's schema definitions and migration stamps over rc.4's, as the
rc.3 notes describe. rc.4's migrations add
only new definitions and data (
0004's index, table and event;0038's revisions), so less is undone than for rc.3, but a restoredresource_stateno longer matches theresource_countrows. Take a fresh backup right after upgrading. - Not covered by the destination check: services alerting-svc's config does not name, such as Loki. Keep them off networks alerting-svc can reach, or behind authentication.
- Unchanged from rc.3: alerts can outlive a deleted resource when ingestion-svc cannot be reached during the delete, or for rows an rc.3 or older agent wrote; and the issues still open from the rc.3 notes.