// Upgrades
Upgrading a running Fleeger hub to 1.0.0-rc.5
From: 1.0.0-rc.4. To: 1.0.0-rc.5. For an older hub, upgrade to rc.4
first with the rc.4 notes.
Each step is tagged [required], [recommended] or [if …]. > Backups taken on rc.5 do not restore on rc.4 or older. The Backup page's
.fbkarchive has a new format. rc.4 refuses an rc.5 archive with the misleading message not a Fleeger backup archive (an archive over 256 MiB is refused for its size first). If you may roll back, take a backup before upgrading (Do this first). Backups taken before the upgrade still restore on rc.5.
No permission changed since rc.4, no migration was added, and no config key was added. The upgrade is mostly about backups: downloads and restores are now streamed end to end, and restores have no size cap.
Summary
| Area | Change | Manual work |
|---|---|---|
| Backup: archive format v2 | The Backup page's download (POST /api/backup/download, inventory-svc's /backup/export) now writes archive format v2 (FLGRBK02). Each part (each database dump, then the metrics) goes out in 1 MiB blocks as its store sends it, so inventory-svc exports in constant memory (about a MiB) and a backup with metrics fits the services' standard 256Mi memory limit. Backups taken before the upgrade (v1) still restore. A backup taken on rc.5 restores only on rc.5 or newer: rc.4 and older refuse it as not a Fleeger backup archive (an archive over 256 MiB is refused for its size first), which really means "made by a newer Fleeger". From now on an archive from a newer Fleeger is refused as this backup archive was made by a newer Fleeger; restore it there, and one whose framing does not hold together as corrupt backup archive, with nothing imported. The scheduled S3 path (scripts/backup.sh, the backup CronJob) and scripts/restore.sh did not change. |
[required, if you may roll back] take a backup under rc.4 before the upgrade, and keep it until you are sure you stay on rc.5 (§1). [required] take a fresh one after the upgrade (Verification step 6). |
| Upgrade order | No new header passes between the services, so no service must go first. A restore from the Backup page needs frontend and gateway-svc on the same release: the rc.5 page sends the .fbk file itself as the body, which rc.4's gateway-svc, which reads only the JSON form, answers with 400 invalid restore request (an archive over 256 MiB is refused for its size first). A download running in the browser while frontend is replaced may fail and must be started again. |
[recommended] Do not download or restore a backup while the rollout is in progress. Roll frontend and gateway-svc back together, if ever. |
| Backup: restore streamed, no size cap | POST /api/backup/restore now takes the .fbk file itself as the body (Content-Type: application/octet-stream), with the password percent-encoded (as encodeURIComponent writes it) in the X-Fleeger-Password header. There is no size cap: the gateway checks the caller and the password before it reads any of the upload, never forwards the password, and relays the upload to inventory-svc as it arrives; inventory-svc decodes it into an anonymous temp file in its TMPDIR (/tmp), which needs free space of about three quarters of the file's size and is removed when the restore ends. Nothing is imported unless the whole upload arrived and its authentication tag checks out; an upload that breaks off, does not decrypt, or fills the disk imports nothing. The old JSON form {"password": …, "archive": …} still works, read whole and capped at 256 MiB as before. Receiving the upload counts against inventory-svc's [backup] timeout_secs and the gateway's [http] backup_timeout_secs. The edge's location = /api/backup/restore now has client_max_body_size 0 (compose, raw manifests, chart); the chart's ingress.bodySize default is now "0" and, whenever it is set, the chart's Ingress also gets proxy-request-buffering: "off"; the example deploy/k8s/optional/ingress.yaml sets both. |
[required] inventory-svc's /tmp needs room for about 0.75x the largest backup you will restore (§2 step 6, §3 step 6). [if] you run your own Ingress, reverse proxy or edge config: lift the body limit and request buffering on that route (§5). [if, Helm] a stored ingress.bodySize (e.g. 256m): set it to the string "0" (§4 step 2). |
| Backup page: download saved straight to disk | The Backup page now saves a download to disk as it arrives, through a service worker at /backup-download/sw.js whose scope is /backup-download/ only. The page still sends the request itself, so the session token and password never reach the worker. The browser shows the download in its own download list with progress, writes it no faster than the disk takes it, and marks it failed if it breaks off. The page has a Cancel button; cancelling there or in the browser, or closing the tab, stops the export on the server at once (after a closed tab, the browser marks the download failed within about three minutes). Leaving the Backup page does not stop a download. The page falls back to collecting the archive in memory and saving it at the end, as before, and says so: over plain HTTP at any address other than localhost, 127.0.0.1 or ::1 (service workers need a secure context), in some private windows, when /backup-download/sw.js cannot be loaded or its worker does not activate within 10 s, and in Safari and every browser on iOS (all WebKit), whose service-worker downloads are not reliable. In that mode a large backup (with metrics) may not fit in the tab's memory. No API change. |
[if] a reverse proxy in front of Fleeger filters paths: let /backup-download/ through to the frontend (§5). [recommended] download backups with metrics over HTTPS in Chrome, Edge or Firefox. |
| Backup: download relayed by the gateway | gateway-svc relays the archive to the browser as inventory-svc sends it, instead of holding it whole in memory. [http] backup_timeout_secs (default 3600) now also covers relaying the archive. A download cut off after it started (inventory-svc's connection lost, or out of that budget) ends as a broken download (the connection is aborted and the Backup page shows an error), never as a shorter file that looks whole; before, it was answered 502 or 504. A failure before the download starts is still 502/504. |
None. |
| Backup: export streamed by inventory-svc | POST /backup/export streams the archive instead of building it whole first, and with format v2 also each part. Because the stores are read only as fast as the download proceeds, [backup] timeout_secs (inventory-svc, default 3000) now covers practically the whole download, not just reading the stores: a download still running when it is spent breaks off (it is reported as a spent budget). A store that fails before the first MiB of the first part is an error the page shows; one that fails later breaks the download, and inventory-svc logs why. |
[if] downloads with metrics over a slow link break off: raise inventory-svc's [backup] timeout_secs and gateway-svc's [http] backup_timeout_secs together, keeping the first below the second. The chart exposes neither key. |
| Backup: export errors shown | A download that inventory-svc refuses or fails before sending the archive now reaches the browser with inventory-svc's own status, content type and JSON error, and without the fleeger-backup.fbk attachment header, so it is not offered as a file; the Backup page shows its message instead of "Failed (status)". An error body over 1 MiB becomes 502. |
None. |
| Ingestion: offboarding deletes logs | Offboarding a location (DELETE /api/agents/{id}), and deleting a location, which offboards it, now also deletes the location's logs from Loki when ingestion-svc has [loki] configured, through Loki's compactor delete API. The delete covers all of the location's log history. If Loki refuses it (4xx, e.g. compactor deletion not enabled), the offboard still completes and ingestion-svc logs offboard: Loki refused to delete the logs of '<location>'; those logs then stay until Loki's own retention drops them. If Loki cannot be reached, the offboard fails before anything is cleared and can be run again. Loki removes the lines when its compactor next runs. |
[if] you run your own Loki: enable compactor deletion (§6). The shipped Loki configs have it. |
| Loki: purges cover all history | The retention reaper's delete of a reaped resource's logs now covers all of the streams' history (start at the Unix epoch), however long Loki's retention_period is; before, it covered only the retention policy's loki_retention_secs window (7 days when unset). The new offboard delete (row above) does the same. The delete no longer sends end, so the interval ends at Loki's own now: before, with the hub's clock even a second ahead of Loki's, Loki refused every purge with 400 deletes in the future are not allowed, which was logged as a warning and never retried. |
None. Logs of resources reaped under rc.4 (outside its window, or refused for the clock) and of locations offboarded under rc.4 stay until Loki's retention_period drops them. |
Retention policy: loki_retention_secs removed |
The Retention page no longer shows a Loki retention field; its section is now "Metrics retention bound". GET /api/retention, the PUT and rollback responses, and newly journalled snapshots no longer carry loki_retention_secs. A PUT that still sends it is accepted and the value ignored; a stored value is no longer served and is dropped on the next save. Rolling back to an old policy snapshot that carries it still works. Nothing reads it any more. No migration: the schema field stays defined. |
[if] scripts read loki_retention_secs from GET /api/retention: stop doing so. |
| Alerting API: control characters refused on save | Saving an alert rule whose name or scope_location, or a notification channel whose name or any string in its config (URLs, ntfy topic and token, Matrix room id and token, SMTP host, user, password and addresses), contains a control character (line break, tab, NUL, DEL, a C1 code) is refused with 400 naming the field, never the value. A scope is checked after its padding is trimmed, so a trailing line break is still just trimmed off. A rule query may keep line breaks and tabs (\n, \r, \t) but refuses any other control character. Non-ASCII is fine, and a rule description may still hold line breaks. On an update only a new name, scope or query is checked, so a client that sends back the value it read is not refused; a channel config sent on an update is always checked. Stored rows are not changed or migrated. |
[if] scripts save such values: strip them. To clean a stored one, edit the rule's name, scope or query, or re-enter the channel's name or config. |
| Alerting API: name length on update | PUT /api/alert-rules/{id} and PUT /api/notification-channels/{id} now refuse a new name that is empty or over 100 characters (not bytes) with 400, as create does. Before, an empty one was a 500 and a long one was stored. A name sent back unchanged is not checked, so a row with an over-long name keeps its other fields editable. |
None; rename any rule or channel with an over-long name when convenient. |
| Alerting: control characters in ntfy titles | An ntfy notification's title (from the rule name) travels in an HTTP header; a control character in it, such as a line break, is now sent as a space, and non-ASCII is kept. Before, such a page failed before it was sent and was retried every cycle. A channel whose stored config cannot form a valid request at all (e.g. a token with a line break) is now reported as unusable: the alert is stamped and warned about once per renotify window, and the channel's test returns 400 … request could not be built from the channel config: …, without the value. |
None; fix any channel the new warning names (re-enter its config). |
| API: scoped answers over 1 MiB | A location-scoped caller now gets /api/locations, /api/metrics/state/locations and /api/agents/connz narrowed when the answer is between 1 and 16 MiB, instead of a 500; a larger answer is still refused. A control mutation body of up to 4 MiB (was 1 MiB) is read for its location and accepted, matching the proxy's limit; before, one between 1 and 4 MiB was a 400. |
None. |
| Gateway: proxied answers relayed as they arrive | gateway-svc no longer reads a proxied service's answer whole before passing it on. Answers it passes on unread (auth, IAM, inventory, control, alerting, the JetStream drain) are relayed as they arrive and are not capped; answers it must read to narrow them for a scoped caller are read no further than 16 MiB (a rollback summary 4 MiB, as before). The [http] timeout_secs budget now also covers relaying the body. An answer cut off after its status line now reaches the client as a broken body (connection aborted) instead of a 500 upstream service did not answer. |
None. |
| Database: migrations | None added since rc.4. The newest applied migrations stay inventory 0038_baseline_revisions, ingestion 0004_resource_state_bound, alerting 0015_mute_cluster, auth 0012_nats_user_token, control 0001_control. |
None. Check them under Verification step 1. |
Do this first
- [required] Back up under rc.4, before anything is upgraded. This is
the only backup you can restore if you roll back to rc.4:
- the scheduled path, all five SurrealDB databases (
auth inventory ingestion alerting control, namespacefleeger), e.g. a one-off run of the backup job:kubectl -n fleeger create job pre-rc5 --from=cronjob/<name>, where<name>isfleeger-backupwith the raw manifests and<release>-fleeger-hub-backupwith Helm (find it withkubectl -n fleeger get cronjob). Its format did not change, soscripts/restore.shrestores it on either release; and/or - a
.fbkdownload from the Backup page while the hub still runs rc.4. rc.4 restores an archive of at most 256 MiB at every hop, so for a backup you may restore on rc.4, leave metrics out unless you know the archive stays under that.
- the scheduled path, all five SurrealDB databases (
- [required] Check that the hub runs rc.4's schema. The newest applied
migration
(
SELECT VALUE migration FROM schema_migration ORDER BY applied_at DESC LIMIT 1;) must be0038_baseline_revisionsininventory,0004_resource_state_boundiningestion,0015_mute_clusterinalerting,0012_nats_user_tokeninauthand0001_controlincontrol. Anything older: upgrade to rc.4 first. - [recommended] Let any backup download or restore finish before you start, and start none until the rollout is done (see "Upgrade order").
1. Backups before and after the upgrade
[required] on every install path.
- The archive format changed: rc.5 writes
.fbkformat v2. rc.5 restores v1 archives (taken on rc.4 or older) and v2 archives. rc.4 and older restore only v1, and refuse a v2 archive as not a Fleeger backup archive (an archive over 256 MiB is refused for its size first). That message does not mean the file is damaged or that the password or key is wrong; it means the archive was made by a newer Fleeger. - So once the hub runs rc.5, every new
.fbkis useless for going back to rc.4. Keep the backup from Do this first step 1 until you are sure you stay on rc.5. - After the upgrade, take a fresh backup (Verification
step 6). With rc.5 a backup with metrics fits inventory-svc's memory limit,
and any backup that downloads can be restored, provided
inventory-svc's
/tmphas room for about three quarters of it. - The scheduled S3 path (
scripts/backup.sh, the backup CronJob,scripts/restore.sh) is not affected: its sets restore the same way on rc.4 and rc.5.
2. Docker Compose (deploy/compose/)
- [required] Do this first.
- [required] In
.env, setTAG=1.0.0-rc.5(or build from the rc.5 tag), thendocker compose pull. Ofdeploy/compose/, onlyedge.nginx.confchanged since rc.4: inlocation = /api/backup/restore,client_max_body_size 256m;becameclient_max_body_size 0;, plus comments. If you edited that file (e.g.set_real_ip_from), make that one change in your copy rather than overwriting it. - [required]
docker compose --profile core-agent up -d(drop the profile if you do not run the bundled core agent). The*-migrateone-shots logDatabase schema is up to date. - [required]
docker compose restart edge, so nginx reads the newedge.nginx.conf.up -ddoes not recreate the edge for a change in its mounted file. Without it, a restore over 256 MiB is refused at the edge. - [if] downloads with metrics over a slow link break off:
raise
timeout_secsunder[backup]inconfig/inventory-svc.tomlandbackup_timeout_secsunder[http]inconfig/gateway-svc.toml, keeping the first below the second, and restart both services. - [required] inventory-svc's
/tmpis in its container's writable layer, i.e. on Docker's data disk: it needs free space for about 0.75x the largest backup you will restore.
3. Raw manifests (deploy/k8s/)
Changed since rc.4: the fleeger-edge ConfigMap in base/10-config.yaml and
optional/ingress.yaml.
- [required] Do this first.
- [required] Copy the images into the registry your cluster pulls from,
if that is not the GitLab registry, as for rc.4 (
docker buildx imagetools create --tag <yours>/<image>:1.0.0-rc.5 registry.thatsnasu.dev/netcode/fleeger/<image>:1.0.0-rc.5). - [required] Set the image tag to
1.0.0-rc.5for every Fleeger image, includingfrontend,backupand the core agent (spoke-agent), and roll out. No service must go first; every init container logsDatabase schema is up to date. - [required] In the
fleeger-edgeConfigMap, changeclient_max_body_size 256m;toclient_max_body_size 0;inlocation = /api/backup/restore, keeping your ownset_real_ip_from, apply it, and restart the edge, which does not reload its config by itself:kubectl -n fleeger rollout restart deploy/edge. - [if] you use
optional/ingress.yamlor an Ingress like it: setnginx.ingress.kubernetes.io/proxy-body-size: "0"andnginx.ingress.kubernetes.io/proxy-request-buffering: "off"(§5). - [required] inventory-svc's
/tmpis anemptyDirwithout a size limit, so it uses the node's ephemeral storage: the node needs free space for about 0.75x the largest backup you will restore. - [if] you run your own Loki: §6.
4. Helm chart (charts/fleeger-hub)
[required] Do this first.
[if]
helm get values fleeger -n fleegershows aningress.bodySize(e.g.256m): a stored value is kept by--reset-then-reuse-valuesand overrides the new default"0", so a restore over it is refused at the Ingress. Add--set-string ingress.bodySize=0in step 3. Use--set-string, not--set:--setmakes it the number0, which the chart reads as unset and renders no body-size annotation at all, leaving ingress-nginx's 1m default. In a values file, writebodySize: "0", quoted. Likewise if you upgrade with plain--reuse-values, which keeps rc.4's default256m.[if]
helm get valuesshows a body-size or buffering annotation underingress.annotations(e.g.nginx.ingress.kubernetes.io/proxy-body-size: 256m), or you use another Ingress controller: the chart merges your annotations over its own, so yours win andingress.bodySizechanges nothing. Set the size to unlimited and turn request buffering off there (§5).[required] Upgrade the release:
helm upgrade fleeger oci://<registry>/charts/fleeger-hub --version 1.0.0-rc.5 \ -n fleeger --reset-then-reuse-valuesNo service must go first, so no
kubectl set imagebeforehand. The edge rolls by itself (its config checksum changed), and the init containers logDatabase schema is up to date. If a storedimage.tagis still shown byhelm get values, clear it as in the rc.4 notes.[required] inventory-svc's
/tmpis anemptyDirwithout a size limit: the node needs ephemeral storage for about 0.75x the largest backup you will restore.[if] you run
fleeger-agentreleases: no agent code and no agent chart changed since rc.4.[if]
loki.urlis set: §6.
5. A proxy or Ingress you run yourself
[if] anything sits in front of Fleeger's edge besides what the chart or the
example manifest renders, or your chart release sets its own body-size or
buffering annotations under ingress.annotations or runs another Ingress
controller.
- Restore body. A restore now uploads the whole
.fbkas one request of any size. On the route/api/backup/restore, lift the body limit and request buffering, or the proxy refuses or spools large restores:- ingress-nginx:
nginx.ingress.kubernetes.io/proxy-body-size: "0"andnginx.ingress.kubernetes.io/proxy-request-buffering: "off"; - nginx:
client_max_body_size 0;andproxy_request_buffering off;inlocation = /api/backup/restoreonly, not for all of/api/, which includes routes reachable without authentication; - other controllers: their own body-size and buffering settings.
- ingress-nginx:
- Download worker. If the proxy filters paths, let
/backup-download/through to the frontend. Otherwise/backup-download/sw.jscannot be loaded, and the Backup page falls back to collecting downloads in memory. - Timeouts. The edge gives both backup routes an hour; a proxy in front with a shorter read or send timeout cuts long downloads and restores off.
- Password header. The restore password now travels percent-encoded in
the
X-Fleeger-Passwordrequest header on/api/backup/restore; before, it was in the JSON body. A proxy, WAF or Ingress in front must not log or strip that header.
6. A Loki you run yourself
[if] loki.url (Helm) or loki_url and [loki] (raw manifests) point at a
Loki you run. For offboarding to delete a location's logs, and for
the retention reaper to delete a resource's, Loki's compactor deletion must be
on, as in the shipped deploy/loki/loki-config.yml:
limits_config:
deletion_mode: filter-and-delete
compactor:
retention_enabled: true
delete_request_store: filesystem
and restart Loki. Without it, Loki refuses the delete, the offboard completes
anyway, ingestion-svc logs offboard: Loki refused to delete the logs of '<location>', and the logs stay until Loki's retention_period drops them.
Keep the limits from the rc.4 notes. Loki's clock no longer matters to the
delete, but keep the hub's and Loki's clocks in sync anyway.
Rollback
- To rc.4: not tested. No migration was added, so switching the image tags
back is all the databases need. A retention policy saved on rc.5 no longer
holds
loki_retention_secs; the schema field is still defined, so rc.4 reads it as unset (its 7-day default) and can store it again. - Backups: rc.4 cannot restore any
.fbktaken on rc.5; it answers not a Fleeger backup archive (an archive over 256 MiB is refused for its size first). Restore the backup from Do this first step 1, or a scheduled set. rc.4 restores at most 256 MiB at every hop again. - frontend and gateway-svc: roll them back together. An rc.5 page in
front of an rc.4 gateway-svc cannot restore (400
invalid restore request; an archive over 256 MiB is refused for its size first); reload the page after the rollback. - What rc.4 brings back: the gateway and inventory-svc hold a whole backup in memory again; restores are capped at 256 MiB; downloads are collected in the tab's memory; offboarding leaves logs in Loki, and Loki purges cover only the policy window and fail when the hub's clock runs ahead of Loki's; control characters and over-long names can be saved again; scoped callers get a 500 for list answers over 1 MiB.
- Edge and Ingress: rc.4's
edge.nginx.conf,fleeger-edgeConfigMap and chart put the restore route back to 256m. A body limit of"0"on your own Ingress or proxy can stay. - Helm:
helm rollbackto the rc.4 revision restores that revision's stored values.
Verification
- Every init container logs
Database schema is up to date, and the newest applied migrations are still those in Do this first step 2. - On the Backup page over HTTPS in Chrome, Edge or Firefox, start a download:
it appears in the browser's own download list with progress, and the page
shows no note about collecting it in memory. If the note appears there,
check that
https://<your host>/backup-download/sw.jsloads (§5). GET /api/retentionno longer returnsloki_retention_secs, and the Retention page shows "Metrics retention bound" without a Loki field.- Each notification channel's Test succeeds. A 400 that says the request could not be built from the channel config means the stored config holds a control character: re-enter it.
- [recommended] Restore a backup taken on rc.5 into a scratch hub that shares the master key, as in the restore drill. That also checks that your Ingress or proxy lets a large restore through.
- [required] Take a fresh backup of all five databases (as in Do this first step 1), and keep the rc.4 one as long as you may roll back.
Known issues
- A v2 archive on rc.4 reads as "not a Fleeger backup archive". rc.4 cannot tell a newer format from a file that is not a backup (an archive over 256 MiB is refused for its size first). rc.5 and later say made by a newer Fleeger; restore it there for any future format.
- An archive holds at most 64 GiB of plaintext (the limit of AES-GCM for
one value; the base64
.fbkfile is about 85 GiB at that point). A larger backup, in practice one with a very large metrics history, breaks off there, and inventory-svc logsencrypt backup. - A store answer that ends cleanly but short (the store stops early without an error) is taken as the whole part; the archive cannot tell it from a complete one. Check a restored platform as in the restore drill.
- Safari, iOS and plain HTTP still collect a download in the tab's memory , which may not fit a large backup with metrics.
- A download fails if the browser stops the download worker, or a Fleeger update retires it, mid-download; the page says so. Start it again.
- Logs that rc.4 did not purge stay in Loki: those of locations offboarded
before the upgrade, and of resources reaped outside rc.4's window or while
the hub's clock ran ahead of Loki's. Loki's own
retention_perioddrops them. - Restoring a backup from an older release** still imports that release's schema definitions and migration stamps, as the rc.4 notes describe. rc.5 added no migration, so a backup taken on rc.4 brings back rc.5's own schema.
- Unchanged from rc.4: the issues still open from the
rc.4 notes, including
resource_countafter a restore.