// Betrieb
The audit log
The hub keeps one audit log of who did what: which user, from which
address, did which action to what, with which parameters, and how it ended.
It is inventory-svc's, in its own SurrealDB database audit (namespace
fleeger), so that no service's own database, and no restore, can take it
away.
What is recorded
Every audited action is journalled in two rows that share a
correlation_id:
- a
proposedrow, written before the action is done, and - a terminal row after it:
completed,failed, orrejected(refused by a check, nothing done).
An action refused before anything was proposed (a wrong password for a backup
download, a control action of an unknown kind) has a single rejected row.
Action (action) |
Service | Target |
|---|---|---|
nats.stream.drain: a JetStream drain |
ingestion |
stream:<name> |
backup.download: an encrypted backup download |
gateway |
platform |
backup.export: an encrypted backup written to a file (inventory-svc backup export) |
inventory |
platform |
backup.restore: an offline restore |
inventory (inventory-svc backup restore), restore.sh |
platform |
audit.prune: the audit log's own retention prune |
inventory |
audit_event |
control.<action>: a control action (control.ping, control.k8s, ...) |
control |
the resource id |
location.delete: a location delete (DELETE /api/locations/{id}) |
gateway |
location:<id> |
agent.offboard: an agent offboard (DELETE /api/agents/{id}) |
gateway |
location:<id> |
registration_token.create: a registration token minted (POST /api/agent-tokens) |
gateway |
location:<id> |
nats.creds.mint: an agent's NATS credential minted by hand (auth-svc mint-creds) |
auth |
location:<id> |
nats.hub_user.revoke: a hub user's earlier NATS JWTs revoked in the revocation ledger (auth-svc nats revoke-hub-user, run by gen-auth.sh fix-perms core-agent) |
auth |
nats_hub_user:<name> |
location.assignments.remove: auth-svc's startup cleanup removing the role assignments of a location inventory-svc no longer has (actor system) |
auth |
location:<id> |
location.rollback: a checkpoint rollback (POST /api/locations/{id}/rollback) |
inventory |
location:<id> |
location.rollback.unlock: a rollback lease lifted by hand (POST /api/locations/{id}/rollback/unlock) |
inventory |
location:<id> |
user.create: a user created (POST /api/users, auth-svc user add) |
auth |
user:<name> |
user.update: a user's email, direct permissions or password changed (PUT /api/users/{name}) |
auth |
user:<name> |
user.delete: a user deleted (DELETE /api/users/{name}, auth-svc user delete) |
auth |
user:<name> |
user.sessions.end: every session of a user ended (POST /api/users/{name}/end-sessions) |
auth |
user:<name> |
user.role.assign: a role assigned (POST /api/users/{name}/roles, auth-svc user assign-role) |
auth |
user:<name> |
user.role.unassign: a role assignment removed (POST /api/users/{name}/unassign) |
auth |
user:<name> |
role.create: a role created (POST /api/roles) |
auth |
role:<name> |
role.update: a role's permissions replaced (PUT /api/roles/{name}) |
auth |
role:<name> |
role.delete: a role and every assignment of it deleted (DELETE /api/roles/{name}) |
auth |
role:<name> |
alert_rule.create: an alert rule created (POST /api/alert-rules) |
alerting |
alert_rule:<id> |
alert_rule.update: an alert rule edited, switched on or off (PUT /api/alert-rules/{id}) |
alerting |
alert_rule:<id> |
alert_rule.delete: an alert rule deleted, its firing alerts retired (DELETE /api/alert-rules/{id}) |
alerting |
alert_rule:<id> |
notification_channel.create: a notification channel created (POST /api/notification-channels) |
alerting |
notification_channel:<id> |
notification_channel.update: a channel's name, enabled or config changed (PUT /api/notification-channels/{id}) |
alerting |
notification_channel:<id> |
notification_channel.delete: a channel deleted and detached from its rules (DELETE /api/notification-channels/{id}) |
alerting |
notification_channel:<id> |
mute.create: a mute or maintenance window created (POST /api/mutes) |
alerting |
mute:<id> |
mute.delete: a mute ended by hand (DELETE /api/mutes/{id}) |
alerting |
mute:<id> |
explorer.panel.delete: a saved explorer panel deleted, and dropped from its presets (DELETE /api/explorer/panels/{id}) |
inventory |
explorer_panel:<id> |
explorer.preset.delete: a saved explorer preset deleted (DELETE /api/explorer/presets/{id}) |
inventory |
explorer_preset:<id> |
iam.import_legacy: users, roles and assignments imported from the old monolith (auth-svc migrate-legacy) |
auth |
legacy:<namespace>/<database> |
db.unlock: a migration lock released (<service> db unlock --force) |
the service whose command ran: auth, inventory, alerting, ingestion, control |
database:<name> (inventory-svc's: database:inventory, database:audit) |
secrets.rekey: stored secrets re-encrypted under the primary key (inventory-svc rekey, alerting-svc rekey) |
inventory, alerting |
table:credential, table:notification_channel |
Each row of a location-scoped action has location set to the location's
id.
Where each row is written
A row is written by the service that carries the action out:
An action spread over several services is written by the gateway, which runs it: a location delete (delete check, NATS revoke in auth-svc, state and metrics purge in ingestion-svc, alert clear in alerting-svc, the delete in inventory-svc, then the removal of its role assignments in auth-svc, whose step names the removed
user:rolepairs and never fails the delete) and an agent offboard (purge, revoke, alert clear). Oneproposedrow is written before the first step and one terminal row after the last, whoseresultlists what each step did, in order, e.g.delete check: ok; revoke: ok; purge: failed (ingestion returned 500 ...); alert clear: not tried (the purge failed). A delete the delete check refuses (nothing done) endsrejected; one that stopped part way endsfailed, and its steps say how far it got.A registration token mint is written by the gateway too: it is the only way to auth-svc's mint, which is one database write.
rejectedis a request auth-svc refused (nothing minted). The token itself is never recorded. The row is written before auth-svc checks the location, so a location that is not one plain path segment of at most 200 bytes is not put intargetandlocation: the row has targetlocation:?, nolocation, and the location as sent (cut at 200 characters) inparameters.location.A checkpoint rollback and a rollback unlock are written by inventory-svc, which does them, straight into its audit database. A rollback's parameters hold the checkpoint and the mode, its
completedrow what it changed; one refused with nothing changed (an unknown checkpoint, a location being rolled back) endsrejected, one whose outcome is unknownfailed, saying so. An unlock'scompletedrow names whose lease it lifted;rejectedmeans none was held. The rollback's own changes are journalled in the location's config history as before.A user or role change is written by auth-svc, which makes it. The
proposedrow's parameters say what changes, with what it was where the request does not say: a create holds the email, direct permissions and assignments asked for;user.updatelists thefieldsset and holds each one'sbeforeandafter(a new password only aspassword_changed: true);role.updateholds the permissionsbeforeandafter; a delete holds what was deleted (before: a user's email, direct permissions and assignments; a role's permissions and who held it); an assign or unassign the role and location (also in the row'slocationwhen the assignment is scoped to one that is one line of at most 256 bytes).beforeis read just before the change (nullwhen there is no such user or role; the change is then refused). auth-svc puts no password, password hash, token or session id in a row's parameters; a terminal row'sresult(the outcome, or the error) is cleared of the request's password, as sent and as JSON or RustDebugwould quote it, and of any argon2 hash ($argon2…, through its salt and digest). A change refused by a check (a missing user or role, a built-in role, the last global admin, a stale role edit, an invalid field) endsrejected, nothing changed. A name that is not one line of at most 200 bytes is not put intarget(user:?,role:?) but inparameters.name, cut at 200 characters. Parameters over the row's 16 KiB (a role held by thousands of users) are cut to their start, markedtruncated, rather than refusing the change. Sign-in and/api/auth/meare not audited and never reach inventory-svc. The auth-svc command-line tools write the same rows; see The command-line tools.An alert rule, notification channel or mute change is written by alerting-svc, which makes it. A create's id is drawn before its
proposedrow, so both rows name the new object; the parameters hold it as asked for. An edit's parameters list thefieldsit sets, each with itsbefore(read just before the change, as auth-svc does) andafter; a rule edit also itsexpected_updated_at. A delete's hold what was deleted (before: the rule; a channel's name, kind and the rules it is detached from,routed_from; the mute). A rule scoped to a location, a location mute and a resource mute scoped to a location also set the row'slocation(as for an assignment, only a value a row can hold). A change refused by a check (an invalid rule or channel, a built-in rule's delete, a missing object, a stale rule edit,STALE, or one that kept conflicting,CONFLICT) endsrejected, nothing changed; a rule delete or edit that stopped part way endsfailed, saying how far it got (e.g.rule '<id>' was disabled but not deleted: retiring its firing alerts failed (…); retry the delete, orrule '<id>' was updated, but retiring its firing alerts failed: …). A delete of something already gone endscompletedwithno such …, nothing deleted, as the API answers 204; what a delete'scompletedrow says it removed is what that request's own delete removed, so one that another request deleted meanwhile is not reported as deleted by this one.A channel's config is never recorded: it holds webhook URLs, tokens and SMTP passwords. A channel row names only the config's fields the channel's kind reads (
config.fields; on an edit alsoconfig.changed, those whose value differs from the stored one,nullwhen the stored config could not be decrypted), and counts any other key (config.unknown_fields) without naming it; never a value. A refused or failed channel change's row never quotes the error's text, which can quote a config value (a parser'sinvalid type: integer …): it saysthe channel or its config was refused (400)or that the error was on alerting-svc's side, and the answer to the request (or the alerting-svc log) has the reason. Only a channel's own not-found and conflict messages are kept. As a second guard every result is cleared of every string of the config, old and new, of four or more characters. Reads, a channel's test send and the alert clear of an offboarded location (in the gateway'sagent.offboard/location.deleterow) write no row.An explorer panel or preset delete is written by inventory-svc, straight into its audit database. The
proposedrow'sbeforeholds the panel (title, query, mode, viz, unit, who created it, andin_presets, the presets it is dropped from) or the preset (name, panel ids, who created it);null, and the deleterejected, when there is none (or another request deleted it meanwhile). A panel deleted whose removal from its presets then failed endsfailed, saying the panel was deleted. Creating and editing panels and presets is not audited: each row keeps its owncreated_by.
A name a row's target cannot hold (over 200 bytes, or not one line) goes in
parameters.name with target <kind>:?, for every service that writes rows
for its own API (auth-svc, alerting-svc, inventory-svc's explorer).
- A command-line tool's change is written by the tool itself; see The command-line tools.
The gateway's calls for each step of a location delete or an agent offboard
(auth-svc's revoke and assignment removal, ingestion-svc's purge,
alerting-svc's alert clear) carry
the acting user and their address (x-fleeger-actor, x-fleeger-client-ip),
as every request the gateway proxies does, so each service knows who acted.
The gateway removes both headers from every request a client sends to
/api, public routes (login, bootstrap) included, so a client can never
supply them; only the auth middleware stamps them again.
A terminal row's result can quote a service's own error text (a database
error, an upstream's answer) as the service saw it, including internal detail
the API answer leaves out. It is readable by whoever holds audit.view.
A request refused before the action is proposed (a malformed id, the meta
location, a reserved pseudo-location; for a rollback or an unlock, a location
id or checkpoint id that is not one line of at most 256 bytes, answered 400)
writes no row: nothing was attempted. So does an IAM change whose current
state (before) cannot be read (500).
A drain preview (dry_run) is not audited: it changes nothing. A drain is no
longer written to the retention sweep log; the reapers' own sweeps still are.
Each row holds correlation_id, phase, actor, client_ip (the address
the edge reported, as the gateway stamps it; empty when unknown), service,
action, target, location (for a location-scoped action), parameters
(JSON), result (on a terminal row) and ts. A row copied from control-svc's
old log also has legacy_id.
The command-line tools
The offline command-line tools run with no inventory-svc to call, so, like the
restore (inventory-svc backup restore, scripts/restore.sh), they write
their rows straight into the audit database, with the database credentials
the service already has: the same SurrealDB, namespace and credentials as the
service's own database, database audit. No new setting. They write through
the same code as inventory-svc's POST /internal/audit
(fleeger_shared::audit::insert), so a row is checked (the same 400 reasons,
the same cuts) and stored identically whichever way it comes. That write
refuses a row while the audit database is not migrated (no audit_event
table, or no append_only event on it), and never creates the table on the
fly.
| Command | Row |
|---|---|
<service> db unlock --force (every service with a database) |
db.unlock on database:<name>, one action per lock held; the parameters name who held the lock and since when (lock.instance_id, lock.locked_at) and whether --without-audit was given |
inventory-svc rekey, alerting-svc rekey |
secrets.rekey on table:credential / table:notification_channel; parameters: the table, the field and how many retired keys the ring holds; completed with the counts; a run that stops on an error ends failed saying how far it got (failed: stopped after N … re-encrypted, M already current, …: <error>): what it re-encrypted stays re-encrypted |
auth-svc user add |
user.create, as the API's: email, no direct permissions, the role as its one assignment |
auth-svc user delete |
user.delete, before as the API's (null when there is no such user: completed, no such user, nothing deleted) |
auth-svc user assign-role |
user.role.assign, the role, location null (global) |
auth-svc migrate-legacy |
iam.import_legacy; parameters: the legacy namespace and database; completed with the counts and the names of the users and roles imported; one that fails part way ends failed naming what it had already written (failed: stopped after importing …; then <error>) |
auth-svc mint-creds --location <id> |
nats.creds.mint on location:<id>, location set; parameters: the location, the minted user's public key (user_public_key, what the revocation ledger records and offboarding revokes), the signing account's public key (account_public_key) and expires: null (the credential does not expire). proposed before the ledger row (when it cannot be written, nothing is recorded or printed); once the ledger row is written, completed is attempted, best effort, and the credential is printed: a terminal row that cannot be written never withholds a credential that is already in the ledger (it is revocable), so such a mint shows a proposed row only. failed when the ledger refuses the row (nothing printed). A location the command refuses (not a valid location id, a reserved one, the hub's own) is refused before anything is connected, with no row |
auth-svc nats revoke-hub-user --user <name> --pubkey <U…> --at <s> |
nats.hub_user.revoke on nats_hub_user:<name>; parameters: the hub user's name (hub_user), its public key (user_public_key) and the second at or before which its JWTs are refused (revoked_at). proposed before the ledger is written (when it cannot be, nothing is changed), completed after; a push to the resolver that fails afterwards is logged, not an error (the reconcile pushes the ledger). A name or key the command refuses is refused before anything is written, with no row. --check only reads the ledger and writes no row |
inventory-svc backup export <file> [--metrics] |
backup.export on platform, as the Backup page's backup.download; parameters: destination: file, the file (absolute path), the archive's format (FLGRBK02), encrypted: true, the databases in it and include_metrics. proposed before the file is created; completed with the size and SHA-256 of the file as written (sha256sum gives the same) and the parts in it; one that fails part way ends failed saying how far it got (failed: stopped after N bytes, in part K of M (<part>); sealed whole before it: …; the unfinished file was removed: <error>; when the file could not be removed, the unfinished file could not be removed; delete <file> in its place). A file that already exists is refused before anything is connected, with no row |
Actor:
cli:<operating-system user>(SUDO_USER, elseUSER, elseLOGNAME), as the restore's. A command has no signed-in Fleeger user: whoever can run it holds the hub's database credentials.client_ipis empty.serviceis the service whose command ran.Fail-closed: the
proposedrow is written before the command changes anything; when it cannot be (SurrealDB unreachable, the audit database not migrated yet: runinventory-svc db migratefirst; compose'sinventory-migrateone-shot and the chart's init container do), the command exits non-zero saying why and changes nothing. The terminal row (completed,rejectedfor a refusal such as an unknown role,failed) is best effort, as for the API. A command killed by a signal before it ends leaves itsproposedrow without a terminal one.No secrets: a password is never in a row (
user addscrubs every result of it, and of any argon2 hash;migrate-legacyreads the legacy users before itsproposedrow and scrubs every result of each one's stored password as it is, argon2, bcrypt or plain text); a rekey row names no key, and its result is scrubbed of every key in the ring.mint-credsnever puts the credential in a row (no JWT, no seed: only public keys), and every result is scrubbed of the creds file, its JWT and its seed; nor is the account's seed. Abackup exportrow names no key (the archive is sealed with the credential key, the primary of the ring), and every result is scrubbed of every key in the ring. The rows go to the audit database and the command's messages to stderr:mint-credsprints only the creds file on stdout, as before, andbackup exportwrites only to its file.A
backup exportarchive holds its ownproposedrow: the audit database is read into the archive after that row is written and before the terminal one, so the archive holds the export'sproposedrow and never itscompletedorfailedrow (the same goes for a Backup page download,backup.download). On a hub rebuilt from that archive (its audit log imported into an empty audit database), that export shows aproposedrow only; its outcome is in the log of the hub it was taken from.A command does not migrate the audit database. That is inventory-svc's
db migrate(or a restore). In particulardb unlocknever does, so it never needs a lock it may be releasing.db unlock --force --without-audit:db unlock --forceis the recovery for a stuck migration lock, the audit database's own included (inventory-svc'sdb unlockreleases both its own database's lock and the audit database's). Without the flag, a lock whoseproposedrow cannot be written is not released, nor is any other: rows already written for the others endrejected, and the command names the flag. With it, such a lock is released all the same, after a warning on stderr; afterwards one terminal row (completed/failed, noproposedrow before it, parameterswithout_audit: true, result ending--without-audit: no proposed row was written before it) is tried once more, best effort, and when that fails too, stderr gets the whole row as its only record. A lock whose row can be written is journalled as usual, flag or not. Nothing held, or no--force: nothing is done and no row written, as before.The audit database takes a row whenever its
audit_eventtable andappend_onlyevent exist, locked or not, so its own lock stuck after its first migration is released and journalled in it like any other. Only a lock stuck before0001_auditcommitted (a firstdb migratethat died after taking the lock) leaves no table to write to: release it withinventory-svc db unlock --force --without-audit, then runinventory-svc db migrate.
Not audited: the read-only commands (user list, role list, db migrate,
backup restore --dry-run), inventory-svc migrate-legacy (the one-time
import of the old monolith's clusters, journalled in each location's config
history) and ingestion-svc stream migrate (the one-time replacement of an
unbounded metrics/logs JetStream stream with a bounded one, which drains
the backlog first and deletes no un-forwarded message).
Fail-closed
An audited action is refused with 503 (UNAVAILABLE) and not done when:
- its
proposedrow cannot be written (inventory-svc down, the audit database not migrated, a write that fails), or - no acting user is named: every audited route is reached through the gateway,
which stamps the verified user on each mutating request, so a missing actor
means the gateway could not identify them. The action is refused rather than
recorded as
system.
The terminal row is best effort: the action is done either way, and a row that cannot be written is logged by the service in full, as its only record.
A request that ends after its proposed row but before its outcome is
recorded (the client went away, the service is shutting down, a panic) gets
a failed row with the result interrupted: ...: the action may or may not
have been done, so check the system before retrying it. The service also logs
that row at ERROR, in full, before writing it: a service shutting down may
not get to write it, and the log line is then its only record.
The restore CLI writes its proposed row after the pre-restore backup and
before replacing any service database; when it cannot, nothing is replaced.
scripts/restore.sh does the same. Its rows' parameters are the set (the
S3 prefix or the --from-dir directory), the databases and the set's
signature: verified, key <key id>, unsigned (--allow-unsigned),
or unchecked (signed, no public key configured, --allow-unsigned).
Append-only, in the database
Every field of audit_event is READONLY, and the table's append_only
event (DEFINE EVENT ... THROW) refuses any UPDATE or DELETE of a row,
whoever sends it, the root user included. An UPDATE that changes nothing is
let through (SurrealDB does not run the event for it); it changes nothing.
Only the retention prune deletes rows: in one transaction it removes the event, deletes a batch of rows older than the cutoff, and defines the event again, so no other transaction ever sees the table without it.
What the database cannot stop, so it is a matter of who holds the root credentials:
- a root user removing the event, the table or the database (DDL), and
- an
OPTION IMPORTsession, in which SurrealDB runs no events.
Retention
audit_retention_days in the retention policy (Settings, Retention, "Audit
log"): rows older than this many days are deleted. The default is 400
days; 0 keeps the log forever; otherwise at least 30 days (and at most
100 years). A policy saved by an older page, without the field, keeps the
stored value.
The inventory reaper prunes on each cycle while the reaper is enabled
(reaper_enabled), so with the reaper disabled nothing is pruned. A prune
with nothing old enough writes nothing. Otherwise it writes a proposed
audit.prune row (actor system; its parameters hold the cutoff, the
retention in days, and the version of the retention policy that set it and
who saved that version, policy_version and policy_updated_by) before the
first delete and a completed or failed row with the count after the last:
the prune is in the log it prunes.
Rolling the retention policy back to a revision saved before the audit
retention existed keeps the current audit_retention_days: a rollback never
turns "forever" (0) into the default.
Restore
A restore never empties the audit database:
- A
.fbkarchive and ascripts/backup.shset hold the audit database like every other. - When the live audit database holds a row, the restore leaves it as it is and does not import the backup's.
- When it holds none (a new hub, a rebuilt one), the backup's audit log is imported, first, and only through a transaction in which the database itself refuses to be emptied while it holds a row.
So a restore cannot roll the audit log back, and a rebuilt hub gets its history back.
Once the live log holds any row, an archive's history is not merged in. A hub that has started and written even one row (a login refusal, a prune, the restore's own row) keeps only its own log: restore onto an empty audit database (before the services first run) to keep an archive's history, or keep the archive to read it elsewhere.
After importing an archive's audit log, the restore defines its append_only
event again right away (the archive's schema is not trusted), and
scripts/restore.sh refuses before writing anything when the live audit
database is not migrated and the set's is not imported. inventory-svc's
db migrate and start define the event again too, whatever the migration
stamps say, so no path leaves the table without it for long.
Reading it
GET /api/audit (permission audit.view, checked globally): newest
first, every filter optional and exact.
| Parameter | |
|---|---|
actor, action, target, location, service, correlation_id, phase |
exact match |
since (inclusive), until (exclusive) |
RFC 3339 |
limit |
rows per page, default 100, at most 500 |
before |
the previous page's next |
The answer is { "entries": [...], "next": "<cursor>" | null }. A page never
holds more than 500 rows, however large the log is.
audit.view is granted to the built-in admin role through *; the
operator and viewer roles do not have it (as with control.view and
alert.view). Give it to a role on the Access page.
The Audit log page (/audit in the web UI) reads it: shown in the menu
only to holders of audit.view (global), with the exact-match filters and
the time range in the page's address (so a filtered view can be linked),
pages read with before/next ("Load more"), and each row's parameters and
correlation id, with a filter that shows one action's proposed and
terminal rows together. A 503 is shown as the log being unavailable. Every
value is shown as text, never as HTML. See
reading the audit log.
The control panel's per-location history (GET /api/control/audit,
control.view at that location) reads the control rows of this log.
Upgrading
- inventory-svc's
db migratecreates and migrates theauditdatabase (services/inventory-svc/migrations-audit/) and defines theappend_onlyevent again;startrefuses to run while it has pending migrations, as for its own database, and also defines the event again. - control-svc writes its rows here from this release (
[inventory] urlincontrol-svc.toml, defaulthttp://localhost:7402). Until inventory-svc answers, control actions answer 503. Its oldaudit_logtable (db=control) is no longer written; its rows are copied into this log once, in the background, retrying until inventory-svc answers, andaudit_copy:marker(db=control) notes how far it got. Copying again changes nothing. It goes in batches of at most 512 KiB, halved when inventory-svc answers 413 or 400. - That copy is the only use of
POST /internal/audit/import(internal token only), and the route takes only what it is for: rows of servicecontrolwithcontrol.actions, an id startingcontrol:of at most 256 bytes (kept, so each row is copied once), and a time no later than a few minutes from now. Anything else refuses the whole batch with 400. - control-svc makes every old row acceptable before sending it, with the
route's own check: a row dated in the future (by control-svc's clock) is
copied with its time clamped to five minutes before the copy, its original
time kept in
parameters.original_ts, so it sorts at the time of the copy, not its own; an id too long or with a control character becomescontrol:sha256:<hex of the old key>; the backend and params are cut to the log's bounds. - A single row refused with 400 all the same stops the copy there, with an
ERROR line naming its legacy id, and the copy is retried with backoff (up
to every 5 minutes): it is never skipped. A single row over the body limit
(413) is left out with an ERROR line naming its legacy id and the copy
carries on: it stays in db=control
audit_log. - The NetworkPolicies let control-svc and auth-svc reach inventory-svc on
- auth-svc writes a row for every user and role change from this release
(
[inventory] urlinauth-svc.toml, defaulthttp://localhost:7402): until inventory-svc's audit log answers, and when the gateway cannot name the acting user, every IAM write (create, edit, password reset, end sessions, assign, unassign, delete of users and roles) answers 503 and changes nothing. Reading users and roles, and signing in, do not depend on it: an upgrade in any order never locks anyone out, it only holds IAM changes back until inventory-svc is migrated and running. - alerting-svc writes a row for every alert rule, notification channel and
mute change from this release, to the inventory-svc it already reads
(
[inventory] urlinalerting-svc.toml, unchanged; its NetworkPolicy already lets it reach 7402): until inventory-svc's audit log answers, and when the gateway cannot name the acting user, creating, editing and deleting rules, channels and mutes answer 503 and change nothing. Alerts are still evaluated, sent and listed; a channel's test send still works. New rules, channels and mutes get an id drawn by alerting-svc (20 characters of0-9a-z, the same form as before). - inventory-svc writes a row for every explorer panel and preset delete; those deletes answer 503 while its audit database cannot take the row (not yet migrated) or no user is named.
- A caller that reaches alerting-svc (:7404) or inventory-svc's explorer
delete routes directly, not through the gateway (a script or tool using the
internal token), sends no
x-fleeger-actor: its rule, channel and mute writes and its explorer deletes now answer 503 and change nothing. Send them through the gateway, as a signed-in user. - A location delete, an agent offboard, a registration token mint, a
checkpoint rollback and a rollback unlock are refused with 503 while
inventory-svc's audit log cannot take their
proposedrow (inventory-svc down, or not yet migrated), and when the gateway cannot name the acting user. Upgrade inventory-svc (db migrate) before, or together with, the gateway. - The offline command-line tools journal themselves from this release
: every service's
db unlock --force,inventory-svc rekey,alerting-svc rekey, andauth-svc user add|delete|assign-roleandmigrate-legacy. Each writes into the audit database with the database credentials it already has (no new setting) and refuses, changing nothing, while the audit database cannot take its row: run inventory-svc'sdb migrate(compose'sinventory-migrateone-shot, the chart's init container) before them. On a new install, create the first admin (auth-svc user add) once inventory-svc is up.db unlock --forcetakes--without-auditfor when the audit log cannot take the row (the audit database's own lock stuck before its first migration): it then releases the lock with a warning on stderr. Scripts that run these commands should treat the new refusal (exit 1) as "not done". auth-svc mint-credsandinventory-svc backup exportjournal themselves too (nats.creds.mint,backup.export), the same way: while the audit database cannot take theproposedrow they exit 1 and change nothing:mint-credsrecords no ledger row and prints no credential (delete the empty file a> <location>.credsredirect leaves behind),backup exportcreates no file. A scheduledbackup export(cron, a Kubernetes CronJob) therefore needs inventory-svc's audit database migrated, and fails, as it should be alerted on, while it is not.backup exportnow also refuses an output file that already exists before connecting to anything (it never overwrote one), and prints the size and SHA-256 of the archive it wrote.- A row written into an audit database that is not migrated is refused, also
through
POST /internal/auditandPOST /internal/audit/import, instead of creating a schemalessaudit_eventtable on the fly.