Zum Inhalt springen
Produkte

// Betrieb

The audit log

The hub keeps one audit log of who did what: which user, from which address, did which action to what, with which parameters, and how it ended. It is inventory-svc's, in its own SurrealDB database audit (namespace fleeger), so that no service's own database, and no restore, can take it away.

What is recorded

Every audited action is journalled in two rows that share a correlation_id:

  • a proposed row, written before the action is done, and
  • a terminal row after it: completed, failed, or rejected (refused by a check, nothing done).

An action refused before anything was proposed (a wrong password for a backup download, a control action of an unknown kind) has a single rejected row.

Action (action) Service Target
nats.stream.drain: a JetStream drain ingestion stream:<name>
backup.download: an encrypted backup download gateway platform
backup.export: an encrypted backup written to a file (inventory-svc backup export) inventory platform
backup.restore: an offline restore inventory (inventory-svc backup restore), restore.sh platform
audit.prune: the audit log's own retention prune inventory audit_event
control.<action>: a control action (control.ping, control.k8s, ...) control the resource id
location.delete: a location delete (DELETE /api/locations/{id}) gateway location:<id>
agent.offboard: an agent offboard (DELETE /api/agents/{id}) gateway location:<id>
registration_token.create: a registration token minted (POST /api/agent-tokens) gateway location:<id>
nats.creds.mint: an agent's NATS credential minted by hand (auth-svc mint-creds) auth location:<id>
nats.hub_user.revoke: a hub user's earlier NATS JWTs revoked in the revocation ledger (auth-svc nats revoke-hub-user, run by gen-auth.sh fix-perms core-agent) auth nats_hub_user:<name>
location.assignments.remove: auth-svc's startup cleanup removing the role assignments of a location inventory-svc no longer has (actor system) auth location:<id>
location.rollback: a checkpoint rollback (POST /api/locations/{id}/rollback) inventory location:<id>
location.rollback.unlock: a rollback lease lifted by hand (POST /api/locations/{id}/rollback/unlock) inventory location:<id>
user.create: a user created (POST /api/users, auth-svc user add) auth user:<name>
user.update: a user's email, direct permissions or password changed (PUT /api/users/{name}) auth user:<name>
user.delete: a user deleted (DELETE /api/users/{name}, auth-svc user delete) auth user:<name>
user.sessions.end: every session of a user ended (POST /api/users/{name}/end-sessions) auth user:<name>
user.role.assign: a role assigned (POST /api/users/{name}/roles, auth-svc user assign-role) auth user:<name>
user.role.unassign: a role assignment removed (POST /api/users/{name}/unassign) auth user:<name>
role.create: a role created (POST /api/roles) auth role:<name>
role.update: a role's permissions replaced (PUT /api/roles/{name}) auth role:<name>
role.delete: a role and every assignment of it deleted (DELETE /api/roles/{name}) auth role:<name>
alert_rule.create: an alert rule created (POST /api/alert-rules) alerting alert_rule:<id>
alert_rule.update: an alert rule edited, switched on or off (PUT /api/alert-rules/{id}) alerting alert_rule:<id>
alert_rule.delete: an alert rule deleted, its firing alerts retired (DELETE /api/alert-rules/{id}) alerting alert_rule:<id>
notification_channel.create: a notification channel created (POST /api/notification-channels) alerting notification_channel:<id>
notification_channel.update: a channel's name, enabled or config changed (PUT /api/notification-channels/{id}) alerting notification_channel:<id>
notification_channel.delete: a channel deleted and detached from its rules (DELETE /api/notification-channels/{id}) alerting notification_channel:<id>
mute.create: a mute or maintenance window created (POST /api/mutes) alerting mute:<id>
mute.delete: a mute ended by hand (DELETE /api/mutes/{id}) alerting mute:<id>
explorer.panel.delete: a saved explorer panel deleted, and dropped from its presets (DELETE /api/explorer/panels/{id}) inventory explorer_panel:<id>
explorer.preset.delete: a saved explorer preset deleted (DELETE /api/explorer/presets/{id}) inventory explorer_preset:<id>
iam.import_legacy: users, roles and assignments imported from the old monolith (auth-svc migrate-legacy) auth legacy:<namespace>/<database>
db.unlock: a migration lock released (<service> db unlock --force) the service whose command ran: auth, inventory, alerting, ingestion, control database:<name> (inventory-svc's: database:inventory, database:audit)
secrets.rekey: stored secrets re-encrypted under the primary key (inventory-svc rekey, alerting-svc rekey) inventory, alerting table:credential, table:notification_channel

Each row of a location-scoped action has location set to the location's id.

Where each row is written

A row is written by the service that carries the action out:

  • An action spread over several services is written by the gateway, which runs it: a location delete (delete check, NATS revoke in auth-svc, state and metrics purge in ingestion-svc, alert clear in alerting-svc, the delete in inventory-svc, then the removal of its role assignments in auth-svc, whose step names the removed user:role pairs and never fails the delete) and an agent offboard (purge, revoke, alert clear). One proposed row is written before the first step and one terminal row after the last, whose result lists what each step did, in order, e.g. delete check: ok; revoke: ok; purge: failed (ingestion returned 500 ...); alert clear: not tried (the purge failed). A delete the delete check refuses (nothing done) ends rejected; one that stopped part way ends failed, and its steps say how far it got.

  • A registration token mint is written by the gateway too: it is the only way to auth-svc's mint, which is one database write. rejected is a request auth-svc refused (nothing minted). The token itself is never recorded. The row is written before auth-svc checks the location, so a location that is not one plain path segment of at most 200 bytes is not put in target and location: the row has target location:?, no location, and the location as sent (cut at 200 characters) in parameters.location.

  • A checkpoint rollback and a rollback unlock are written by inventory-svc, which does them, straight into its audit database. A rollback's parameters hold the checkpoint and the mode, its completed row what it changed; one refused with nothing changed (an unknown checkpoint, a location being rolled back) ends rejected, one whose outcome is unknown failed, saying so. An unlock's completed row names whose lease it lifted; rejected means none was held. The rollback's own changes are journalled in the location's config history as before.

  • A user or role change is written by auth-svc, which makes it. The proposed row's parameters say what changes, with what it was where the request does not say: a create holds the email, direct permissions and assignments asked for; user.update lists the fields set and holds each one's before and after (a new password only as password_changed: true); role.update holds the permissions before and after; a delete holds what was deleted (before: a user's email, direct permissions and assignments; a role's permissions and who held it); an assign or unassign the role and location (also in the row's location when the assignment is scoped to one that is one line of at most 256 bytes). before is read just before the change (null when there is no such user or role; the change is then refused). auth-svc puts no password, password hash, token or session id in a row's parameters; a terminal row's result (the outcome, or the error) is cleared of the request's password, as sent and as JSON or Rust Debug would quote it, and of any argon2 hash ($argon2…, through its salt and digest). A change refused by a check (a missing user or role, a built-in role, the last global admin, a stale role edit, an invalid field) ends rejected, nothing changed. A name that is not one line of at most 200 bytes is not put in target (user:?, role:?) but in parameters.name, cut at 200 characters. Parameters over the row's 16 KiB (a role held by thousands of users) are cut to their start, marked truncated, rather than refusing the change. Sign-in and /api/auth/me are not audited and never reach inventory-svc. The auth-svc command-line tools write the same rows; see The command-line tools.

  • An alert rule, notification channel or mute change is written by alerting-svc, which makes it. A create's id is drawn before its proposed row, so both rows name the new object; the parameters hold it as asked for. An edit's parameters list the fields it sets, each with its before (read just before the change, as auth-svc does) and after; a rule edit also its expected_updated_at. A delete's hold what was deleted (before: the rule; a channel's name, kind and the rules it is detached from, routed_from; the mute). A rule scoped to a location, a location mute and a resource mute scoped to a location also set the row's location (as for an assignment, only a value a row can hold). A change refused by a check (an invalid rule or channel, a built-in rule's delete, a missing object, a stale rule edit, STALE, or one that kept conflicting, CONFLICT) ends rejected, nothing changed; a rule delete or edit that stopped part way ends failed, saying how far it got (e.g. rule '<id>' was disabled but not deleted: retiring its firing alerts failed (…); retry the delete, or rule '<id>' was updated, but retiring its firing alerts failed: …). A delete of something already gone ends completed with no such …, nothing deleted, as the API answers 204; what a delete's completed row says it removed is what that request's own delete removed, so one that another request deleted meanwhile is not reported as deleted by this one.

    A channel's config is never recorded: it holds webhook URLs, tokens and SMTP passwords. A channel row names only the config's fields the channel's kind reads (config.fields; on an edit also config.changed, those whose value differs from the stored one, null when the stored config could not be decrypted), and counts any other key (config.unknown_fields) without naming it; never a value. A refused or failed channel change's row never quotes the error's text, which can quote a config value (a parser's invalid type: integer …): it says the channel or its config was refused (400) or that the error was on alerting-svc's side, and the answer to the request (or the alerting-svc log) has the reason. Only a channel's own not-found and conflict messages are kept. As a second guard every result is cleared of every string of the config, old and new, of four or more characters. Reads, a channel's test send and the alert clear of an offboarded location (in the gateway's agent.offboard / location.delete row) write no row.

  • An explorer panel or preset delete is written by inventory-svc, straight into its audit database. The proposed row's before holds the panel (title, query, mode, viz, unit, who created it, and in_presets, the presets it is dropped from) or the preset (name, panel ids, who created it); null, and the delete rejected, when there is none (or another request deleted it meanwhile). A panel deleted whose removal from its presets then failed ends failed, saying the panel was deleted. Creating and editing panels and presets is not audited: each row keeps its own created_by.

A name a row's target cannot hold (over 200 bytes, or not one line) goes in parameters.name with target <kind>:?, for every service that writes rows for its own API (auth-svc, alerting-svc, inventory-svc's explorer).

The gateway's calls for each step of a location delete or an agent offboard (auth-svc's revoke and assignment removal, ingestion-svc's purge, alerting-svc's alert clear) carry the acting user and their address (x-fleeger-actor, x-fleeger-client-ip), as every request the gateway proxies does, so each service knows who acted. The gateway removes both headers from every request a client sends to /api, public routes (login, bootstrap) included, so a client can never supply them; only the auth middleware stamps them again.

A terminal row's result can quote a service's own error text (a database error, an upstream's answer) as the service saw it, including internal detail the API answer leaves out. It is readable by whoever holds audit.view.

A request refused before the action is proposed (a malformed id, the meta location, a reserved pseudo-location; for a rollback or an unlock, a location id or checkpoint id that is not one line of at most 256 bytes, answered 400) writes no row: nothing was attempted. So does an IAM change whose current state (before) cannot be read (500).

A drain preview (dry_run) is not audited: it changes nothing. A drain is no longer written to the retention sweep log; the reapers' own sweeps still are.

Each row holds correlation_id, phase, actor, client_ip (the address the edge reported, as the gateway stamps it; empty when unknown), service, action, target, location (for a location-scoped action), parameters (JSON), result (on a terminal row) and ts. A row copied from control-svc's old log also has legacy_id.

The command-line tools

The offline command-line tools run with no inventory-svc to call, so, like the restore (inventory-svc backup restore, scripts/restore.sh), they write their rows straight into the audit database, with the database credentials the service already has: the same SurrealDB, namespace and credentials as the service's own database, database audit. No new setting. They write through the same code as inventory-svc's POST /internal/audit (fleeger_shared::audit::insert), so a row is checked (the same 400 reasons, the same cuts) and stored identically whichever way it comes. That write refuses a row while the audit database is not migrated (no audit_event table, or no append_only event on it), and never creates the table on the fly.

Command Row
<service> db unlock --force (every service with a database) db.unlock on database:<name>, one action per lock held; the parameters name who held the lock and since when (lock.instance_id, lock.locked_at) and whether --without-audit was given
inventory-svc rekey, alerting-svc rekey secrets.rekey on table:credential / table:notification_channel; parameters: the table, the field and how many retired keys the ring holds; completed with the counts; a run that stops on an error ends failed saying how far it got (failed: stopped after N … re-encrypted, M already current, …: <error>): what it re-encrypted stays re-encrypted
auth-svc user add user.create, as the API's: email, no direct permissions, the role as its one assignment
auth-svc user delete user.delete, before as the API's (null when there is no such user: completed, no such user, nothing deleted)
auth-svc user assign-role user.role.assign, the role, location null (global)
auth-svc migrate-legacy iam.import_legacy; parameters: the legacy namespace and database; completed with the counts and the names of the users and roles imported; one that fails part way ends failed naming what it had already written (failed: stopped after importing …; then <error>)
auth-svc mint-creds --location <id> nats.creds.mint on location:<id>, location set; parameters: the location, the minted user's public key (user_public_key, what the revocation ledger records and offboarding revokes), the signing account's public key (account_public_key) and expires: null (the credential does not expire). proposed before the ledger row (when it cannot be written, nothing is recorded or printed); once the ledger row is written, completed is attempted, best effort, and the credential is printed: a terminal row that cannot be written never withholds a credential that is already in the ledger (it is revocable), so such a mint shows a proposed row only. failed when the ledger refuses the row (nothing printed). A location the command refuses (not a valid location id, a reserved one, the hub's own) is refused before anything is connected, with no row
auth-svc nats revoke-hub-user --user <name> --pubkey <U…> --at <s> nats.hub_user.revoke on nats_hub_user:<name>; parameters: the hub user's name (hub_user), its public key (user_public_key) and the second at or before which its JWTs are refused (revoked_at). proposed before the ledger is written (when it cannot be, nothing is changed), completed after; a push to the resolver that fails afterwards is logged, not an error (the reconcile pushes the ledger). A name or key the command refuses is refused before anything is written, with no row. --check only reads the ledger and writes no row
inventory-svc backup export <file> [--metrics] backup.export on platform, as the Backup page's backup.download; parameters: destination: file, the file (absolute path), the archive's format (FLGRBK02), encrypted: true, the databases in it and include_metrics. proposed before the file is created; completed with the size and SHA-256 of the file as written (sha256sum gives the same) and the parts in it; one that fails part way ends failed saying how far it got (failed: stopped after N bytes, in part K of M (<part>); sealed whole before it: …; the unfinished file was removed: <error>; when the file could not be removed, the unfinished file could not be removed; delete <file> in its place). A file that already exists is refused before anything is connected, with no row
  • Actor: cli:<operating-system user> (SUDO_USER, else USER, else LOGNAME), as the restore's. A command has no signed-in Fleeger user: whoever can run it holds the hub's database credentials. client_ip is empty. service is the service whose command ran.

  • Fail-closed: the proposed row is written before the command changes anything; when it cannot be (SurrealDB unreachable, the audit database not migrated yet: run inventory-svc db migrate first; compose's inventory-migrate one-shot and the chart's init container do), the command exits non-zero saying why and changes nothing. The terminal row (completed, rejected for a refusal such as an unknown role, failed) is best effort, as for the API. A command killed by a signal before it ends leaves its proposed row without a terminal one.

  • No secrets: a password is never in a row (user add scrubs every result of it, and of any argon2 hash; migrate-legacy reads the legacy users before its proposed row and scrubs every result of each one's stored password as it is, argon2, bcrypt or plain text); a rekey row names no key, and its result is scrubbed of every key in the ring. mint-creds never puts the credential in a row (no JWT, no seed: only public keys), and every result is scrubbed of the creds file, its JWT and its seed; nor is the account's seed. A backup export row names no key (the archive is sealed with the credential key, the primary of the ring), and every result is scrubbed of every key in the ring. The rows go to the audit database and the command's messages to stderr: mint-creds prints only the creds file on stdout, as before, and backup export writes only to its file.

  • A backup export archive holds its own proposed row: the audit database is read into the archive after that row is written and before the terminal one, so the archive holds the export's proposed row and never its completed or failed row (the same goes for a Backup page download, backup.download). On a hub rebuilt from that archive (its audit log imported into an empty audit database), that export shows a proposed row only; its outcome is in the log of the hub it was taken from.

  • A command does not migrate the audit database. That is inventory-svc's db migrate (or a restore). In particular db unlock never does, so it never needs a lock it may be releasing.

  • db unlock --force --without-audit: db unlock --force is the recovery for a stuck migration lock, the audit database's own included (inventory-svc's db unlock releases both its own database's lock and the audit database's). Without the flag, a lock whose proposed row cannot be written is not released, nor is any other: rows already written for the others end rejected, and the command names the flag. With it, such a lock is released all the same, after a warning on stderr; afterwards one terminal row (completed / failed, no proposed row before it, parameters without_audit: true, result ending --without-audit: no proposed row was written before it) is tried once more, best effort, and when that fails too, stderr gets the whole row as its only record. A lock whose row can be written is journalled as usual, flag or not. Nothing held, or no --force: nothing is done and no row written, as before.

    The audit database takes a row whenever its audit_event table and append_only event exist, locked or not, so its own lock stuck after its first migration is released and journalled in it like any other. Only a lock stuck before 0001_audit committed (a first db migrate that died after taking the lock) leaves no table to write to: release it with inventory-svc db unlock --force --without-audit, then run inventory-svc db migrate.

Not audited: the read-only commands (user list, role list, db migrate, backup restore --dry-run), inventory-svc migrate-legacy (the one-time import of the old monolith's clusters, journalled in each location's config history) and ingestion-svc stream migrate (the one-time replacement of an unbounded metrics/logs JetStream stream with a bounded one, which drains the backlog first and deletes no un-forwarded message).

Fail-closed

An audited action is refused with 503 (UNAVAILABLE) and not done when:

  • its proposed row cannot be written (inventory-svc down, the audit database not migrated, a write that fails), or
  • no acting user is named: every audited route is reached through the gateway, which stamps the verified user on each mutating request, so a missing actor means the gateway could not identify them. The action is refused rather than recorded as system.

The terminal row is best effort: the action is done either way, and a row that cannot be written is logged by the service in full, as its only record.

A request that ends after its proposed row but before its outcome is recorded (the client went away, the service is shutting down, a panic) gets a failed row with the result interrupted: ...: the action may or may not have been done, so check the system before retrying it. The service also logs that row at ERROR, in full, before writing it: a service shutting down may not get to write it, and the log line is then its only record.

The restore CLI writes its proposed row after the pre-restore backup and before replacing any service database; when it cannot, nothing is replaced. scripts/restore.sh does the same. Its rows' parameters are the set (the S3 prefix or the --from-dir directory), the databases and the set's signature: verified, key <key id>, unsigned (--allow-unsigned), or unchecked (signed, no public key configured, --allow-unsigned).

Append-only, in the database

Every field of audit_event is READONLY, and the table's append_only event (DEFINE EVENT ... THROW) refuses any UPDATE or DELETE of a row, whoever sends it, the root user included. An UPDATE that changes nothing is let through (SurrealDB does not run the event for it); it changes nothing.

Only the retention prune deletes rows: in one transaction it removes the event, deletes a batch of rows older than the cutoff, and defines the event again, so no other transaction ever sees the table without it.

What the database cannot stop, so it is a matter of who holds the root credentials:

  • a root user removing the event, the table or the database (DDL), and
  • an OPTION IMPORT session, in which SurrealDB runs no events.

Retention

audit_retention_days in the retention policy (Settings, Retention, "Audit log"): rows older than this many days are deleted. The default is 400 days; 0 keeps the log forever; otherwise at least 30 days (and at most 100 years). A policy saved by an older page, without the field, keeps the stored value.

The inventory reaper prunes on each cycle while the reaper is enabled (reaper_enabled), so with the reaper disabled nothing is pruned. A prune with nothing old enough writes nothing. Otherwise it writes a proposed audit.prune row (actor system; its parameters hold the cutoff, the retention in days, and the version of the retention policy that set it and who saved that version, policy_version and policy_updated_by) before the first delete and a completed or failed row with the count after the last: the prune is in the log it prunes.

Rolling the retention policy back to a revision saved before the audit retention existed keeps the current audit_retention_days: a rollback never turns "forever" (0) into the default.

Restore

A restore never empties the audit database:

  • A .fbk archive and a scripts/backup.sh set hold the audit database like every other.
  • When the live audit database holds a row, the restore leaves it as it is and does not import the backup's.
  • When it holds none (a new hub, a rebuilt one), the backup's audit log is imported, first, and only through a transaction in which the database itself refuses to be emptied while it holds a row.

So a restore cannot roll the audit log back, and a rebuilt hub gets its history back.

Once the live log holds any row, an archive's history is not merged in. A hub that has started and written even one row (a login refusal, a prune, the restore's own row) keeps only its own log: restore onto an empty audit database (before the services first run) to keep an archive's history, or keep the archive to read it elsewhere.

After importing an archive's audit log, the restore defines its append_only event again right away (the archive's schema is not trusted), and scripts/restore.sh refuses before writing anything when the live audit database is not migrated and the set's is not imported. inventory-svc's db migrate and start define the event again too, whatever the migration stamps say, so no path leaves the table without it for long.

Reading it

GET /api/audit (permission audit.view, checked globally): newest first, every filter optional and exact.

Parameter
actor, action, target, location, service, correlation_id, phase exact match
since (inclusive), until (exclusive) RFC 3339
limit rows per page, default 100, at most 500
before the previous page's next

The answer is { "entries": [...], "next": "<cursor>" | null }. A page never holds more than 500 rows, however large the log is.

audit.view is granted to the built-in admin role through *; the operator and viewer roles do not have it (as with control.view and alert.view). Give it to a role on the Access page.

The Audit log page (/audit in the web UI) reads it: shown in the menu only to holders of audit.view (global), with the exact-match filters and the time range in the page's address (so a filtered view can be linked), pages read with before/next ("Load more"), and each row's parameters and correlation id, with a filter that shows one action's proposed and terminal rows together. A 503 is shown as the log being unavailable. Every value is shown as text, never as HTML. See reading the audit log.

The control panel's per-location history (GET /api/control/audit, control.view at that location) reads the control rows of this log.

Upgrading

  • inventory-svc's db migrate creates and migrates the audit database (services/inventory-svc/migrations-audit/) and defines the append_only event again; start refuses to run while it has pending migrations, as for its own database, and also defines the event again.
  • control-svc writes its rows here from this release ([inventory] url in control-svc.toml, default http://localhost:7402). Until inventory-svc answers, control actions answer 503. Its old audit_log table (db=control) is no longer written; its rows are copied into this log once, in the background, retrying until inventory-svc answers, and audit_copy:marker (db=control) notes how far it got. Copying again changes nothing. It goes in batches of at most 512 KiB, halved when inventory-svc answers 413 or 400.
  • That copy is the only use of POST /internal/audit/import (internal token only), and the route takes only what it is for: rows of service control with control. actions, an id starting control: of at most 256 bytes (kept, so each row is copied once), and a time no later than a few minutes from now. Anything else refuses the whole batch with 400.
  • control-svc makes every old row acceptable before sending it, with the route's own check: a row dated in the future (by control-svc's clock) is copied with its time clamped to five minutes before the copy, its original time kept in parameters.original_ts, so it sorts at the time of the copy, not its own; an id too long or with a control character becomes control:sha256:<hex of the old key>; the backend and params are cut to the log's bounds.
  • A single row refused with 400 all the same stops the copy there, with an ERROR line naming its legacy id, and the copy is retried with backoff (up to every 5 minutes): it is never skipped. A single row over the body limit (413) is left out with an ERROR line naming its legacy id and the copy carries on: it stays in db=control audit_log.
  • The NetworkPolicies let control-svc and auth-svc reach inventory-svc on
  • auth-svc writes a row for every user and role change from this release ([inventory] url in auth-svc.toml, default http://localhost:7402): until inventory-svc's audit log answers, and when the gateway cannot name the acting user, every IAM write (create, edit, password reset, end sessions, assign, unassign, delete of users and roles) answers 503 and changes nothing. Reading users and roles, and signing in, do not depend on it: an upgrade in any order never locks anyone out, it only holds IAM changes back until inventory-svc is migrated and running.
  • alerting-svc writes a row for every alert rule, notification channel and mute change from this release, to the inventory-svc it already reads ([inventory] url in alerting-svc.toml, unchanged; its NetworkPolicy already lets it reach 7402): until inventory-svc's audit log answers, and when the gateway cannot name the acting user, creating, editing and deleting rules, channels and mutes answer 503 and change nothing. Alerts are still evaluated, sent and listed; a channel's test send still works. New rules, channels and mutes get an id drawn by alerting-svc (20 characters of 0-9a-z, the same form as before).
  • inventory-svc writes a row for every explorer panel and preset delete; those deletes answer 503 while its audit database cannot take the row (not yet migrated) or no user is named.
  • A caller that reaches alerting-svc (:7404) or inventory-svc's explorer delete routes directly, not through the gateway (a script or tool using the internal token), sends no x-fleeger-actor: its rule, channel and mute writes and its explorer deletes now answer 503 and change nothing. Send them through the gateway, as a signed-in user.
  • A location delete, an agent offboard, a registration token mint, a checkpoint rollback and a rollback unlock are refused with 503 while inventory-svc's audit log cannot take their proposed row (inventory-svc down, or not yet migrated), and when the gateway cannot name the acting user. Upgrade inventory-svc (db migrate) before, or together with, the gateway.
  • The offline command-line tools journal themselves from this release : every service's db unlock --force, inventory-svc rekey, alerting-svc rekey, and auth-svc user add|delete|assign-role and migrate-legacy. Each writes into the audit database with the database credentials it already has (no new setting) and refuses, changing nothing, while the audit database cannot take its row: run inventory-svc's db migrate (compose's inventory-migrate one-shot, the chart's init container) before them. On a new install, create the first admin (auth-svc user add) once inventory-svc is up. db unlock --force takes --without-audit for when the audit log cannot take the row (the audit database's own lock stuck before its first migration): it then releases the lock with a warning on stderr. Scripts that run these commands should treat the new refusal (exit 1) as "not done".
  • auth-svc mint-creds and inventory-svc backup export journal themselves too (nats.creds.mint, backup.export), the same way: while the audit database cannot take the proposed row they exit 1 and change nothing: mint-creds records no ledger row and prints no credential (delete the empty file a > <location>.creds redirect leaves behind), backup export creates no file. A scheduled backup export (cron, a Kubernetes CronJob) therefore needs inventory-svc's audit database migrated, and fails, as it should be alerted on, while it is not. backup export now also refuses an output file that already exists before connecting to anything (it never overwrote one), and prints the size and SHA-256 of the archive it wrote.
  • A row written into an audit database that is not migrated is refused, also through POST /internal/audit and POST /internal/audit/import, instead of creating a schemaless audit_event table on the fly.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.