Zum Inhalt springen
Produkte

// Betrieb

Agent credentials & revocation

How a spoke-agent gets its NATS credentials, and how offboarding revokes them so the broker rejects a retired agent.

Trust chain (NATS decentralized auth)

Fleeger runs NATS in operator mode:

  • an operator ("fleeger") signs the account JWTs,
  • the SPOKES account holds every agent's user, and JetStream,
  • the SYS account is used for server administration (pushing account updates),
  • each agent gets a user JWT signed by the SPOKES account, scoped per location (see Transport & isolation).

auth-svc holds the SPOKES account seed (to mint user JWTs), the operator seed (to re-sign the account JWT for revocation), and SYS creds (to push updates).

All of this material is generated by deploy/nats/gen-auth.sh (gitignored output). Regenerating rotates everything — every agent re-onboards, with its old credentials removed first (re-onboarding).

Hub service credentials

Each hub service gets its own least-privilege user in the SPOKES account, with permissions defined once in gen-auth.sh's user_perms table:

user publish subscribe
ingestion $JS.API.>, $JS.ACK.> _INBOX._ingestion.>, $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.metrics.ingestion, $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.logs.ingestion-logs
gateway deny > (publishes nothing) metrics.>
alerting deny > (publishes nothing) metrics.>
inventory _INBOX.> (replies) inv.>
control commands.>, inv._iac.> _INBOX.>
core-agent metrics._core.>, logs._core.>, inv._core.>, _INBOX._core.> commands._core.>, _INBOX._core.>

An empty pub block in a NATS user JWT means UNRESTRICTED publish, not "publishes nothing" — the server's publish check short-circuits to allowed when there is no allow list and no deny list. A user created with only --allow-sub flags can publish to any subject in the account. Since CommandMessage carries no signature, broker publish permission is the entire authorization for control commands, so this is not a theoretical gap: such a credential can forge a command to any location's agent.

Every user in the table therefore carries either an explicit --allow-pub or an explicit --deny-pub ">". Never express "no publish" by omission.

control publishes to inv._iac.> for two ops today: inv._iac.resource_get, the inventory lookup behind the cross-location and resource-kind checks on every control action, and inv._iac.location_get, which tells it whether the location is local_collector, so that its commands go to the core agent's commands._core.<location>.<resource_id>. The subject is otherwise kept reserved (see Reserved pseudo-locations).

core-agent talks to every monitored target, so it is treated as potentially hostile even though it lives inside the hub. Its inventory publish is inv._core.>, not inv.>: it asks for a location on inv._core.<location>.<op> and lists its locations on inv._core.list_local. No agent is ever minted the reserved _core location, so the subject itself says the core agent sent it, and the broker enforces that. inv._core.> also leaves out control-svc's reserved inv._iac.>, which the former inv.> grant needed a --deny-pub inv._iac.> to carve back out. Its subscribe is _INBOX._core.>, not _INBOX.>. A global inbox subscribe would let a compromised core-agent passively collect every remote location's request replies, credential reveals included — the very thing the per-location inbox scoping in nats_creds.rs exists to prevent. It does not need a broad grant: a request-reply client only subscribes to the inbox it creates itself, and the agent sets custom_inbox_prefix to fleeger_metrics::inbox_prefix(CORE_LOCATION) = _INBOX._core.

Its metrics, logs and commands are scoped under _core the same way:

  • It publishes metrics on metrics._core.<location>.<source>.<resource_id> and logs on logs._core.<location>.… (fleeger_metrics::core_metrics_subject, core_logs_subject), granted metrics._core.> and logs._core.>, not metrics.> and logs.>. ingestion-svc records a _core message only for a local_collector location and maps it to that location, as it maps a location's own metrics.<location>.…; for any other location it drops and counts it. With metrics.> the core agent could write any remote location's series on that location's own subject, which ingestion-svc cannot tell from that location's agent. The JetStream metrics and logs streams take metrics.> and logs.>, which hold both shapes.
  • It subscribes to commands on commands._core.>, not commands.>. control-svc sends a local_collector location's commands on commands._core.<location>.<resource_id>, and every other location's on its own commands.<location>.<resource_id>, which the core agent can no longer read or answer.
  • It replies on _INBOX._core.>, not _INBOX.>. control-svc gives a _core command the reply subject _INBOX._core.<command_id> (fleeger_metrics::core_command_reply). With _INBOX.> the core agent could answer any request of any location's agent before the real responder: a forged inventory reply, or another location's command reply.

inventory-svc answers a request on inv._core.<location>.<op>, whatever the op (resources, reveal, discovery, hardware, collect_status), only while <location> is local_collector, and otherwise refuses it with a 403 before anything is read or written, and before the request takes a place in the location's line. So a command run inside the hub with the core-agent credential can neither read nor write a remote location's inventory. A hub that has not yet run ./gen-auth.sh fix-perms core-agent still hands the core agent the old inv.> grant, which reaches every location's own inv.<location>.<op>, a subject inventory-svc cannot tell from that location's agent; the core agent logs a warning at start-up while its credential still reaches it.

The core agent reads its own credential at start-up to choose its subjects: while it may still publish a location's own metrics.<location>.… it keeps those subjects, and while it may still subscribe to a location's own commands.<location>.… it subscribes to commands.>, which takes a command on either subject; it logs a warning while either holds. Once ./gen-auth.sh fix-perms core-agent has narrowed the credential and the core agent has restarted, it uses the _core subjects only. Run that only after every replica of ingestion-svc, control-svc, inventory-svc and alerting-svc runs a release with the _core subjects: an older ingestion-svc drops every _core message as forged, an older control-svc never sends a command to commands._core.…, control-svc learns which locations are local_collector from inventory-svc's location_get (an older inventory-svc replica answers it "unknown inventory op", and that command goes to the location's own subject, which the narrowed core agent no longer hears), and an older alerting-svc does not read metrics._core.<location> as a heartbeat of that location, so every local_collector location would look unreachable.

fix-perms core-agent (core-agent named) also revokes the core agent's earlier JWTs: a re-issued JWT does not make the old, wider one invalid (NATS user JWTs do not expire), so whoever holds the old core-agent.creds could still connect with metrics.>, commands.> and _INBOX.>. The revocation is kept in auth-svc's revocation ledger (migration 0018, nats_hub_revocation: the user key, and the second at or before which its JWTs are refused), so auth-svc re-asserts it on every push of the SPOKES account from then on, as it does an offboarded agent's: its reconcile after a push of the base SPOKES.jwt (the chart hook, compose's push-accounts.sh), a lost NATS volume, and the rebuild from the mounted base. The store's SPOKES.jwt is not touched, so a push-accounts.sh re-run after it stays a no-op. fix-perms reaches auth-svc through FLEEGER_AUTH_SVC, the command that runs its binary where it is deployed:

# compose
FLEEGER_AUTH_SVC="docker compose -f ../compose/compose.yaml exec -T auth-svc /usr/local/bin/service" \
  ./gen-auth.sh fix-perms core-agent
# Kubernetes
FLEEGER_AUTH_SVC="kubectl -n fleeger exec deploy/auth-svc -- /usr/local/bin/service" \
  ./gen-auth.sh fix-perms core-agent

In order, so that a failure changes nothing: it checks that auth-svc reaches the ledger and that the ledger defines the revocation table, migration 0018 (auth-svc nats revoke-hub-user --check); edits a copy of the store, with a config of its own naming the copy (nsc edits the store its config's absolute store_root names, whatever XDG_DATA_HOME says) and every nsc call naming the copy, and fails if the live store changed meanwhile; issues the new JWT there, in a later second than the one it replaces (it refuses a JWT dated more than 120 s ahead of its clock); has auth-svc record the revocation, dated one second before the new JWT's iat (auth-svc nats revoke-hub-user --user core-agent --pubkey <U…> --at <s>; --at is required and refused more than 60 s past auth-svc's clock; journalled in the audit log first as nats.hub_user.revoke, refused when it cannot be, and pushed to the resolver at once; a push that fails is left to the reconcile, within 10 s); checks the date the ledger answers (revoked-at <s>; a later date always wins there) is before the new JWT's, and stops otherwise, as the new JWT would be refused too; and only then moves the copy over the store (putting the old one back if that fails) and replaces the .creds. A failure before the ledger holds the revocation leaves the store, the creds and the ledger as they were. After it (the date check, an exec connection dropped after the write, or a failed move) the old credential is already refused, so the running core agent is disconnected: run the command again. A stop at the date check or a failed store move (the old store is put back) moves nothing into place; a .creds copy that fails after the store moved leaves the new store in place, the .creds of the users before it replaced, and names where the new one is.

The command's push and a push auth-svc makes at the same moment (an offboard, a reconcile pass) are not serialised with each other, as they run in different processes: a push that read the ledger just before the revocation was recorded can land after it and leave it out. auth-svc's next reconcile pass sees the resolver's copy lacks it and pushes it again, so the old JWT can be admitted for at most about 10 s more.

After restoring the auth database from a backup taken before fix-perms core-agent ran, the ledger no longer holds the core agent's revocation, and the next push leaves it out: the old, wide core-agent.creds connects again. Run ./gen-auth.sh fix-perms core-agent again after such a restore (Backup & disaster recovery). auth-svc cannot warn about it itself: it never sees the core agent's credential or the nsc store. Once the revocation reaches the resolver, the running core agent is disconnected: give it the new core-agent.creds (on Kubernetes, update it in the NATS auth Secret) and restart it. A plain fix-perms (no user named) re-issues core-agent without the revocation, and says that its earlier JWTs still connect.

A credential whose seed may have leaked needs a new nkey, not fix-perms: the seed proves the user key for any JWT of that key, the newly issued one included. Re-create the user with a new key (nsc delete user --account SPOKES --name core-agent, then ./gen-auth.sh fix-perms core-agent against a store where nsc add user --account SPOKES core-agent has re-added it), or rotate the whole trust chain with a full generate.

Rolling back a hub service to a release without the _core subjects after fix-perms core-agent has run breaks the core agent's locations (an older ingestion-svc drops its _core messages, an older control-svc sends its commands where it no longer listens). Widen the credential first: check out the gen-auth.sh of that release, run its ./gen-auth.sh fix-perms core-agent (which re-issues the wide JWT; the revocation in the ledger dates before it, so it is accepted), give the core agent the new creds and restart it, then roll back.

As a second barrier for secrets, inventory-svc answers a reveal only to an inbox that an agent collecting that location reads: the location's own _INBOX.<location>.>, or the core agent's _INBOX._core.> while the location is local_collector. A reveal sent with any other reply subject, a core-agent reveal of a location a spoke serves included, is refused, also on a hub whose core-agent credential is not narrowed yet.

ingestion gets the same treatment for the same reason: its subscribe is _INBOX._ingestion.>, not _INBOX.>. It is a JetStream client — stream and consumer management, pull-consumer delivery — all of which is request-reply, so it needs an inbox, but only the one it creates itself. ingestion-svc therefore sets custom_inbox_prefix to fleeger_metrics::inbox_prefix("_ingestion") = _INBOX._ingestion (services/ingestion-svc/src/nats.rs, used by both the metrics and the logs consumer), and the grant is scoped to exactly that subtree. Acks are unaffected — they go to the message's own $JS.ACK.… reply subject, covered by the separate --allow-pub. The prefix and the grant must change together; a unit test pins the two to each other so they cannot drift.

gateway and alerting are subscribe-only in the code (run_nats_tap in gateway-svc, heartbeat.rs in alerting-svc): no publish, no request, no JetStream. They therefore get --deny-pub ">" and, having no request-reply, no _INBOX.> subscribe either — that grant was dead permission, and on gateway (the only hub service reachable from the internet) it also meant the ability to observe every other service's request replies, including credential reveals.

Changing a hub user's permissions without re-onboarding agents

Never re-run the full ./gen-auth.sh to change permissions. It does rm -rf auth/ and a fresh nsc add operator, rotating the entire trust chain — every onboarded agent's credential dies.

cd deploy/nats
./gen-auth.sh fix-perms                    # re-assert the whole table
./gen-auth.sh fix-perms gateway alerting   # ...or just these users

fix-perms edits the existing nsc store in place: it strips every subject the table has ever granted (nsc edit user --rm, idempotent) and re-applies user_perms, then reissues only those users' .creds.

  • The operator key, the SPOKES account key and the account JWT are untouched, so ./push-accounts.sh is not needed.
  • Every agent credential minted by auth-svc is untouched. No site re-onboards.
  • Each edited user keeps its same NKEY — only the JWT inside the .creds changes. A user's permissions live inside its user JWT, so reissuing the file is unavoidable; the service's identity is not affected.

Then reload only the services whose creds changed:

docker compose up -d --force-recreate --no-deps gateway-svc alerting-svc   # compose
kubectl rollout restart deploy/gateway-svc deploy/alerting-svc             # k8s

Always make the change in user_perms itself, never by a one-off nsc command, so a future full regenerate keeps it.

When the nsc store is not at hand

fix-perms needs the nsc store the creds were generated from. A deployment whose only copy of the material is a Kubernetes Secret (or the Terraform variables that render it) can rebuild one with import. The files it needs are exactly the Secret's: resolver.conf, SPOKES.jwt, operator.nk, spokes-account.nk, every hub user's .creds (core-agent.creds only if the hub runs the bundled core agent), and optionally sys.creds.

# 1. dump the Secret into a private directory OUTSIDE this checkout
MATERIAL=$(mktemp -d)
kubectl -n fleeger get secret fleeger-nats-auth -o json \
  | jq -r '.data | to_entries[] | "\(.key) \(.value)"' \
  | while read -r k v; do printf '%s' "$v" | base64 -d > "$MATERIAL/$k"; done

# 2. rebuild a store from it, outside this checkout, and keep it
cd deploy/nats
GEN_AUTH_OUT=~/fleeger-prod-nats ./gen-auth.sh import "$MATERIAL"
rm -rf "$MATERIAL"

# 3. re-assert the table for the users that need it
GEN_AUTH_OUT=~/fleeger-prod-nats ./gen-auth.sh fix-perms gateway alerting

Relative paths for GEN_AUTH_OUT and the import directory are resolved from where you run the script. import refuses to overwrite an existing store and removes its partial output if it fails. A full generate ignores GEN_AUTH_OUT on purpose and refuses to run with it set.

import mints nothing: every key and user identity comes in as it is, and fix-perms then reissues only the users you name, with the same NKEYs. Put the reissued .creds back wherever the deployment reads them (the Secret, or the variable that renders it, e.g. a Terraform fleeger_nats_auth map), then restart the matching services as above.

Never run ./push-accounts.sh from an imported store. In a running hub auth-svc owns the live SPOKES account JWT: it re-signs it to add revocations, and pushing an imported copy that differs from the deployed base would silently drop them until auth-svc's next reconcile pass. (A copy identical to the deployed base is recognised and not pushed.)

The store holds the operator and SPOKES account seeds — the root of that deployment's trust chain. gen-auth.sh makes it readable only by uid 10001, the same as ./auth, so reading a reissued .creds out of it (or backing the store up) takes root or a container, e.g. docker run --rm -v ~/fleeger-prod-nats:/s:ro natsio/nats-box:0.20.1 cat /s/gateway.creds. Keep it somewhere private and backed up.

Narrowings that an existing hub still has to apply

A hub generated before these landed is still running the wider permissions — the JWT the service presents at connect is what the broker enforces, so nothing takes effect until the creds are reissued and the service reloads them.

change reissue restart
gateway/alerting: --deny-pub ">", no _INBOX.> subscribe ./gen-auth.sh fix-perms gateway alerting gateway-svc, alerting-svc
core-agent: inbox subscribe narrowed to _INBOX._core.> ./gen-auth.sh fix-perms core-agent the bundled core agent
core-agent: inventory publish narrowed from inv.> to inv._core.>. Roll out the new image first: an older core agent cannot work with the narrowed credential ./gen-auth.sh fix-perms core-agent the bundled core agent
core-agent: metrics/logs publish narrowed from metrics.>/logs.> to metrics._core.>/logs._core.>, command subscribe from commands.> to commands._core.>, reply publish from _INBOX.> to _INBOX._core.>, and its earlier JWTs revoked in auth-svc's revocation ledger. Ordered: first every replica of ingestion-svc, control-svc, inventory-svc, alerting-svc and auth-svc, and the core agent, on the new release (in any order), then fix-perms with FLEEGER_AUTH_SVC set (above), then restart the core agent. Never fix-perms before; to roll back, widen the credential first (above) ./gen-auth.sh fix-perms core-agent the bundled core agent
ingestion: inbox subscribe narrowed to _INBOX._ingestion.> ./gen-auth.sh fix-perms ingestion ingestion-svc

The ingestion user also gained a subscribe grant on its two consumers' MSG_TERMINATED advisories: the broker's only report of a message it discarded at a stream cap while ingestion held it unacked. The same ./gen-auth.sh fix-perms ingestion and restart apply it, in any order; until then ingestion-svc runs as before, logs that the subscription was refused, and counts only the messages discarded before they were delivered (NATS & JetStream capacity).

The ingestion one is ordered, because it is the one narrowing that also required a code change. The new creds allow subscribing only to _INBOX._ingestion.>, which an old ingestion-svc build cannot use — it asks async_nats for a random _INBOX.<nuid> inbox, so every JetStream request (stream/consumer setup, every pull for messages) would get no reply and ingestion would sit in its reconnect loop, silently ingesting nothing. A new build against old creds is fine, since _INBOX._ingestion.> sits inside the old _INBOX.> grant. So:

# 1. deploy the new ingestion-svc image first (it works under either cred)
docker compose pull ingestion-svc && docker compose up -d ingestion-svc
#    kubectl set image deploy/ingestion-svc ...                        (k8s)

# 2. then narrow the credential
cd deploy/nats && ./gen-auth.sh fix-perms ingestion

# 3. then restart so it loads the reissued .creds
docker compose up -d --force-recreate --no-deps ingestion-svc          # compose
kubectl rollout restart deploy/ingestion-svc                           # k8s

Agent credentials are untouched by all of this: no location re-onboards, and ./push-accounts.sh is not needed. Confirm with:

docker compose logs -f ingestion-svc | grep "ingestion consuming"

The account resolver

The server uses the nats-based account resolver (a dir of account JWTs), not a static config. Account JWTs are updated at runtime by pushing them to $SYS.REQ.CLAIMS.UPDATE from a SYS-account connection. This is what makes revocation possible: we can update the SPOKES account JWT while the server runs. deploy/nats/push-accounts.sh loads SPOKES into a fresh resolver.

The resolver keeps whatever was pushed last, so a push of the base SPOKES.jwt over auth-svc's re-signed copy would drop every revocation until auth-svc's next reconcile pass, and for as long as auth-svc is down. Every pusher (the chart's nats-push hook, the raw manifests' Job, push-accounts.sh) therefore runs deploy/nats/push-spokes.sh, which looks the account up first. auth-svc tags every JWT it pushes with fleeger-base:<sha256 of the base it was built on>; when the resolver's copy carries the tag of the very base the pusher would push, it pushes nothing. A base that changed (regenerated auth material, or an edited SPOKES.jwt) is still pushed, and auth-svc adds the revocations back on its next pass.

Transport & isolation

An agent's entire conversation with the hub — metrics and inventory (reading targets, revealing credentials, reporting discovery/hardware) — rides its one NATS connection. That connection is:

  • Encrypted + single-endpoint. NATS speaks WebSocket, so the agent connects wss://<hub>/nats through the hub's HTTPS edge. Only that one port is exposed; raw NATS (4222) and inventory-svc stay internal. TLS terminates at the edge.

  • Authenticated with no secret on the wire. The NATS user JWT + nkey use a challenge-response — the seed never leaves the agent.

  • Confined per location by the broker. The minted user JWT allows only:

    subject why
    publish metrics.<location>.> its own metrics
    publish inv.<location>.> inventory request-reply (list/reveal/discovery/hardware)
    subscribe _INBOX.<location>.> request replies + JS acks — a per-location inbox, so it can't observe another agent's replies (e.g. a credential reveal)
    subscribe commands.<location>.> reserved for hub→agent commands

    Because the broker enforces the publish scope, the <location> in an inventory request subject is guaranteed to be the agent's own. inventory-svc's responder additionally checks that any resource id in the payload belongs to that location before acting — so revealing a credential is doubly bounded to the requesting site. (The agent sets a matching custom_inbox_prefix, and inventory-svc answers a reveal only to that location's inbox, or to the core agent's for a local_collector location; see the core-agent note above.)

Net: inventory-svc is never exposed, and there is no unauthenticated credential-reveal endpoint — the reveal is authenticated, encrypted, per-location scoped, and revocable.

Onboarding

  1. An admin mints a one-time registration token for a location (/api/agent-tokens → auth-svc /internal/registration-tokens).
  2. The agent exchanges it (/api/bootstrap → /internal/bootstrap) for scoped NATS creds. auth-svc mints the credential first, then burns the token and records the minted user public key for the location in the nats_user table — this is what makes the credential revocable — in one transaction, so an exchange auth-svc could not record leaves the token unused. For 5 minutes after the burn the same token may be exchanged again (at most 4 times), for an agent that never received the answer: each retry mints a fresh credential and revokes the one before it, and the revocation is pushed at once.

Minting a credential by hand (mint-creds)

auth-svc's mint-creds subcommand mints a credential for a location without a registration token, and writes the creds file to stdout (logs go to stderr). On a hub, run it inside the auth-svc container, whose binary is /usr/local/bin/service and which has the account seed and the database settings:

# Docker Compose (from deploy/compose); -T keeps the output free of a TTY's \r
docker compose exec -T auth-svc /usr/local/bin/service \
  mint-creds --location <location-id> > <location-id>.creds

# Kubernetes (manifests or the Helm chart; add -n <namespace>)
kubectl exec deploy/auth-svc -- /usr/local/bin/service \
  mint-creds --location <location-id> > <location-id>.creds

From a source checkout, cargo run -p auth-svc -- mint-creds --location <location-id> does the same against the database its config names (see onboarding a location).

A minted credential does not expire, and offboarding revokes only the user keys the nats_user ledger holds, so no credential leaves auth-svc without its ledger row. mint-creds records the row through the same checked write the exchange's rows get, and only then prints the creds: when it cannot connect or sign in to the database, or the database refuses the row, it exits non-zero and prints nothing; delete the empty file the redirect left behind. If printing fails after the row is written, the row is left; it revokes a credential nobody holds, which is harmless.

Each mint is recorded in the audit log as nats.creds.mint on location:<id>, by cli:<operating-system user>: a proposed row before the ledger row, naming the location and the minted user's public key (the one the ledger holds), then completed (or failed when the ledger refuses the row). It is fail-closed: while the audit database cannot take the row (not migrated yet: run inventory-svc db migrate first), mint-creds exits non-zero saying why, records nothing and prints nothing. The completed row is best effort: once the ledger row is written the credential is printed even if that row cannot be, since it is already revocable. The credential itself (JWT, seed) is never in a row.

Recording a credential minted before rc.3

Before 1.0.0-rc.3, mint-creds printed the credential even when its ledger row was not written: a connect failure only logged a warning, and a refused write was silent. Offboarding cannot revoke such a credential. For each credential minted with mint-creds that is still in use, check its user key in the ledger and add the row if it is missing; offboarding the location then revokes it.

# The user public key: the `sub` claim of the JWT on line 2 of the creds file.
p=$(sed -n 2p <location-id>.creds | cut -d. -f2 | tr '_-' '/+')
while [ $(( ${#p} % 4 )) -ne 0 ]; do p="$p="; done
SUB=$(printf '%s' "$p" | base64 -d | grep -o '"sub":"U[A-Z0-9]*"' | cut -d'"' -f4)
echo "$SUB"   # U..., 56 characters

Then, as root on namespace fleeger, database auth (the ledger is closed to every other session):

# Docker Compose (from deploy/compose): the container has the root credentials
docker compose exec -T surrealdb /surreal sql --endpoint http://localhost:8000 \
  --ns fleeger --db auth --hide-welcome \
  <<< "SELECT location, revoked_at FROM nats_user WHERE user_pubkey = '$SUB';"

# Kubernetes: through a port-forward, with a local `surreal` CLI
kubectl port-forward svc/surrealdb 8000:8000 &
export SURREAL_PASS="$(kubectl get secret fleeger-db -o jsonpath='{.data.db-password}' | base64 -d)"
echo "SELECT location, revoked_at FROM nats_user WHERE user_pubkey = '$SUB';" | \
  surreal sql --endpoint http://localhost:8000 --user root --auth-level root \
    --ns fleeger --db auth --hide-welcome

An empty result means the row is missing. Add it with the same command, using this statement instead:

CREATE nats_user SET location = '<location-id>', user_pubkey = '<SUB>';

If the creds file is gone, the user key cannot be recovered; the only way to cut such a credential off is to rotate the SPOKES account key, which re-onboards every location.

Reserved pseudo-locations

_iac and _core (fleeger_metrics::{IAC_LOCATION, CORE_LOCATION}) are not locations. They exist so control-svc's inventory ops (inv._iac.<op>, today only resource_get), the core agent's cross-location list_local (inv._core.list_local) and its requests for a location (inv._core.<location>.<op>) have subjects that only a hub credential can publish to. The responder's only gate on those ops is the location token in the subject, so a per-location agent credential minted for _iac would reach control-svc's location-unscoped lookups across every location. _iac stays reserved even though the IaC feature it was named for has been removed, so a possible future IaC feature can reuse it without a location ever having claimed the name.

Both /internal/registration-tokens and CredMinter::mint therefore refuse them, and refuse any spelling that sanitizes onto them (.iac, >core, …) — subject_token maps ., , * and > to _, and it is the sanitized value that lands in the subject. The guard lives in mint() itself, the single chokepoint every credential passes through, so a new caller cannot bypass it.

The hub's own meta location (META_LOCATION) is refused the same way, by the same check: its agent cannot be offboarded, so a credential minted for it could never be revoked through the API, and the hub's self-monitoring needs no agent. Only the exact id meta is refused, as the offboard refusal matches it. A token for meta minted before the refusal exchanges for nothing (400); a meta credential already minted is left as it is.

The same chokepoint first refuses any location that is not a valid location id (1 to 50 ASCII letters, digits, _ and -, as validate_record_id checks), with the message /internal/registration-tokens answers. So mint-creds refuses what the API refuses, and a stored token for such an id exchanges for nothing (400).

Revoking an older meta credential

The API refuses to offboard meta, so revoke such a credential on auth-svc directly: POST /internal/nats/revoke/meta with the internal token in X-Fleeger-Internal. auth-svc listens on 7401, reachable only inside the hub, and the hub images have no curl.

Docker Compose (project fleeger, so the network is fleeger_default; run it from deploy/compose, where .env holds the token):

set -a; . ./.env; set +a
docker run --rm --network fleeger_default curlimages/curl -fsS -X POST \
  -H "X-Fleeger-Internal: $FLEEGER_INTERNAL_TOKEN" \
  http://auth-svc:7401/internal/nats/revoke/meta

Kubernetes (manifests or the Helm chart; add -n <namespace>):

TOKEN=$(kubectl exec deploy/auth-svc -- printenv FLEEGER_INTERNAL_TOKEN)
kubectl port-forward svc/auth-svc 7401:7401 &
curl -fsS -X POST -H "X-Fleeger-Internal: $TOKEN" \
  http://localhost:7401/internal/nats/revoke/meta
kill %1

A 204 means the credential is revoked and the broker now rejects it.

auth-svc cannot (yet) check that the location record exists — the location table lives in inventory-svc's database, not auth's. A token can still be minted for a location that was never created; it just cannot be a reserved one.

Revocation (offboarding)

NATS revokes a user via the account JWT's revocations map: user_pubkey -> unix time, and the server rejects that user's JWTs (issued at/before then) — including kicking a currently-connected agent.

When a location is offboarded (the Agents tab of the Services page or of the location, or deleting the location; the hub's own meta location is refused with 409, and a reserved pseudo-location, in any spelling the mint guard refuses, with 400 before any upstream call):

sequenceDiagram
    participant UI
    participant Gateway
    participant Auth as auth-svc
    participant NATS
    UI->>Gateway: DELETE /api/agents/{location}
    Gateway->>Auth: POST /internal/nats/revoke/{location}
    Auth->>Auth: mark location's nats_user keys revoked
    Auth->>NATS: look up the resolver's SPOKES account
    Auth->>Auth: add ALL revoked keys to that copy,<br/>re-sign with operator seed
    Auth->>NATS: $SYS.REQ.CLAIMS.UPDATE (sys.creds)
    NATS-->>Auth: {"code":200,"jwt updated"}
    Note over NATS: broker now rejects the revoked agent

Each push carries the full current revoked set, so revocation is idempotent. It is added to the resolver's own copy of the account, re-signed with the operator seed, so every other claim in that copy (connection limits, grants, revocations an operator added with nsc) is kept as whoever pushed it last left it. The copy is re-signed only once its signature verifies against the operator key, its issuer (iss) is that key and its subject (sub) is the SPOKES account: the lookup is answered by whichever SYS-account client replies first, so an unverified copy must never get the operator's signature. When the resolver holds no SPOKES account, or a copy that fails any of those checks (logged at WARN with the reason, once per copy), the JWT is rebuilt from the mounted base SPOKES.jwt. Operator signing keys are not supported for SPOKES: a copy signed by one counts as foreign, and the base is pushed over it once.

An expired SPOKES is refused, loudly. A copy the operator signed whose exp has passed (by auth-svc's clock: exp set and now past it) is not replaced from the base, which would revive an account the operator expired, and it is not pushed over either: nats-server (checked live against 2.10.29) answers 200 to such a push and stores it, but does not apply an expired account JWT to an account it has loaded (it logs account validation failed), so the offboarded agent would stay admitted while the offboard reported success. Instead auth-svc pushes nothing and reports:

the resolver's SPOKES account expired at <time>; re-issue it with nsc (no expiry) and push it; revocations are not applied until then
  • The offboard still marks the ledger and answers 204, as on any failed push, and logs that line at ERROR. A bootstrap retry's push logs it at ERROR too.
  • Every reconcile pass fails with it (WARN, then ERROR once it keeps failing, retried with the capped backoff) and never reports the resolver in sync while the copy stays expired, even when the stored copy already lists every revoked key.
  • Until SPOKES is re-issued, nothing auth-svc revokes is applied. If SPOKES expired while the server held it, no agent is admitted at all. If an already-expired SPOKES was pushed over a loaded account, the server keeps enforcing the claims it had, so an agent offboarded since is still admitted. Do not issue a time-limited SPOKES; if one expired, re-issue it without an expiry (nsc edit account SPOKES --expiry 0) and push it.

auth-svc and the server judge expiry by their own clocks, so a skew between them can refuse a copy a few seconds before or after the server expires it.

The check proves who signed a copy, not that it is current. A SYS-account client that answers the lookup before the server can replay an older genuine operator-signed SPOKES JWT; auth-svc then re-signs and pushes it, rolling back limits and grants and lifting hub-user revocations an operator added with nsc since. The ledger's revocations are still added, so no offboarded agent is re-admitted, but the reconcile can report the resolver in sync when it is not. A replayed copy carrying a past exp is refused like any expired copy: nothing is pushed, and the offboard, a bootstrap retry and every reconcile pass report the expiry at ERROR for as long as the replay answers the lookup, rather than pushing a copy the server stores without applying. That is possible only if a time-limited SPOKES was ever issued. Hold SYS-account credentials as tightly as the operator seed. Older releases rebuilt it from the base on every offboard and so undid such edits. Re-onboarding a location mints a fresh user (new key, not revoked); the old key stays revoked.

A push can still be lost: the resolver can be given an older JWT (a push of the base SPOKES.jwt), lose its store (a new nats-data volume), or miss a push while it is down. So auth-svc also reconciles: at start and every 10 seconds it looks up the resolver's SPOKES account and, when that lacks any key the nats_user table says is revoked, or a hub user revocation of the nats_hub_revocation table at its date, or the resolver holds no SPOKES account at all, pushes the revoked set as above. It logs each repair, for example nats revocation reconcile: resolver held no SPOKES account; pushed it with N revoked credential(s). An offboarded agent is therefore re-admitted by a lost push only until the next pass, and for as long as auth-svc is down.

The gateway offboard path calls this alongside ingestion's state/metric cleanup, so a single offboard both erases the metrics, logs and state and kills the credentials. The logs are deleted only when ingestion-svc has Loki configured and Loki accepts the delete; see what offboarding does.

The revoke also expires the location's unused registration tokens, so no agent can bootstrap a credential for it with a token issued before. It fails only when the ledger write fails, and then nothing is revoked. Each of its two writes is retried when it conflicts with a concurrent exchange of one of the location's tokens (a first exchange, or a retry), so the revoke still succeeds and revokes the credential that exchange recorded. Once the ledger is marked, a failed push is logged and the revoke still succeeds: the reconcile above pushes it. A location with nothing live to revoke pushes nothing, so it does not depend on NATS at all.

Deleting a location (DELETE /api/locations/{id}) calls the revoke before inventory-svc deletes the location, and refuses the delete when the revoke fails. Before the revoke, the gateway asks inventory-svc whether it would refuse the delete (GET /internal/locations/{id}/delete-check, which shares the delete's refusal and deletes nothing, and which the gateway does not forward); a refusal is answered as inventory-svc gave it and revokes nothing, and a check that fails or cannot be reached refuses the delete before anything is revoked. An inventory-svc without the check (404 or 405) is deleted from as before it. So every credential that was live, or could be minted from an unused token, when the location was deleted is revoked on record before it is gone. Not covered: an exchange already in flight at that moment (its token burned, its credential not yet recorded), and a registration token issued for the location id after it was deleted. Both steps are idempotent, so a retried delete completes whichever one failed.

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.