// Betrieb
Agent credentials & revocation
How a spoke-agent gets its NATS credentials, and how offboarding revokes them so the broker rejects a retired agent.
Trust chain (NATS decentralized auth)
Fleeger runs NATS in operator mode:
- an operator ("fleeger") signs the account JWTs,
- the SPOKES account holds every agent's user, and JetStream,
- the SYS account is used for server administration (pushing account updates),
- each agent gets a user JWT signed by the SPOKES account, scoped per location (see Transport & isolation).
auth-svc holds the SPOKES account seed (to mint user JWTs), the operator seed (to re-sign the account JWT for revocation), and SYS creds (to push updates).
All of this material is generated by deploy/nats/gen-auth.sh (gitignored
output). Regenerating rotates everything — every agent re-onboards, with its old
credentials removed first
(re-onboarding).
Hub service credentials
Each hub service gets its own least-privilege user in the SPOKES account, with
permissions defined once in gen-auth.sh's user_perms table:
| user | publish | subscribe |
|---|---|---|
ingestion |
$JS.API.>, $JS.ACK.> |
_INBOX._ingestion.>, $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.metrics.ingestion, $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.logs.ingestion-logs |
gateway |
deny > (publishes nothing) |
metrics.> |
alerting |
deny > (publishes nothing) |
metrics.> |
inventory |
_INBOX.> (replies) |
inv.> |
control |
commands.>, inv._iac.> |
_INBOX.> |
core-agent |
metrics._core.>, logs._core.>, inv._core.>, _INBOX._core.> |
commands._core.>, _INBOX._core.> |
An empty
pubblock in a NATS user JWT means UNRESTRICTED publish, not "publishes nothing" — the server's publish check short-circuits to allowed when there is no allow list and no deny list. A user created with only--allow-subflags can publish to any subject in the account. SinceCommandMessagecarries no signature, broker publish permission is the entire authorization for control commands, so this is not a theoretical gap: such a credential can forge a command to any location's agent.Every user in the table therefore carries either an explicit
--allow-pubor an explicit--deny-pub ">". Never express "no publish" by omission.
control publishes to inv._iac.> for two ops today: inv._iac.resource_get,
the inventory lookup behind the cross-location and resource-kind checks on
every control action, and inv._iac.location_get, which tells it whether the
location is local_collector, so that its commands go to the core agent's
commands._core.<location>.<resource_id>.
The subject is otherwise kept reserved (see
Reserved pseudo-locations).
core-agent talks to every monitored target, so it is treated as potentially
hostile even though it lives inside the hub. Its inventory publish is
inv._core.>, not inv.>: it asks for a location on
inv._core.<location>.<op> and lists its locations on inv._core.list_local.
No agent is ever minted the reserved _core location, so the subject itself
says the core agent sent it, and the broker enforces that. inv._core.> also
leaves out control-svc's reserved inv._iac.>, which the former inv.> grant
needed a --deny-pub inv._iac.> to carve back out. Its subscribe is
_INBOX._core.>, not _INBOX.>.
A global inbox subscribe would let a compromised core-agent passively collect every
remote location's request replies, credential reveals included — the very thing
the per-location inbox scoping in nats_creds.rs exists to prevent. It does not
need a broad grant: a request-reply client only subscribes to the inbox it
creates itself, and the agent sets custom_inbox_prefix to
fleeger_metrics::inbox_prefix(CORE_LOCATION) = _INBOX._core.
Its metrics, logs and commands are scoped under _core the same way:
- It publishes metrics on
metrics._core.<location>.<source>.<resource_id>and logs onlogs._core.<location>.…(fleeger_metrics::core_metrics_subject,core_logs_subject), grantedmetrics._core.>andlogs._core.>, notmetrics.>andlogs.>. ingestion-svc records a_coremessage only for alocal_collectorlocation and maps it to that location, as it maps a location's ownmetrics.<location>.…; for any other location it drops and counts it. Withmetrics.>the core agent could write any remote location's series on that location's own subject, which ingestion-svc cannot tell from that location's agent. The JetStreammetricsandlogsstreams takemetrics.>andlogs.>, which hold both shapes. - It subscribes to commands on
commands._core.>, notcommands.>. control-svc sends alocal_collectorlocation's commands oncommands._core.<location>.<resource_id>, and every other location's on its owncommands.<location>.<resource_id>, which the core agent can no longer read or answer. - It replies on
_INBOX._core.>, not_INBOX.>. control-svc gives a_corecommand the reply subject_INBOX._core.<command_id>(fleeger_metrics::core_command_reply). With_INBOX.>the core agent could answer any request of any location's agent before the real responder: a forged inventory reply, or another location's command reply.
inventory-svc answers a request on inv._core.<location>.<op>, whatever the
op (resources, reveal, discovery, hardware, collect_status), only
while <location> is local_collector, and otherwise refuses it with a 403
before anything is read or written, and before the request takes a
place in the location's line. So a command run inside the hub with
the core-agent credential can neither read nor write a remote location's
inventory. A hub that has not yet run ./gen-auth.sh fix-perms core-agent
still hands the core agent the old inv.> grant, which reaches every
location's own inv.<location>.<op>, a subject inventory-svc cannot tell from
that location's agent; the core agent logs a warning at start-up while its
credential still reaches it.
The core agent reads its own credential at start-up to choose its subjects: while it may still publish a location's own metrics.<location>.… it
keeps those subjects, and while it may still subscribe to a location's own
commands.<location>.… it subscribes to commands.>, which takes a command on
either subject; it logs a warning while either holds. Once
./gen-auth.sh fix-perms core-agent has narrowed the credential and the core
agent has restarted, it uses the _core subjects only. Run that only after
every replica of ingestion-svc, control-svc, inventory-svc and
alerting-svc runs a release with the _core subjects: an older ingestion-svc drops every
_core message as forged, an older control-svc never sends a command to
commands._core.…, control-svc learns which locations are local_collector
from inventory-svc's location_get (an older inventory-svc replica answers
it "unknown inventory op", and that command goes to the location's own
subject, which the narrowed core agent no longer hears), and an older
alerting-svc does not read metrics._core.<location> as a heartbeat of that
location, so every local_collector location would look unreachable.
fix-perms core-agent (core-agent named) also revokes the core agent's
earlier JWTs: a re-issued JWT does not make the old, wider one invalid (NATS
user JWTs do not expire), so whoever holds the old core-agent.creds could
still connect with metrics.>, commands.> and _INBOX.>. The revocation is
kept in auth-svc's revocation ledger (migration 0018,
nats_hub_revocation: the user key, and the second at or before which its JWTs
are refused), so auth-svc re-asserts it on every push of the SPOKES account
from then on, as it does an offboarded agent's: its reconcile after a push of
the base SPOKES.jwt (the chart hook, compose's push-accounts.sh), a lost
NATS volume, and the rebuild from the mounted base. The store's SPOKES.jwt is
not touched, so a push-accounts.sh re-run after it stays a no-op. fix-perms
reaches auth-svc through FLEEGER_AUTH_SVC, the command that runs its binary
where it is deployed:
# compose
FLEEGER_AUTH_SVC="docker compose -f ../compose/compose.yaml exec -T auth-svc /usr/local/bin/service" \
./gen-auth.sh fix-perms core-agent
# Kubernetes
FLEEGER_AUTH_SVC="kubectl -n fleeger exec deploy/auth-svc -- /usr/local/bin/service" \
./gen-auth.sh fix-perms core-agentIn order, so that a failure changes nothing: it checks that auth-svc reaches
the ledger and that the ledger defines the revocation table, migration 0018
(auth-svc nats revoke-hub-user --check); edits a copy of the store, with a
config of its own naming the copy (nsc edits the store its config's absolute
store_root names, whatever XDG_DATA_HOME says) and every nsc call naming
the copy, and fails if the live store changed meanwhile; issues the new JWT
there, in a later second than the one it replaces (it refuses a JWT dated more
than 120 s ahead of its clock); has auth-svc record the revocation, dated one
second before the new JWT's iat (auth-svc nats revoke-hub-user --user core-agent --pubkey <U…> --at <s>; --at is required and refused more than
60 s past auth-svc's clock; journalled in the audit log first as
nats.hub_user.revoke, refused when it cannot be, and pushed to the resolver
at once; a push that fails is left to the reconcile, within 10 s); checks the
date the ledger answers (revoked-at <s>; a later date always wins there) is
before the new JWT's, and stops otherwise, as the new JWT would be refused
too; and only then moves the copy over the store (putting the old one back if
that fails) and replaces the .creds. A failure before the ledger holds the
revocation leaves the store, the creds and the ledger as they were. After it
(the date check, an exec connection dropped after the write, or a failed move)
the old credential is already refused, so the running core agent is
disconnected: run the command again. A stop at the date check or a failed
store move (the old store is put back) moves nothing into place; a .creds
copy that fails after the store moved leaves the new store in place, the
.creds of the users before it replaced, and names where the new one is.
The command's push and a push auth-svc makes at the same moment (an offboard, a reconcile pass) are not serialised with each other, as they run in different processes: a push that read the ledger just before the revocation was recorded can land after it and leave it out. auth-svc's next reconcile pass sees the resolver's copy lacks it and pushes it again, so the old JWT can be admitted for at most about 10 s more.
After restoring the auth database from a backup taken before fix-perms core-agent ran, the ledger no longer holds the core agent's revocation, and
the next push leaves it out: the old, wide core-agent.creds connects again.
Run ./gen-auth.sh fix-perms core-agent again after such a restore
(Backup & disaster recovery). auth-svc cannot warn about it itself: it
never sees the core agent's credential or the nsc store. Once the revocation reaches
the resolver, the running core agent is disconnected: give it the new
core-agent.creds (on Kubernetes, update it in the NATS auth Secret) and
restart it. A plain fix-perms (no user named) re-issues core-agent without
the revocation, and says that its earlier JWTs still connect.
A credential whose seed may have leaked needs a new nkey, not fix-perms:
the seed proves the user key for any JWT of that key, the newly issued one
included. Re-create the user with a new key (nsc delete user --account SPOKES --name core-agent, then ./gen-auth.sh fix-perms core-agent against a
store where nsc add user --account SPOKES core-agent has re-added it), or
rotate the whole trust chain with a full generate.
Rolling back a hub service to a release without the _core subjects after fix-perms core-agent has run breaks the core agent's locations (an older ingestion-svc
drops its _core messages, an older control-svc sends its commands where it
no longer listens). Widen the credential first: check out the gen-auth.sh of
that release, run its ./gen-auth.sh fix-perms core-agent (which re-issues
the wide JWT; the revocation in the ledger dates before it, so it is
accepted), give
the core agent the new creds and restart it, then roll back.
As a second barrier for secrets, inventory-svc answers a reveal only to an
inbox that an agent collecting that location reads: the location's own
_INBOX.<location>.>, or the core agent's _INBOX._core.> while the location
is local_collector. A reveal sent with any other reply subject, a core-agent
reveal of a location a spoke serves included, is refused, also on a hub whose
core-agent credential is not narrowed yet.
ingestion gets the same treatment for the same reason: its subscribe is
_INBOX._ingestion.>, not _INBOX.>. It is a JetStream client — stream and
consumer management, pull-consumer delivery — all of which is request-reply, so
it needs an inbox, but only the one it creates itself. ingestion-svc therefore
sets custom_inbox_prefix to fleeger_metrics::inbox_prefix("_ingestion") =
_INBOX._ingestion (services/ingestion-svc/src/nats.rs, used by both the
metrics and the logs consumer), and the grant is scoped to exactly that subtree.
Acks are unaffected — they go to the message's own $JS.ACK.… reply subject,
covered by the separate --allow-pub. The prefix and the grant must change
together; a unit test pins the two to each other so they cannot drift.
gateway and alerting are subscribe-only in the code (run_nats_tap in
gateway-svc, heartbeat.rs in alerting-svc): no publish, no request, no
JetStream. They therefore get --deny-pub ">" and, having no request-reply,
no _INBOX.> subscribe either — that grant was dead permission, and on gateway
(the only hub service reachable from the internet) it also meant the ability to
observe every other service's request replies, including credential reveals.
Changing a hub user's permissions without re-onboarding agents
Never re-run the full ./gen-auth.sh to change permissions. It does
rm -rf auth/ and a fresh nsc add operator, rotating the entire trust chain —
every onboarded agent's credential dies.
cd deploy/nats
./gen-auth.sh fix-perms # re-assert the whole table
./gen-auth.sh fix-perms gateway alerting # ...or just these usersfix-perms edits the existing nsc store in place: it strips every subject the
table has ever granted (nsc edit user --rm, idempotent) and re-applies
user_perms, then reissues only those users' .creds.
- The operator key, the SPOKES account key and the account JWT are untouched,
so
./push-accounts.shis not needed. - Every agent credential minted by auth-svc is untouched. No site re-onboards.
- Each edited user keeps its same NKEY — only the JWT inside the
.credschanges. A user's permissions live inside its user JWT, so reissuing the file is unavoidable; the service's identity is not affected.
Then reload only the services whose creds changed:
docker compose up -d --force-recreate --no-deps gateway-svc alerting-svc # compose
kubectl rollout restart deploy/gateway-svc deploy/alerting-svc # k8sAlways make the change in user_perms itself, never by a one-off nsc command,
so a future full regenerate keeps it.
When the nsc store is not at hand
fix-perms needs the nsc store the creds were generated from. A deployment
whose only copy of the material is a Kubernetes Secret (or the Terraform
variables that render it) can rebuild one with import. The files it needs are
exactly the Secret's: resolver.conf, SPOKES.jwt, operator.nk,
spokes-account.nk, every hub user's .creds (core-agent.creds only if the
hub runs the bundled core agent), and optionally sys.creds.
# 1. dump the Secret into a private directory OUTSIDE this checkout
MATERIAL=$(mktemp -d)
kubectl -n fleeger get secret fleeger-nats-auth -o json \
| jq -r '.data | to_entries[] | "\(.key) \(.value)"' \
| while read -r k v; do printf '%s' "$v" | base64 -d > "$MATERIAL/$k"; done
# 2. rebuild a store from it, outside this checkout, and keep it
cd deploy/nats
GEN_AUTH_OUT=~/fleeger-prod-nats ./gen-auth.sh import "$MATERIAL"
rm -rf "$MATERIAL"
# 3. re-assert the table for the users that need it
GEN_AUTH_OUT=~/fleeger-prod-nats ./gen-auth.sh fix-perms gateway alertingRelative paths for GEN_AUTH_OUT and the import directory are resolved from
where you run the script. import refuses to overwrite an existing store and
removes its partial output if it fails. A full generate ignores GEN_AUTH_OUT
on purpose and refuses to run with it set.
import mints nothing: every key and user identity comes in as it is, and
fix-perms then reissues only the users you name, with the same NKEYs. Put the
reissued .creds back wherever the deployment reads them (the Secret, or the
variable that renders it, e.g. a Terraform fleeger_nats_auth map), then
restart the matching services as above.
Never run ./push-accounts.sh from an imported store. In a running hub
auth-svc owns the live SPOKES account JWT: it re-signs it to add revocations,
and pushing an imported copy that differs from the deployed base would
silently drop them until auth-svc's next reconcile pass. (A copy identical to
the deployed base is recognised and not pushed.)
The store holds the operator and SPOKES account seeds — the root of that
deployment's trust chain. gen-auth.sh makes it readable only by uid 10001,
the same as ./auth, so reading a reissued .creds out of it (or backing the
store up) takes root or a container, e.g. docker run --rm -v ~/fleeger-prod-nats:/s:ro natsio/nats-box:0.20.1 cat /s/gateway.creds. Keep it
somewhere private and backed up.
Narrowings that an existing hub still has to apply
A hub generated before these landed is still running the wider permissions — the JWT the service presents at connect is what the broker enforces, so nothing takes effect until the creds are reissued and the service reloads them.
| change | reissue | restart |
|---|---|---|
gateway/alerting: --deny-pub ">", no _INBOX.> subscribe |
./gen-auth.sh fix-perms gateway alerting |
gateway-svc, alerting-svc |
core-agent: inbox subscribe narrowed to _INBOX._core.> |
./gen-auth.sh fix-perms core-agent |
the bundled core agent |
core-agent: inventory publish narrowed from inv.> to inv._core.>. Roll out the new image first: an older core agent cannot work with the narrowed credential |
./gen-auth.sh fix-perms core-agent |
the bundled core agent |
core-agent: metrics/logs publish narrowed from metrics.>/logs.> to metrics._core.>/logs._core.>, command subscribe from commands.> to commands._core.>, reply publish from _INBOX.> to _INBOX._core.>, and its earlier JWTs revoked in auth-svc's revocation ledger. Ordered: first every replica of ingestion-svc, control-svc, inventory-svc, alerting-svc and auth-svc, and the core agent, on the new release (in any order), then fix-perms with FLEEGER_AUTH_SVC set (above), then restart the core agent. Never fix-perms before; to roll back, widen the credential first (above) |
./gen-auth.sh fix-perms core-agent |
the bundled core agent |
ingestion: inbox subscribe narrowed to _INBOX._ingestion.> |
./gen-auth.sh fix-perms ingestion |
ingestion-svc |
The ingestion user also gained a subscribe grant on its two consumers'
MSG_TERMINATED advisories: the broker's only report of a message it
discarded at a stream cap while ingestion held it unacked. The same
./gen-auth.sh fix-perms ingestion and restart apply it, in any order; until
then ingestion-svc runs as before, logs that the subscription was refused, and
counts only the messages discarded before they were delivered
(NATS & JetStream capacity).
The ingestion one is ordered, because it is the one narrowing that also
required a code change. The new creds allow subscribing only to
_INBOX._ingestion.>, which an old ingestion-svc build cannot use — it asks
async_nats for a random _INBOX.<nuid> inbox, so every JetStream request
(stream/consumer setup, every pull for messages) would get no reply and ingestion
would sit in its reconnect loop, silently ingesting nothing. A new build
against old creds is fine, since _INBOX._ingestion.> sits inside the old
_INBOX.> grant. So:
# 1. deploy the new ingestion-svc image first (it works under either cred)
docker compose pull ingestion-svc && docker compose up -d ingestion-svc
# kubectl set image deploy/ingestion-svc ... (k8s)
# 2. then narrow the credential
cd deploy/nats && ./gen-auth.sh fix-perms ingestion
# 3. then restart so it loads the reissued .creds
docker compose up -d --force-recreate --no-deps ingestion-svc # compose
kubectl rollout restart deploy/ingestion-svc # k8sAgent credentials are untouched by all of this: no location re-onboards, and
./push-accounts.sh is not needed. Confirm with:
docker compose logs -f ingestion-svc | grep "ingestion consuming"The account resolver
The server uses the nats-based account resolver (a dir of account JWTs),
not a static config. Account JWTs are updated at runtime by pushing them to
$SYS.REQ.CLAIMS.UPDATE from a SYS-account connection. This is what makes
revocation possible: we can update the SPOKES account JWT while the server runs.
deploy/nats/push-accounts.sh loads SPOKES into a fresh resolver.
The resolver keeps whatever was pushed last, so a push of the base
SPOKES.jwt over auth-svc's re-signed copy would drop every revocation until
auth-svc's next reconcile pass, and for as long as auth-svc is down.
Every pusher (the chart's nats-push hook, the raw manifests' Job,
push-accounts.sh) therefore runs deploy/nats/push-spokes.sh, which looks the
account up first. auth-svc tags every JWT it pushes with
fleeger-base:<sha256 of the base it was built on>; when the resolver's copy
carries the tag of the very base the pusher would push, it pushes nothing. A
base that changed (regenerated auth material, or an edited SPOKES.jwt) is
still pushed, and auth-svc adds the revocations back on its next pass.
Transport & isolation
An agent's entire conversation with the hub — metrics and inventory (reading targets, revealing credentials, reporting discovery/hardware) — rides its one NATS connection. That connection is:
Encrypted + single-endpoint. NATS speaks WebSocket, so the agent connects
wss://<hub>/natsthrough the hub's HTTPS edge. Only that one port is exposed; raw NATS (4222) and inventory-svc stay internal. TLS terminates at the edge.Authenticated with no secret on the wire. The NATS user JWT + nkey use a challenge-response — the seed never leaves the agent.
Confined per location by the broker. The minted user JWT allows only:
subject why publish metrics.<location>.>its own metrics publish inv.<location>.>inventory request-reply (list/reveal/discovery/hardware) subscribe _INBOX.<location>.>request replies + JS acks — a per-location inbox, so it can't observe another agent's replies (e.g. a credential reveal) subscribe commands.<location>.>reserved for hub→agent commands Because the broker enforces the publish scope, the
<location>in an inventory request subject is guaranteed to be the agent's own. inventory-svc's responder additionally checks that any resource id in the payload belongs to that location before acting — so revealing a credential is doubly bounded to the requesting site. (The agent sets a matchingcustom_inbox_prefix, and inventory-svc answers a reveal only to that location's inbox, or to the core agent's for alocal_collectorlocation; see the core-agent note above.)
Net: inventory-svc is never exposed, and there is no unauthenticated credential-reveal endpoint — the reveal is authenticated, encrypted, per-location scoped, and revocable.
Onboarding
- An admin mints a one-time registration token for a location
(
/api/agent-tokens→ auth-svc/internal/registration-tokens). - The agent exchanges it (
/api/bootstrap→/internal/bootstrap) for scoped NATS creds. auth-svc mints the credential first, then burns the token and records the minted user public key for the location in thenats_usertable — this is what makes the credential revocable — in one transaction, so an exchange auth-svc could not record leaves the token unused. For 5 minutes after the burn the same token may be exchanged again (at most 4 times), for an agent that never received the answer: each retry mints a fresh credential and revokes the one before it, and the revocation is pushed at once.
Minting a credential by hand (mint-creds)
auth-svc's mint-creds subcommand mints a credential for a location without a
registration token, and writes the creds file to stdout (logs go to stderr). On
a hub, run it inside the auth-svc container, whose binary is
/usr/local/bin/service and which has the account seed and the database
settings:
# Docker Compose (from deploy/compose); -T keeps the output free of a TTY's \r
docker compose exec -T auth-svc /usr/local/bin/service \
mint-creds --location <location-id> > <location-id>.creds
# Kubernetes (manifests or the Helm chart; add -n <namespace>)
kubectl exec deploy/auth-svc -- /usr/local/bin/service \
mint-creds --location <location-id> > <location-id>.credsFrom a source checkout, cargo run -p auth-svc -- mint-creds --location <location-id> does the same against the database its config names (see
onboarding a location).
A minted credential does not expire, and offboarding revokes only the user keys
the nats_user ledger holds, so no credential leaves auth-svc without its
ledger row. mint-creds records the row through the same checked write the
exchange's rows get, and only then prints the creds: when it cannot connect or
sign in to the database, or the database refuses the row, it exits non-zero
and prints nothing; delete the empty file the redirect left behind. If
printing fails after the row is written, the row is left; it revokes a
credential nobody holds, which is harmless.
Each mint is recorded in the audit log as nats.creds.mint on location:<id>, by cli:<operating-system user>: a proposed row before the ledger row, naming the location and the
minted user's public key (the one the ledger holds), then completed (or
failed when the ledger refuses the row). It is fail-closed: while the audit
database cannot take the row (not migrated yet: run inventory-svc db migrate first), mint-creds exits non-zero saying why, records nothing and
prints nothing. The completed row is best effort: once the ledger row is
written the credential is printed even if that row cannot be, since it is
already revocable. The credential itself (JWT, seed) is never in a row.
Recording a credential minted before rc.3
Before 1.0.0-rc.3, mint-creds printed the credential even when its ledger row
was not written: a connect failure only logged a warning, and a refused write
was silent. Offboarding cannot revoke such a credential. For each credential
minted with mint-creds that is still in use, check its user key in the ledger
and add the row if it is missing; offboarding the location then revokes it.
# The user public key: the `sub` claim of the JWT on line 2 of the creds file.
p=$(sed -n 2p <location-id>.creds | cut -d. -f2 | tr '_-' '/+')
while [ $(( ${#p} % 4 )) -ne 0 ]; do p="$p="; done
SUB=$(printf '%s' "$p" | base64 -d | grep -o '"sub":"U[A-Z0-9]*"' | cut -d'"' -f4)
echo "$SUB" # U..., 56 charactersThen, as root on namespace fleeger, database auth (the ledger is closed to
every other session):
# Docker Compose (from deploy/compose): the container has the root credentials
docker compose exec -T surrealdb /surreal sql --endpoint http://localhost:8000 \
--ns fleeger --db auth --hide-welcome \
<<< "SELECT location, revoked_at FROM nats_user WHERE user_pubkey = '$SUB';"
# Kubernetes: through a port-forward, with a local `surreal` CLI
kubectl port-forward svc/surrealdb 8000:8000 &
export SURREAL_PASS="$(kubectl get secret fleeger-db -o jsonpath='{.data.db-password}' | base64 -d)"
echo "SELECT location, revoked_at FROM nats_user WHERE user_pubkey = '$SUB';" | \
surreal sql --endpoint http://localhost:8000 --user root --auth-level root \
--ns fleeger --db auth --hide-welcomeAn empty result means the row is missing. Add it with the same command, using this statement instead:
CREATE nats_user SET location = '<location-id>', user_pubkey = '<SUB>';
If the creds file is gone, the user key cannot be recovered; the only way to cut such a credential off is to rotate the SPOKES account key, which re-onboards every location.
Reserved pseudo-locations
_iac and _core (fleeger_metrics::{IAC_LOCATION, CORE_LOCATION}) are not
locations. They exist so control-svc's inventory ops (inv._iac.<op>, today
only resource_get), the core agent's cross-location list_local
(inv._core.list_local) and its requests for a location
(inv._core.<location>.<op>) have subjects that only a hub credential can
publish to.
The responder's only gate on those ops is the location token in the subject, so
a per-location agent credential minted for _iac would reach control-svc's
location-unscoped lookups across every location. _iac stays reserved even
though the IaC feature it was named for has been removed, so a possible future
IaC feature can reuse it without a location ever having claimed the name.
Both /internal/registration-tokens and CredMinter::mint therefore refuse
them, and refuse any spelling that sanitizes onto them (.iac, >core, …) —
subject_token maps ., , * and > to _, and it is the sanitized
value that lands in the subject. The guard lives in mint() itself, the single
chokepoint every credential passes through, so a new caller cannot bypass it.
The hub's own meta location (META_LOCATION) is refused the same way, by the
same check: its agent cannot be offboarded, so a credential minted
for it could never be revoked through the API, and the hub's self-monitoring
needs no agent. Only the exact id meta is refused, as the offboard refusal
matches it. A token for meta minted before the refusal exchanges for nothing
(400); a meta credential already minted is left as it is.
The same chokepoint first refuses any location that is not a valid location id
(1 to 50 ASCII letters, digits, _ and -, as validate_record_id checks),
with the message /internal/registration-tokens answers. So
mint-creds refuses what the API refuses, and a stored token for such
an id exchanges for nothing (400).
Revoking an older meta credential
The API refuses to offboard meta, so revoke such a credential on
auth-svc directly: POST /internal/nats/revoke/meta with the internal token in
X-Fleeger-Internal. auth-svc listens on 7401, reachable only inside the hub,
and the hub images have no curl.
Docker Compose (project fleeger, so the network is fleeger_default; run it
from deploy/compose, where .env holds the token):
set -a; . ./.env; set +a
docker run --rm --network fleeger_default curlimages/curl -fsS -X POST \
-H "X-Fleeger-Internal: $FLEEGER_INTERNAL_TOKEN" \
http://auth-svc:7401/internal/nats/revoke/metaKubernetes (manifests or the Helm chart; add -n <namespace>):
TOKEN=$(kubectl exec deploy/auth-svc -- printenv FLEEGER_INTERNAL_TOKEN)
kubectl port-forward svc/auth-svc 7401:7401 &
curl -fsS -X POST -H "X-Fleeger-Internal: $TOKEN" \
http://localhost:7401/internal/nats/revoke/meta
kill %1A 204 means the credential is revoked and the broker now rejects it.
auth-svc cannot (yet) check that the location record exists — the
locationtable lives in inventory-svc's database, not auth's. A token can still be minted for a location that was never created; it just cannot be a reserved one.
Revocation (offboarding)
NATS revokes a user via the account JWT's revocations map: user_pubkey -> unix time, and the server rejects that user's JWTs (issued at/before then) —
including kicking a currently-connected agent.
When a location is offboarded (the Agents tab of the Services page or of the
location, or deleting the location; the hub's own meta location is refused
with 409, and a reserved pseudo-location, in any spelling the mint
guard refuses, with 400 before any upstream call):
sequenceDiagram
participant UI
participant Gateway
participant Auth as auth-svc
participant NATS
UI->>Gateway: DELETE /api/agents/{location}
Gateway->>Auth: POST /internal/nats/revoke/{location}
Auth->>Auth: mark location's nats_user keys revoked
Auth->>NATS: look up the resolver's SPOKES account
Auth->>Auth: add ALL revoked keys to that copy,<br/>re-sign with operator seed
Auth->>NATS: $SYS.REQ.CLAIMS.UPDATE (sys.creds)
NATS-->>Auth: {"code":200,"jwt updated"}
Note over NATS: broker now rejects the revoked agentEach push carries the full current revoked set, so revocation is
idempotent. It is added to the resolver's own copy of the account, re-signed
with the operator seed, so every other claim in that copy (connection limits,
grants, revocations an operator added with nsc) is kept as whoever pushed it
last left it. The copy is re-signed only once its signature verifies against
the operator key, its issuer (iss) is that key and its subject (sub) is the
SPOKES account: the lookup is answered by whichever SYS-account client replies
first, so an unverified copy must never get the operator's signature. When the
resolver holds no SPOKES account, or a copy that fails any of those checks
(logged at WARN with the reason, once per copy), the JWT is rebuilt from the
mounted base SPOKES.jwt. Operator signing keys are not supported for
SPOKES: a copy signed by one counts as foreign, and the base is pushed over it
once.
An expired SPOKES is refused, loudly. A copy the operator signed whose
exp has passed (by auth-svc's clock: exp set and now past it) is not
replaced from the base, which would revive an account the operator expired,
and it is not pushed over either: nats-server (checked live against 2.10.29)
answers 200 to such a push and stores it, but does not apply an expired
account JWT to an account it has loaded (it logs account validation failed), so the offboarded agent would stay admitted while the offboard
reported success. Instead auth-svc pushes nothing and reports:
the resolver's SPOKES account expired at <time>; re-issue it with nsc (no expiry) and push it; revocations are not applied until then
- The offboard still marks the ledger and answers 204, as on any failed push, and logs that line at ERROR. A bootstrap retry's push logs it at ERROR too.
- Every reconcile pass fails with it (WARN, then ERROR once it keeps failing, retried with the capped backoff) and never reports the resolver in sync while the copy stays expired, even when the stored copy already lists every revoked key.
- Until SPOKES is re-issued, nothing auth-svc revokes is applied. If SPOKES
expired while the server held it, no agent is admitted at all. If an
already-expired SPOKES was pushed over a loaded account, the server keeps
enforcing the claims it had, so an agent offboarded since is still
admitted. Do not issue a time-limited SPOKES; if one expired, re-issue it
without an expiry (
nsc edit account SPOKES --expiry 0) and push it.
auth-svc and the server judge expiry by their own clocks, so a skew between them can refuse a copy a few seconds before or after the server expires it.
The check proves who signed a copy, not that it is current. A SYS-account
client that answers the lookup before the server can replay an older genuine
operator-signed SPOKES JWT; auth-svc then re-signs and pushes it, rolling back
limits and grants and lifting hub-user revocations an operator added with
nsc since. The ledger's revocations are still added, so no offboarded agent
is re-admitted, but the reconcile can report the resolver in sync when it is
not. A replayed copy carrying a past exp is refused like any expired copy:
nothing is pushed, and the offboard, a bootstrap retry and every reconcile pass
report the expiry at ERROR for as long as the replay answers the lookup, rather
than pushing a copy the server stores without applying. That is possible only
if a time-limited SPOKES was ever issued.
Hold SYS-account credentials as tightly as the operator seed.
Older releases rebuilt it from the base on every offboard and so undid such edits.
Re-onboarding a location mints a fresh user (new key, not revoked); the old key
stays revoked.
A push can still be lost: the resolver can be given an older JWT (a push of
the base SPOKES.jwt), lose its store (a new nats-data volume), or miss a
push while it is down. So auth-svc also reconciles: at start and every 10
seconds it looks up the resolver's SPOKES account and, when that lacks any key
the nats_user table says is revoked, or a hub user revocation of the
nats_hub_revocation table at its date, or the resolver holds no SPOKES
account at all, pushes the revoked set as above. It logs each repair, for
example nats revocation reconcile: resolver held no SPOKES account; pushed it with N revoked credential(s). An offboarded agent is therefore re-admitted by
a lost push only until the next pass, and for as long as auth-svc is down.
The gateway offboard path calls this alongside ingestion's state/metric cleanup, so a single offboard both erases the metrics, logs and state and kills the credentials. The logs are deleted only when ingestion-svc has Loki configured and Loki accepts the delete; see what offboarding does.
The revoke also expires the location's unused registration tokens, so no agent can bootstrap a credential for it with a token issued before. It fails only when the ledger write fails, and then nothing is revoked. Each of its two writes is retried when it conflicts with a concurrent exchange of one of the location's tokens (a first exchange, or a retry), so the revoke still succeeds and revokes the credential that exchange recorded. Once the ledger is marked, a failed push is logged and the revoke still succeeds: the reconcile above pushes it. A location with nothing live to revoke pushes nothing, so it does not depend on NATS at all.
Deleting a location (DELETE /api/locations/{id}) calls the revoke before
inventory-svc deletes the location, and refuses the delete when the revoke
fails. Before the revoke, the gateway asks inventory-svc whether it would
refuse the delete (GET /internal/locations/{id}/delete-check, which shares
the delete's refusal and deletes nothing, and which the gateway does not
forward); a refusal is answered as inventory-svc gave it and revokes nothing,
and a check that fails or cannot be reached refuses the delete before anything
is revoked. An inventory-svc without the check (404 or 405) is
deleted from as before it. So every credential that was live, or could be
minted from an unused token, when the location was deleted is revoked on
record before it is gone. Not covered: an exchange already in flight
at that moment (its token burned, its credential not yet recorded), and a
registration token issued for the location id after it was deleted. Both steps are idempotent, so a retried
delete completes whichever one failed.