// Betrieb
NATS & JetStream capacity
Every metric and log an agent sends is first stored in a JetStream stream on the hub's NATS server, until ingestion-svc forwards it to VictoriaMetrics or Loki. How large those streams may grow, what happens when they are full, and how to watch and drain them decides how much disk NATS needs and how long an outage of ingestion, VictoriaMetrics or Loki the hub can ride out.
Stream limits
Messages are published to a
JetStream stream, read through a durable pull consumer, so a backlog built
while ingestion-svc is down or cannot forward survives its restart, within
bounds. On a hub whose streams ingestion-svc created, or that ran
service stream migrate (a hub that did neither still has unbounded
streams, see JetStream storage and the drain),
they are WorkQueue-retention, so a message leaves the stream as soon as ingestion acks
it, and an unacked backlog is capped per stream by age, size and count: by
default 3 days, 192 MiB and 1,000,000 messages (stream_max_age_secs,
stream_max_bytes, stream_max_msgs in [ingestion]; on a Helm install the
ingestion.streamMaxAgeSecs, ingestion.streamMaxBytes and
ingestion.streamMaxMsgs values; a BUMACS_INGESTION_* variable cannot
reach these keys, and ingestion-svc refuses one that names them). Past any of them the broker discards
the oldest messages ("Discarding at a stream's cap" in that section). If the
NATS account cannot afford the byte cap, ingestion-svc halves it until it fits
(not below 1 MiB; if even that does not fit, the stream is not created) and
logs a warning, so the cap in force can be smaller than
configured; fleeger_nats_stream_max_bytes shows it. A stream_max_age_secs
below a stream's duplicate window (2 minutes unless an operator changed it)
shortens the window to the new age, with a warning, since NATS refuses a window
longer than the age. Changed limits are not
applied to an existing stream that holds more than a new cap, so nothing
unforwarded is deleted (a warning says so): the stream keeps its old limits
until ingestion has drained it below and ingestion-svc restarts.
Storage budget, monitoring and draining
The NATS account's JetStream budget (--js-disk-storage, 2 GB as shipped by
deploy/nats/gen-auth.sh) is what every agent publish spends. When it is full
the broker refuses every publish (resource limits exceeded for account, error
code 10002) rather than discarding the oldest message, so all metric and log
ingestion stops and nothing heals it. The streams this service creates
are WorkQueue-retention and capped, so they release a message once it is
acked; a hub that never ran service stream migrate still has
Limits-retention streams that keep every forwarded message until the account
is full.
Every 30s ingestion-svc reads the account and both streams and writes gauges to
VictoriaMetrics directly (not through NATS, so they keep flowing while the
account is full), all labelled location="meta", source="fleeger",
service="ingestion-svc":
| Gauge | Labels | Meaning |
|---|---|---|
fleeger_nats_account_jetstream_enabled |
1 with every sample; 0, written alone, when the account read is answered "JetStream not enabled for account" (10039), and for no other failure (below) |
|
fleeger_nats_account_storage_used_bytes |
file storage the account uses | |
fleeger_ingestion_new_series_dropped_total |
pod, series_location |
points of new series of series_location dropped past its hourly bound since this process started (above); written for each location that dropped any, first as 0, so increase sees the first drop; pod is the replica's pod name, else a random id per process. Written by the sampler, so a sampler failure (nats_sampler_stale) also leaves it unwritten |
fleeger_nats_account_storage_reserved_bytes |
the file streams' positive max_bytes, each times its replicas, summed by ingestion-svc over every stream in the account (the broker's own figure is off by one per uncapped stream). On a single node, as shipped, this is the reservation the broker checks a new stream's cap against; on a tiered account it is not comparable to a tier's limit. Absent when the streams cannot be listed (storage_reserved_error in GET /jetstream says why) |
|
fleeger_nats_account_storage_limit_bytes |
the account's limit; absent when it has none | |
fleeger_nats_stream_info |
stream, retention |
always 1; retention="limits" marks a stream still to migrate |
fleeger_nats_stream_messages / _bytes |
stream |
what the stream holds |
fleeger_nats_stream_oldest_age_seconds |
stream |
age of its oldest message (0 when empty) |
fleeger_nats_stream_max_bytes / _max_messages / _max_age_seconds |
stream |
its caps; absent when unset |
fleeger_nats_consumer_pending_messages |
stream, consumer |
stored, not yet delivered to ingestion |
fleeger_nats_consumer_ack_pending_messages |
stream, consumer |
delivered, not yet acked |
fleeger_nats_stream_discarded_unforwarded_messages_total |
stream |
messages the broker deleted before ingestion forwarded them, since ingestion-svc started; a lower bound (below) |
fleeger_ingestion_log_batches_dropped_total |
pod, reason |
log batches this ingestion-svc process dropped since it started because Loki refused them for good (reason="loki_refused"; Observability & self-monitoring); pod is the replica's pod name, else a random id per process |
With Loki not configured, nothing forwards the logs stream, so before each
reading ingestion-svc deletes it (and with it its ingestion-logs consumer) if it
exists, and logs how many messages it dropped: a stream nobody reads would
fill to its caps and keep nats_stream_near_cap and nats_stream_discarding
firing for good. Those messages are not counted in the discarded counter below;
the log line is their only record. Nothing is written for a stream that does
not exist, so its gauges go stale and those alerts resolve. Only the stream
named logs is deleted, only while [loki] is unset, and only in the shape
ingestion-svc creates: subjects exactly logs.>, not parked by service stream migrate (anything else is left alone and logged once). It is checked on every
reading rather than once, because during a rolling update the outgoing
ingestion-svc, still configured with Loki, recreates it; and it is not deleted
while its consumer is active, so that pod loses nothing: active means a pull
request is open (num_waiting > 0; a running consumer always has one), or a
delivery or ack within the last 60s. A dead consumer's unacked messages
(num_ack_pending) do not count, or the stream would be kept forever. Agents' publishes to logs.<location>.> are
then answered "no stream" at once; an agent logs that once and drops its pod logs
until the stream is back. When ingestion-svc starts with [loki] set again, its
log consumer recreates the stream (bounded) and the consumer.
A sampling failure is logged (first, then at most every 10 minutes) and never
stops ingestion. A stream that cannot be read is skipped (and listed under
unavailable in the snapshot) without holding back the account gauges. The
built-in NATS storage almost full alert (nats_storage_high, see
Alerting) fires at 80% of the limit, and holds its last
reading while the gauges stop coming; a drain writes a sample at once. NATS
storage sampler silent (nats_sampler_stale) fires about 10–11 minutes after
ingestion-svc's last sample, since every NATS storage and stream alert is blind
until it writes again. One failure is told apart: when NATS has JetStream
on but not for the SPOKES account (its grant exceeds the server's
max_file_store), the sampler writes fleeger_nats_account_jetstream_enabled 0,
and NATS: JetStream disabled for the agents' account — ingestion halted
(nats_account_jetstream_disabled, critical) fires within about a minute and
a half, naming the cause and the fix.
Discarding at a stream's cap. The streams are capped (stream_max_bytes,
stream_max_msgs, stream_max_age_secs) with DiscardPolicy::Old, so while
ingestion cannot forward — ingestion-svc down, or VictoriaMetrics or Loki
failing — the broker makes room by deleting the oldest messages. The agents'
publishes still succeed, so the agents report nothing. Two built-in alerts cover
it (see Alerting): NATS stream near its cap
(nats_stream_near_cap, 90% of the nearest cap for 5 minutes) and NATS stream
discarding unforwarded data (nats_stream_discarding, any loss in 15 minutes).
The second one reads the counter above. Measured on nats-server 2.10, a discard
shows in exactly two places, and the counter adds them up:
- Never delivered. The broker does not move the consumer's delivered stream
sequence when it discards, so a message deleted before ingestion was handed it
leaves the stream's first sequence above
delivered + 1. Acks need a delivery first, so acking never opens this gap; a purge (the drain) movesdeliveredpast the purged messages, so a drain does not count either. The broker skips the gap at the consumer's next pull, so it is read on every sample (stream first, then consumer) and once by each consumer before it starts pulling, which is how an outage of ingestion-svc itself is counted when it comes back. - Delivered, never acked. A message ingestion holds unacked while its
downstream fails is already past
delivered, and when the broker deletes it, it advances the ack floor over it as if it were acked. Its one report is a$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<stream>.<consumer>advisory with reasonUnacknowledged message was deleted, which ingestion-svc subscribes to. With ingestion up and VictoriaMetrics or Loki down this is the only signal (the gap reads zero). It needs the subscribe grantdeploy/nats/gen-auth.shnow gives theingestionuser; with older creds the broker refuses the subscription, ingestion-svc logs a warning,GET /jetstreamreports"discard_advisories": "denied", and only never-delivered losses are counted.
Semantics: the counter counts since ingestion-svc started and restarts at 0 with
it (increase() reads that as a counter reset). Each stream's first sample after
a start is written as 0 whatever was already counted, and the count from the
next sample on: VictoriaMetrics' increase() takes a large first sample as the
series' baseline, which would hide the loss of an outage counted at startup. A
drain counts any discard below its bound, then marks the bound accounted for
before it purges, since the broker updates the stream before it moves the
consumer past the purged messages. Only the sampler and the consumers' check
feed the count; GET /jetstream only reads it, and a reading of a stream older
than one already seen (recreated by service stream migrate meanwhile) is
ignored. It is a lower bound: an
advisory sent while ingestion-svc is not connected is not seen, nor is a message
that was delivered-and-unacked when ingestion-svc stopped and was deleted before
it came back. nats-server 2.10.x also sends no advisory for a delivered, unacked
message when the consumer's ack-wait sweep runs between a cap discard and the
stream telling the consumer about it: the sweep drops the message from
the pending set silently. The window is short and the sweep runs once per ack
wait (30s by default), so this misses an occasional message, not a share of
them. A gap still open across a restart is counted again by the new
process, so a crash-looping ingestion-svc can over-count it. While a stream is
losing messages ingestion-svc logs a warning at once and then at most every 5
minutes (with the count since the last line), and once more when it stops; the
check before consuming logs the deleted sequence range. GET /jetstream
reports each stream's discarded_unforwarded split into never_delivered and
delivered_unacked.
The same snapshot is served as JSON by GET /jetstream (gateway:
GET /api/nats/jetstream, resource.view held globally) and shown in the
NATS storage panel on the Services page. From there an admin can drain
a stream: POST /jetstream/streams/{metrics|logs}/drain (gateway:
/api/nats/jetstream/streams/{stream}/drain, iam.manage) with
{"keep_secs": N, "dry_run": true|false} removes every message older than
now - keep_secs (0 to 30 days):
- Cutoff. A binary search over stream sequences reads message timestamps
with
$JS.API.STREAM.MSG.GET(next_by_subj: ">", so deleted sequences are skipped) to find the first message at or after the cutoff, then purges with that sequence as the bound: exactly the older messages go, and anything stored meanwhile has a higher sequence and is untouched. It creates no consumer, which aWorkQueuestream would refuse, and works the same onLimitsstreams. It assumes stored timestamps rise with the sequence, which they do (the broker stamps a message when it stores it). The cutoff is taken from ingestion-svc's clock and compared with the broker's stamps, so clock skew between the two shifts the cutoff by the skew; it never affects which messages count as unforwarded, which comes from the ack floor alone. - Stream identity. Right before the purge the stream's creation time is
read again; if the stream was deleted or recreated since it was planned
(
service stream migratedoes both, restarting the sequences), the drain is refused with409(STREAM_REPLACED) and nothing is deleted. - Forwarded or not. Only the ingestion consumer's ack floor proves a message
reached VictoriaMetrics or Loki. On a
WorkQueuestream every stored message is unforwarded (acked ones are gone); otherwise those above the ack floor are, and all of them when the consumer is missing. - Safety default. A drain that would delete unforwarded messages is answered
409(code: UNFORWARDED_MESSAGES) with the counts and deletes nothing, unless the request says"allow_unforwarded": truetogether with"max_unforwarded": N, the count the operator confirmed — which the UI sends only after a second, explicit confirmation naming it. If the fresh plan would delete more thanNunforwarded messages (the backlog grew since the preview), it is refused with409(UNFORWARDED_ABOVE_CONFIRMED) and deletes nothing. A drain over forwarded messages only goes ahead.jetstream::UNFORWARDED_POLICYin ingestion-svc switches this to refusing outright. - Preview.
dry_runreturns the same plan — message count (exact unless the stream has more than 50,000 interior gaps), a byte estimate from the average message size, oldest/newest timestamps, the unforwarded count — without deleting. A real drain plans afresh rather than trusting a preview. - Audit. Every drain is recorded in the hub-wide audit log
as
nats.stream.drainonstream:<name>, with its actor, client address and parameters (keep_secs,allow_unforwarded,max_unforwarded): aproposedrow before anything is planned or purged, thencompleted(what was removed, the cutoff, how many were not yet forwarded),rejected(a refusal, which deleted nothing) orfailed. Fail-closed: a drain whoseproposedrow cannot be written, or that names no acting user, is refused with 503 and deletes nothing. A preview is not audited. Drains are no longer written to the retention sweep log. - When the audit row cannot be written (inventory-svc down, its
SurrealDB disk full — often the very situation a full JetStream disk is
part of), every drain is refused with 503. If the stream must be cut now,
an operator can purge it by hand with the NATS CLI and the hub's
ingestion.creds(the one hub credential with the JetStream API), e.g. from nats-box on the hub's network with the NATS auth directory mounted at/s(as in Agent credentials & revocation):nats --server nats://nats:4222 --creds /s/ingestion.creds --inbox-prefix _INBOX._ingestion stream purge metrics --keep 100000 -f(--seq <n>to drop everything below a sequence instead).--inbox-prefix _INBOX._ingestionis required: that credential may only subscribe to_INBOX._ingestion.>(deploy/nats/gen-auth.sh), so with the CLI's default inbox every reply is a permissions violation and the command times out. Any othernatscommand run with it needs the same. This bypasses every check above — it deletes unforwarded messages too — and it is not in the audit log, so keep who did it, when and why in the incident notes.