Zum Inhalt springen
Produkte

// Betrieb

NATS & JetStream capacity

Every metric and log an agent sends is first stored in a JetStream stream on the hub's NATS server, until ingestion-svc forwards it to VictoriaMetrics or Loki. How large those streams may grow, what happens when they are full, and how to watch and drain them decides how much disk NATS needs and how long an outage of ingestion, VictoriaMetrics or Loki the hub can ride out.

Stream limits

Messages are published to a JetStream stream, read through a durable pull consumer, so a backlog built while ingestion-svc is down or cannot forward survives its restart, within bounds. On a hub whose streams ingestion-svc created, or that ran service stream migrate (a hub that did neither still has unbounded streams, see JetStream storage and the drain), they are WorkQueue-retention, so a message leaves the stream as soon as ingestion acks it, and an unacked backlog is capped per stream by age, size and count: by default 3 days, 192 MiB and 1,000,000 messages (stream_max_age_secs, stream_max_bytes, stream_max_msgs in [ingestion]; on a Helm install the ingestion.streamMaxAgeSecs, ingestion.streamMaxBytes and ingestion.streamMaxMsgs values; a BUMACS_INGESTION_* variable cannot reach these keys, and ingestion-svc refuses one that names them). Past any of them the broker discards the oldest messages ("Discarding at a stream's cap" in that section). If the NATS account cannot afford the byte cap, ingestion-svc halves it until it fits (not below 1 MiB; if even that does not fit, the stream is not created) and logs a warning, so the cap in force can be smaller than configured; fleeger_nats_stream_max_bytes shows it. A stream_max_age_secs below a stream's duplicate window (2 minutes unless an operator changed it) shortens the window to the new age, with a warning, since NATS refuses a window longer than the age. Changed limits are not applied to an existing stream that holds more than a new cap, so nothing unforwarded is deleted (a warning says so): the stream keeps its old limits until ingestion has drained it below and ingestion-svc restarts.

Storage budget, monitoring and draining

The NATS account's JetStream budget (--js-disk-storage, 2 GB as shipped by deploy/nats/gen-auth.sh) is what every agent publish spends. When it is full the broker refuses every publish (resource limits exceeded for account, error code 10002) rather than discarding the oldest message, so all metric and log ingestion stops and nothing heals it. The streams this service creates are WorkQueue-retention and capped, so they release a message once it is acked; a hub that never ran service stream migrate still has Limits-retention streams that keep every forwarded message until the account is full.

Every 30s ingestion-svc reads the account and both streams and writes gauges to VictoriaMetrics directly (not through NATS, so they keep flowing while the account is full), all labelled location="meta", source="fleeger", service="ingestion-svc":

Gauge Labels Meaning
fleeger_nats_account_jetstream_enabled 1 with every sample; 0, written alone, when the account read is answered "JetStream not enabled for account" (10039), and for no other failure (below)
fleeger_nats_account_storage_used_bytes file storage the account uses
fleeger_ingestion_new_series_dropped_total pod, series_location points of new series of series_location dropped past its hourly bound since this process started (above); written for each location that dropped any, first as 0, so increase sees the first drop; pod is the replica's pod name, else a random id per process. Written by the sampler, so a sampler failure (nats_sampler_stale) also leaves it unwritten
fleeger_nats_account_storage_reserved_bytes the file streams' positive max_bytes, each times its replicas, summed by ingestion-svc over every stream in the account (the broker's own figure is off by one per uncapped stream). On a single node, as shipped, this is the reservation the broker checks a new stream's cap against; on a tiered account it is not comparable to a tier's limit. Absent when the streams cannot be listed (storage_reserved_error in GET /jetstream says why)
fleeger_nats_account_storage_limit_bytes the account's limit; absent when it has none
fleeger_nats_stream_info stream, retention always 1; retention="limits" marks a stream still to migrate
fleeger_nats_stream_messages / _bytes stream what the stream holds
fleeger_nats_stream_oldest_age_seconds stream age of its oldest message (0 when empty)
fleeger_nats_stream_max_bytes / _max_messages / _max_age_seconds stream its caps; absent when unset
fleeger_nats_consumer_pending_messages stream, consumer stored, not yet delivered to ingestion
fleeger_nats_consumer_ack_pending_messages stream, consumer delivered, not yet acked
fleeger_nats_stream_discarded_unforwarded_messages_total stream messages the broker deleted before ingestion forwarded them, since ingestion-svc started; a lower bound (below)
fleeger_ingestion_log_batches_dropped_total pod, reason log batches this ingestion-svc process dropped since it started because Loki refused them for good (reason="loki_refused"; Observability & self-monitoring); pod is the replica's pod name, else a random id per process

With Loki not configured, nothing forwards the logs stream, so before each reading ingestion-svc deletes it (and with it its ingestion-logs consumer) if it exists, and logs how many messages it dropped: a stream nobody reads would fill to its caps and keep nats_stream_near_cap and nats_stream_discarding firing for good. Those messages are not counted in the discarded counter below; the log line is their only record. Nothing is written for a stream that does not exist, so its gauges go stale and those alerts resolve. Only the stream named logs is deleted, only while [loki] is unset, and only in the shape ingestion-svc creates: subjects exactly logs.>, not parked by service stream migrate (anything else is left alone and logged once). It is checked on every reading rather than once, because during a rolling update the outgoing ingestion-svc, still configured with Loki, recreates it; and it is not deleted while its consumer is active, so that pod loses nothing: active means a pull request is open (num_waiting > 0; a running consumer always has one), or a delivery or ack within the last 60s. A dead consumer's unacked messages (num_ack_pending) do not count, or the stream would be kept forever. Agents' publishes to logs.<location>.> are then answered "no stream" at once; an agent logs that once and drops its pod logs until the stream is back. When ingestion-svc starts with [loki] set again, its log consumer recreates the stream (bounded) and the consumer.

A sampling failure is logged (first, then at most every 10 minutes) and never stops ingestion. A stream that cannot be read is skipped (and listed under unavailable in the snapshot) without holding back the account gauges. The built-in NATS storage almost full alert (nats_storage_high, see Alerting) fires at 80% of the limit, and holds its last reading while the gauges stop coming; a drain writes a sample at once. NATS storage sampler silent (nats_sampler_stale) fires about 10–11 minutes after ingestion-svc's last sample, since every NATS storage and stream alert is blind until it writes again. One failure is told apart: when NATS has JetStream on but not for the SPOKES account (its grant exceeds the server's max_file_store), the sampler writes fleeger_nats_account_jetstream_enabled 0, and NATS: JetStream disabled for the agents' account — ingestion halted (nats_account_jetstream_disabled, critical) fires within about a minute and a half, naming the cause and the fix.

Discarding at a stream's cap. The streams are capped (stream_max_bytes, stream_max_msgs, stream_max_age_secs) with DiscardPolicy::Old, so while ingestion cannot forward — ingestion-svc down, or VictoriaMetrics or Loki failing — the broker makes room by deleting the oldest messages. The agents' publishes still succeed, so the agents report nothing. Two built-in alerts cover it (see Alerting): NATS stream near its cap (nats_stream_near_cap, 90% of the nearest cap for 5 minutes) and NATS stream discarding unforwarded data (nats_stream_discarding, any loss in 15 minutes). The second one reads the counter above. Measured on nats-server 2.10, a discard shows in exactly two places, and the counter adds them up:

  • Never delivered. The broker does not move the consumer's delivered stream sequence when it discards, so a message deleted before ingestion was handed it leaves the stream's first sequence above delivered + 1. Acks need a delivery first, so acking never opens this gap; a purge (the drain) moves delivered past the purged messages, so a drain does not count either. The broker skips the gap at the consumer's next pull, so it is read on every sample (stream first, then consumer) and once by each consumer before it starts pulling, which is how an outage of ingestion-svc itself is counted when it comes back.
  • Delivered, never acked. A message ingestion holds unacked while its downstream fails is already past delivered, and when the broker deletes it, it advances the ack floor over it as if it were acked. Its one report is a $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<stream>.<consumer> advisory with reason Unacknowledged message was deleted, which ingestion-svc subscribes to. With ingestion up and VictoriaMetrics or Loki down this is the only signal (the gap reads zero). It needs the subscribe grant deploy/nats/gen-auth.sh now gives the ingestion user; with older creds the broker refuses the subscription, ingestion-svc logs a warning, GET /jetstream reports "discard_advisories": "denied", and only never-delivered losses are counted.

Semantics: the counter counts since ingestion-svc started and restarts at 0 with it (increase() reads that as a counter reset). Each stream's first sample after a start is written as 0 whatever was already counted, and the count from the next sample on: VictoriaMetrics' increase() takes a large first sample as the series' baseline, which would hide the loss of an outage counted at startup. A drain counts any discard below its bound, then marks the bound accounted for before it purges, since the broker updates the stream before it moves the consumer past the purged messages. Only the sampler and the consumers' check feed the count; GET /jetstream only reads it, and a reading of a stream older than one already seen (recreated by service stream migrate meanwhile) is ignored. It is a lower bound: an advisory sent while ingestion-svc is not connected is not seen, nor is a message that was delivered-and-unacked when ingestion-svc stopped and was deleted before it came back. nats-server 2.10.x also sends no advisory for a delivered, unacked message when the consumer's ack-wait sweep runs between a cap discard and the stream telling the consumer about it: the sweep drops the message from the pending set silently. The window is short and the sweep runs once per ack wait (30s by default), so this misses an occasional message, not a share of them. A gap still open across a restart is counted again by the new process, so a crash-looping ingestion-svc can over-count it. While a stream is losing messages ingestion-svc logs a warning at once and then at most every 5 minutes (with the count since the last line), and once more when it stops; the check before consuming logs the deleted sequence range. GET /jetstream reports each stream's discarded_unforwarded split into never_delivered and delivered_unacked.

The same snapshot is served as JSON by GET /jetstream (gateway: GET /api/nats/jetstream, resource.view held globally) and shown in the NATS storage panel on the Services page. From there an admin can drain a stream: POST /jetstream/streams/{metrics|logs}/drain (gateway: /api/nats/jetstream/streams/{stream}/drain, iam.manage) with {"keep_secs": N, "dry_run": true|false} removes every message older than now - keep_secs (0 to 30 days):

  • Cutoff. A binary search over stream sequences reads message timestamps with $JS.API.STREAM.MSG.GET (next_by_subj: ">", so deleted sequences are skipped) to find the first message at or after the cutoff, then purges with that sequence as the bound: exactly the older messages go, and anything stored meanwhile has a higher sequence and is untouched. It creates no consumer, which a WorkQueue stream would refuse, and works the same on Limits streams. It assumes stored timestamps rise with the sequence, which they do (the broker stamps a message when it stores it). The cutoff is taken from ingestion-svc's clock and compared with the broker's stamps, so clock skew between the two shifts the cutoff by the skew; it never affects which messages count as unforwarded, which comes from the ack floor alone.
  • Stream identity. Right before the purge the stream's creation time is read again; if the stream was deleted or recreated since it was planned (service stream migrate does both, restarting the sequences), the drain is refused with 409 (STREAM_REPLACED) and nothing is deleted.
  • Forwarded or not. Only the ingestion consumer's ack floor proves a message reached VictoriaMetrics or Loki. On a WorkQueue stream every stored message is unforwarded (acked ones are gone); otherwise those above the ack floor are, and all of them when the consumer is missing.
  • Safety default. A drain that would delete unforwarded messages is answered 409 (code: UNFORWARDED_MESSAGES) with the counts and deletes nothing, unless the request says "allow_unforwarded": true together with "max_unforwarded": N, the count the operator confirmed — which the UI sends only after a second, explicit confirmation naming it. If the fresh plan would delete more than N unforwarded messages (the backlog grew since the preview), it is refused with 409 (UNFORWARDED_ABOVE_CONFIRMED) and deletes nothing. A drain over forwarded messages only goes ahead. jetstream::UNFORWARDED_POLICY in ingestion-svc switches this to refusing outright.
  • Preview. dry_run returns the same plan — message count (exact unless the stream has more than 50,000 interior gaps), a byte estimate from the average message size, oldest/newest timestamps, the unforwarded count — without deleting. A real drain plans afresh rather than trusting a preview.
  • Audit. Every drain is recorded in the hub-wide audit log as nats.stream.drain on stream:<name>, with its actor, client address and parameters (keep_secs, allow_unforwarded, max_unforwarded): a proposed row before anything is planned or purged, then completed (what was removed, the cutoff, how many were not yet forwarded), rejected (a refusal, which deleted nothing) or failed. Fail-closed: a drain whose proposed row cannot be written, or that names no acting user, is refused with 503 and deletes nothing. A preview is not audited. Drains are no longer written to the retention sweep log.
  • When the audit row cannot be written (inventory-svc down, its SurrealDB disk full — often the very situation a full JetStream disk is part of), every drain is refused with 503. If the stream must be cut now, an operator can purge it by hand with the NATS CLI and the hub's ingestion.creds (the one hub credential with the JetStream API), e.g. from nats-box on the hub's network with the NATS auth directory mounted at /s (as in Agent credentials & revocation): nats --server nats://nats:4222 --creds /s/ingestion.creds --inbox-prefix _INBOX._ingestion stream purge metrics --keep 100000 -f (--seq <n> to drop everything below a sequence instead). --inbox-prefix _INBOX._ingestion is required: that credential may only subscribe to _INBOX._ingestion.> (deploy/nats/gen-auth.sh), so with the CLI's default inbox every reply is a permissions violation and the command times out. Any other nats command run with it needs the same. This bypasses every check above — it deletes unforwarded messages too — and it is not in the audit log, so keep who did it, when and why in the incident notes.

See also

// Zusammenarbeit

Bereit, Ihre digitale Infrastruktur zu bauen?

Ob individuelle Automatisierung für Ihr Unternehmen oder ein durchgeplantes Netzwerk für Ihr Zuhause — lassen Sie uns über das Projekt sprechen.