// Deployment
Agent modes: remote spoke vs. bundled core
A Fleeger agent collects a location's metrics and executes its control actions. It runs in one of two modes.
Remote spoke (the default)
One agent per location, running at the location (or wherever it can reach that
location's targets). It onboards with a one-time registration
token, which it exchanges for a
per-location NATS credential scoped to metrics.<location>.>,
inv.<location>.>, and commands.<location>.>. That credential is the security
boundary: the broker guarantees a remote spoke can only ever act on its own
location, even if compromised.
Use this for locations the hub can't reach directly (a separate site, a private network, a customer environment).
Bundled core agent (opt-in)
An agent that ships inside the hub stack, co-located with the services. It
serves every location flagged local_collector — locations whose targets are
reachable over the network from the core (e.g. an internet-reachable Kubernetes
API, a LAN Proxmox). One bundled agent can serve several such locations, so
you don't onboard a separate agent for each.
Because it runs inside the hub trust boundary it is a trusted hub component,
not an untrusted spoke, so it holds a hub-level core-agent credential
(pub metrics._core.>/logs._core.>/inv._core.>/_INBOX._core.>,
sub commands._core.>/_INBOX._core.>). No registration token — it's
pre-provisioned with the stack's NATS material. Everything it sends or hears
for a location is on its own _core subject, never on the location's: it asks
inventory-svc on inv._core.<location>.<op>, publishes metrics on
metrics._core.<location>.… and logs on logs._core.<location>.…, and serves
commands on commands._core.<location>.…. inventory-svc answers it,
ingestion-svc records it and control-svc sends it commands only for a
local_collector location. Until ./gen-auth.sh fix-perms core-agent narrows
an older credential it keeps the location's own metrics, logs and command
subjects (see Agent credentials & revocation).
The security model — the credential is the boundary, not the flag
Multi-location collection is authorized by the credential, never by the
FLEEGER_CORE_MODE config flag. A remote spoke that set the flag could still do
nothing, because:
- its per-location cred is broker-denied from publishing
metrics.<other>,inv.<other>, or reaching the reservedinv._core.list_localop (verified:Permissions Violation for Publish to "inv._core.list_local"); and - even on a subject its cred does allow, inventory-svc rejects the
list_localop unless it arrives on the reserved_coresubject (verified:{"ok":false,"error":"list_local is only valid on the core subject"}).
So only the hub-level core-agent credential can enumerate and serve the
local_collector locations.
Enabling it
- Provision the credential.
deploy/nats/gen-auth.shemitscore-agent.credsalongside the other hub creds. On Kubernetes it must be a key in thefleeger-nats-authSecret before you turn the core agent on — the Deployment projects that one key explicitly, so if it is absent kubelet refuses to create the container and the pod sits inContainerCreatingwith nothing in its logs. (helm install/upgradepreflights this and fails with the missing key name; see Deploying Fleeger on Kubernetes.) - Turn it on in the stack (off by default):
Docker Compose:
docker compose --profile core-agent up -dHelm (
fleeger-hub):--set coreAgent.enabled=trueon the install, or on a running hub:helm upgrade fleeger charts/fleeger-hub --namespace fleeger --reset-then-reuse-values \ --set coreAgent.enabled=trueWithout
--reset-then-reuse-valuesthe upgrade resets every other value you set at install (see the Helm chart reference's Upgrade section).Standalone: run the agent binary with
FLEEGER_CORE_MODE=true,FLEEGER_NATS_URL, andFLEEGER_NATS_CREDS=<core-agent.creds>.
- Point locations at it. On a location → Settings →
Collect from the core agent. The bundled agent picks it up on its next
cycle (each served location heartbeats, so it shows reachable), and control
actions for it route through the core agent too. Changing this setting
needs
location.editon all locations: a role scoped to the location cannot move it onto the core agent or off it.
It never probes the hub itself
The core agent runs inside the hub's network, where the hub's own services
(VictoriaMetrics, NATS monitoring, SurrealDB, the *-svc services) answer
whoever reaches them. So that a check added to a local_collector location
cannot reach them (an http_check could otherwise POST to VictoriaMetrics'
delete API, or read other locations' series through its body assertion), the
core agent refuses targets that are the hub's own:
- the HTTP, TCP and TLS checks: their host is resolved once and refused if
its name is a hub service's, or any address it resolves to is a hub
service's, loopback, unspecified or link-local. Otherwise the probe connects
to exactly the addresses judged, never to the name resolved again, so a DNS
answer that changes in between cannot reach a hub service; the HTTP probe
also bypasses any
HTTP(S)_PROXY, which would resolve the name itself. A refused check reportsprobe_success 0withreason="blocked_target", and the check's collect error says why. DNS checks connect to nothing and are not affected. - Proxmox and iDRAC registrations: their address is judged the
same way before each collection and before each Proxmox VM or iDRAC power
action, before their credential is revealed. A refused one is not
collected, with a collect error starting
refused:, and a refused action is answeredrefused: …; nothing was changed. Otherwise the collection and the action connect to exactly the addresses judged, bypass anyHTTP(S)_PROXY, and follow no redirect to another host (the Proxmox client follows none at all; the Redfish client only within the BMC's own scheme, host and port, so a redirect fromhttptohttpsis not followed either: store thehttpsaddress). The host judged is the one the client connects to, also for an address the URL parser reads unusually (http:/10.0.0.5names10.0.0.5). - Kubernetes registrations with a kubeconfig: the server of the
kubeconfig's current cluster is judged as a check's host before each
collection and before each Kubernetes control action. A refused one is not
collected, with a collect error starting
refused:that names the field, and a refused action is answeredrefused: …; nothing was changed. Otherwise the agent connects to exactly the server addresses judged, as the checks do, while the request still names the kubeconfig's host (for theHostheader and the certificate). A kubeconfig with aproxy-urlis not judged but refused, by every agent, with a collect error startingkubeconfig refused:: the agent has no proxy support, and its Kubernetes client ignoresHTTPS_PROXY/https_proxy.
The hub's services are FLEEGER_HUB_SERVICES (Compose, .env) or
coreAgent.hubServices (Helm): host names, IP addresses or CIDR ranges. Unset,
it is the service names the Compose stack and the chart give them, each
refused by name and by the addresses it resolves to (resolved at start,
before the first control command is served, and again every cycle); the
host of the agent's own FLEEGER_NATS_URL is always added, and so, when
the core agent runs in a pod, is the hub cluster's own Kubernetes API:
the names kubernetes, kubernetes.default, kubernetes.default.svc and
kubernetes.default.svc.cluster.local, and the KUBERNETES_SERVICE_HOST
address. A list you set replaces the defaults, not those. The chart
also adds the host of loki.url, writing the default names out with it
when coreAgent.hubServices is empty. Remote spokes are not affected: their
location's own network is what they probe.
On Kubernetes the names resolve to ClusterIPs, not to the pod IPs behind them,
so the core agent also refuses its own release's pods by their addresses. The chart and the raw manifests give it the label selector of the
release's pods, FLEEGER_HUB_POD_SELECTOR
(app.kubernetes.io/part-of=fleeger,app.kubernetes.io/instance=<release>),
and its namespace from the Downward API, FLEEGER_HUB_POD_NAMESPACE (unset,
the service account's own). It lists those pods with its service account when
it starts and on every refresh of the hub services' addresses, and refuses
every podIPs address, IPv4 and IPv6, of a pod that has not finished, except
a hostNetwork pod's: that address is its node's, and refusing it would
refuse checks at the node itself (none of the release's pods uses
hostNetwork). A refused check reports blocked_target, as for a ClusterIP.
A pod restarted at another address is refused there, and no longer at the old
one, from the next cycle on: within one poll interval
(coreAgent.pollIntervalSecs, 15 s by default) plus the time the cycle takes.
A listing gets 5 s, so an API server that hangs adds up to 5 s to each
refresh. The listing needs pods list in the namespace, which the chart's
and the raw manifests' ClusterRole grants. If a listing fails, the agent warns
once (and again only after a listing has succeeded in between) and keeps
refusing the addresses it last listed: dropping them would let a check at a
hub pod through for as long as the listing fails, while a kept address can at
worst refuse a check at a pod that has taken it since. Before the first
listing succeeds, only the names and the configured ranges are refused.
Without the selector (Compose, where a name resolves to the container itself,
or outside a pod), no pod is listed. A hub service outside the release (an
external Loki) is refused by its name and ClusterIP only; add its pod CIDR to
the list if it must be refused by pod IP.
One check applies to every agent, remote spokes included: the Redfish
paths the iDRAC collector and the power action request come from the BMC's
own answers (@odata.id), and one that is not an absolute path on the BMC's
own scheme, host and port is not requested. The hardware walk counts it as a
failed GET and reports no inventory that cycle; the sensor collection fails
with a collect error when the service root's Chassis link is such a path,
and skips such a chassis, Thermal or Power link; a power action fails
with the BMC named a system outside its own address. A path in an iDRAC
address is ignored, since Redfish always lives at /redfish/v1.
A Kubernetes registration with no credential is collected in-cluster, with
the core agent's ServiceAccount, in the hub's own cluster, so only a caller
holding location.edit on all locations may register one at a
local_collector location (see RBAC & per-location access). The core agent
collects in-cluster only a registration with no credential at all: one whose
credential has no kubeconfig, which a remote spoke reads as in-cluster, is
refused with a collect error. The API server's own addresses (a
control-plane node's IP, as opposed to the kubernetes service's ClusterIP)
are not refused unless you add them to the list.
A location is served by exactly one collector: flag it local_collector or
onboard a remote agent for it — not both (a second collector would double-report).