Monitoring & Observability Architecture

Status: active Last verified: 2026-07-20 (against the current monitoring coverage inventory, accepted watcher decisions, consolidated metrics deployment, syslog service record, and Ops Console source documentation) Scope: collection layers, freshness, alert transport, incidents, and authority boundaries

The monitoring stack combines several narrow systems rather than forcing every signal into one tool. Infrastructure polling, time-series metrics, centralized logs, active watchdogs, and durable incidents answer different questions.

The objective is actionable evidence with quiet escalation. Monitoring observes and records; operators decide and act through each system’s authoritative tooling.

This page omits private hostnames, addresses, endpoints, notification routes, channel names, credentials, and deployment paths.

Architecture at a glance

infrastructure ──polling──→ LibreNMS
services ───────metrics───→ Prometheus ──→ dashboards and rules
hosts/devices ───logs─────→ central syslog collectors
critical jobs ─watchdogs──→ Hermes ──────→ external escalation
checks/producers ─────────→ Ops Console ─→ events and incidents

These are overlapping evidence paths, not one universal event bus. A service may expose metrics, write logs, answer an active check, and emit a state-change event. The overlap is intentional: it allows one path to reveal that another has failed.

Infrastructure polling

LibreNMS provides broad reachability and read-only infrastructure polling. It covers network devices, virtualization hosts, core service guests, and wireless infrastructure. Power events use a separate dedicated integration.

Typical observations include:

• ICMP reachability
• SNMP device identity and uptime
• interface state and counters
• hardware sensors and resource usage
• wireless controller and access-point state

Polling is source-restricted and read-only. LibreNMS describes observed device health; it does not define intended topology or make configuration changes.

The current design does not assume LibreNMS can report its own failure. An independent off-host watcher checks the real application surface, validates a body marker rather than accepting any successful web response, and requires consecutive misses before escalating. Recovery is a separate signal.

This watcher-in-watcher pattern detects application, container, host, and path failures that the monitored system cannot reliably announce about itself.

Service metrics

Prometheus-compatible exporters expose application and platform behavior such as:

• request rate, error rate, and latency
• queue depth and saturation
• model-serving throughput and resource pressure
• network-device sensors and link state
• traffic-shaping state and freshness
• worker, retrieval, and agent-service health

A central Prometheus view combines network and application metrics, including selected series federated from a specialized scraper. Grafana consumes that consolidated view for dashboards.

Metrics are evidence, not durable incidents by themselves. Alert rules can identify threshold violations, but a rule is not an externally delivered alert until a notification path is configured and verified. The architecture records that distinction instead of treating a visible rule as proven paging coverage.

Centralized logs

Two rsyslog collectors receive infrastructure and network-device logs and organize them by source, program, and date. Dual-destination forwarding is the intended client pattern so one collector can continue receiving new events when its peer is unavailable.

Rollout state matters: redundant collectors do not guarantee duplicate log streams unless each client is confirmed to send to both. Devices without durable local logging can lose evidence while forwarding is interrupted.

Centralized logs support chronology, cross-host correlation, and detailed diagnosis. They are not equivalent to active monitoring. A log file can look quiet because the system is healthy, the source stopped forwarding, the collector failed, or the event simply has not occurred.

A freshness checker therefore evaluates recent arrivals for an allow-listed set of known sources and escalates sustained silence through Hermes. Its scope is intentionally narrower than all syslog clients, so uncovered sources and the checker itself remain explicit failure boundaries.

Dedicated watchdogs

Some critical conditions need purpose-built checks rather than generic polling:

Watchdogs should suppress unchanged repeats, preserve recovery as a distinct event, and expose their own readiness where possible. A watchdog closes one detection gap but introduces a new process, timer, credential, and transport path that must also be observed.

Signal transport and durable incidents

Hermes and Ops Console have different responsibilities.

Hermes is a lightweight authenticated transport for existing watchdog and service notifications. It concentrates external delivery policy so each sender does not own a separate notification integration. A successful Hermes submission proves transport acceptance, not durable incident management.

Ops Console is the durable operational cockpit. Its worker stores health snapshots, while authenticated producers submit events and heartbeats. Related events are grouped into incidents with acknowledgment, snooze, mute, notes, and resolution state.

watchdog signal → Hermes → external notification

health check or producer signal
          ↓
      Ops Console
          ↓
 snapshot → event → deduplicated incident
          ↓
 local history + policy-controlled escalation

Not every Hermes notification is automatically an Ops Console incident, and not every Ops Console event needs external delivery. Treating the two systems as interchangeable would hide gaps. Their overlap should be deliberate and verified per producer or signal path.

Ops Console also watches producer freshness and its own worker heartbeat. This makes “nothing is wrong,” “nothing changed,” and “the monitoring path stopped reporting” distinguishable states.

Quiet escalation

The durable dashboard is the complete operational record; external notifications are an interruption budget.

The escalation policy favors:

1. grouping repeated signals into one continuing incident
2. suppressing covered noise during maintenance windows
3. quieting routine delivery during quiet hours
4. allowing critical conditions to override silence
5. suppressing flapping notification repeats
6. preserving recovery and still-broken reminders as distinct operator signals
7. recording delivery failures rather than losing them silently

Dedicated infrastructure watchdogs may continue using Hermes for direct external escalation while Ops Console adoption expands. That coexistence is explicit; migration is not assumed complete merely because both systems exist.

Backup monitoring

Backup protection is monitored separately from backup storage itself. A polling watchdog inspects completed task history for failed backup and datastore-maintenance operations, then routes failures through the shared signal transport.

This is intentionally independent of a backup server’s native notification configuration. The trade-off is bounded detection delay and another watchdog process to monitor. A successful backup notification path also does not prove restorability; restore validation remains part of the backup architecture.

Authority boundaries

Each layer is authoritative only for its own evidence:

For risky changes, inspect the live system and its source of truth. Dashboards, alerts, and documentation provide context; they do not authorize remediation.

Failure boundaries

The design prefers failures that remain visible through another independent layer. Where that is not yet true, the gap should be named rather than hidden behind a green dashboard.

Related documentation

• Ops Console — durable events, incidents, freshness, and escalation policy
• Backup and Recovery — backup and restore boundaries
• Local Inference Stack — model-serving readiness and promotion gates
• RAG Pipeline — retrieval metrics and evidence boundaries
• Network Monitoring & Observability — network-specific monitoring intent
• Lab Philosophy — authority and automation principles