# Ops Console Architecture > How health snapshots, producer events, incidents, freshness watchdogs, and quiet escalation form a private operations cockpit. Status: active · Kind: reference · Last verified: 2026-07-19 · Date: 2026-07-19 > Status: active > Last verified: 2026-07-19 (against the current Ops Console overview, worker and watchdog implementation, adversarial review, and immutable-release runbook) > Scope: operational data flow, incident lifecycle, freshness, notification policy, and authority boundaries Ops Console is a private operations cockpit for the homelab. It gathers periodic health results and pushed signals into one durable timeline, groups related failures into incidents, and keeps external notifications deliberately quiet. It is not a general automation controller. Operators can manage incident and maintenance state inside the console, but remediation of the monitored systems remains in their authoritative tools and runbooks. This page omits private hostnames, addresses, endpoints, producer identities, authentication details, and deployment paths. --- ## Runtime shape The application has two primary processes sharing one persistent database: | Component | Responsibility | |---|---| | Web application | Serve the dashboard and bounded APIs for event intake, health views, incidents, maintenance, audit, and operational history | | Async worker | Run scheduled checks, store health snapshots, detect state changes, evaluate freshness, and dispatch eligible notifications | | SQLite database | Preserve snapshots, events, incidents, producer state, notification attempts, maintenance windows, and audit records | The stack is intentionally small and inspectable. The web and worker can be diagnosed independently, while the database provides continuity across process restarts. ```text scheduled checks authenticated producers ↓ ↓ health snapshots events and heartbeats ↓ state changes ↓ └──────────── events ─────────────────┘ ↓ dedupe key incidents ↓ policy dashboard and quiet alerts ``` --- ## Signals have different meanings ### Health snapshots A health snapshot is the result of one scheduled check at one point in time. The worker stores snapshots on every cycle, not only when something breaks. This creates a history for answering when a service changed state and whether the check runner itself was still active. A state transition can create an event. Repeated identical health results remain useful history without generating repeated incident noise. ### Events An event is an atomic signal: one state change, producer observation, security condition, recovery, or watchdog result. Events are append-oriented evidence. They retain severity, domain, service context, timestamps, metadata, and a deduplication key. External producers submit events through an authenticated intake boundary. They can also send heartbeats when there is no state-changing event to report. ### Incidents An incident is the operator-facing grouping of related events. The deduplication key joins repeated signals to one continuing problem instead of creating a new inbox item on every poll. Incident state supports acknowledgment, snoozing, muting, notes, and resolution. Those actions change Ops Console's record of operator intent; they do not execute fixes against the affected service. ### Producer state Producer state records the latest accepted event or heartbeat. This matters because silence can mean either “nothing changed” or “the producer stopped reporting.” The console keeps those cases separate. --- ## Freshness watchdogs Each registered producer can declare an expected cadence and a missing threshold. The watchdog compares the latest heartbeat or event with that policy. ```text producer cadence + last seen time ↓ freshness check ↙ ↘ current stale ↓ ↓ resolve prior create or update missing incident missing-producer incident ``` A producer that suppresses unchanged events must still send heartbeats. Otherwise a healthy but quiet monitored system is indistinguishable from a failed producer. The worker also records its own heartbeat while storing snapshots. Self-health views combine worker freshness, producer coverage, notification failures, backup status, and application readiness so the cockpit does not present stale data as current without warning. --- ## Quiet escalation The default objective is durable evidence without constant interruption. Notification policy applies several controls: 1. Events are grouped before alerting so repeated signals share one incident. 2. Maintenance windows suppress covered event noise while preserving the maintenance record. 3. Quiet hours suppress routine external delivery while allowing critical conditions through. 4. Flapping suppression quiets repeated firing notifications for unstable incidents. 5. Recovery messages and still-broken reminders retain distinct operator meaning and are not treated as ordinary repeated firing noise. 6. Every delivery attempt, including failure, is stored in notification history. The dashboard remains the complete local record even when an external notification is suppressed. A quiet channel therefore does not imply an empty event history. --- ## Read and mutation boundaries Most of the console is observational: health, events, incidents, producer coverage, notification history, backups, audit records, and release identity can be read without changing monitored systems. Allowed operator mutations are deliberately narrow: - acknowledge, snooze, mute, resolve, or annotate incidents - create or cancel maintenance windows - request an Ops Console database backup - activate or roll back a reviewed Ops Console application release through the deployment workflow Incident, maintenance, and backup mutations are written to the audit history. Bulk incident operations are validated and bounded. The console does **not** restart monitored services, alter network policy, execute arbitrary runbooks, or treat a dashboard action as remediation. It records and coordinates operator intent; system-specific tools retain authority over the systems they manage. --- ## Persistence and recovery SQLite runs in write-ahead logging mode and stores the operational history outside immutable application releases. Database backups use the online backup interface, verify database integrity, and retain independent recovery files. Restore verifies the selected backup and creates a safety copy before replacing the active database. The database is authoritative for Ops Console's own incident and audit history, but not for the underlying infrastructure state. Health snapshots and producer events are observations; live system inspection remains the final authority during diagnosis. --- ## Release safety Application releases follow an attended, immutable workflow: 1. Build an archive from a reviewed source commit. 2. Bind the archive, source identity, and managed service definitions in an integrity manifest. 3. Transfer the matching artifact pair. 4. Dry-run activation and verify integrity without changing the active release. 5. Activate through an atomic current-release pointer. 6. Restart services explicitly rather than as a hidden activation side effect. 7. Read back release identity, liveness, readiness, worker state, and the operator surface. Runtime configuration, the database, producer state, and backups stay outside release artifacts. Rollback uses the retained target-side release ledger and verifies the selected release before switching. Missing or mismatched integrity evidence fails closed; rollback is considered usable only after a controlled round trip and readback have succeeded. --- ## Failure boundaries | Failure | Expected behavior | |---|---| | One health check fails | Store the unhealthy snapshot, create a state-change event when appropriate, and group it into an incident | | Producer stops reporting | Watchdog creates or updates a missing-producer incident after the configured freshness threshold | | Worker stops | Snapshots and self-heartbeat stop advancing; readiness and self-health expose stale worker state | | Web process stops | Dashboard and intake become unavailable while already stored history remains in the database | | External notification delivery fails | Incident remains visible locally and the failed delivery is recorded for diagnosis | | Maintenance or quiet-hours policy applies | External noise is suppressed according to policy; durable local history remains available | | Database is unavailable or corrupt | Intake and history fail; recover from an integrity-checked backup before resuming normal operation | | Release activation is interrupted | Reconcile managed service definitions and verify the current-release ledger before restart | | Monitored service needs repair | Use that system's authoritative tool or runbook; Ops Console does not remediate it | The design makes silence inspectable. A healthy check, a quiet producer, a suppressed notification, and a dead monitoring path are represented as different states rather than collapsed into “no alert.” --- ## Related documentation - [Monitoring](/homelab/software/monitoring) — broader collection and alert-routing architecture - [RAG Pipeline](/homelab/software/rag-pipeline) — source-backed context available to operational agents - [Backup and Recovery](/homelab/software/backup-and-recovery) — infrastructure recovery boundaries - [Lab Philosophy](/homelab/philosophy) — authority and automation principles