Ops Console Architecture

Status: active Last verified: 2026-07-19 (against the current Ops Console overview, worker and watchdog implementation, adversarial review, and immutable-release runbook) Scope: operational data flow, incident lifecycle, freshness, notification policy, and authority boundaries

Ops Console is a private operations cockpit for the homelab. It gathers periodic health results and pushed signals into one durable timeline, groups related failures into incidents, and keeps external notifications deliberately quiet.

It is not a general automation controller. Operators can manage incident and maintenance state inside the console, but remediation of the monitored systems remains in their authoritative tools and runbooks.

This page omits private hostnames, addresses, endpoints, producer identities, authentication details, and deployment paths.

Runtime shape

The application has two primary processes sharing one persistent database:

The stack is intentionally small and inspectable. The web and worker can be diagnosed independently, while the database provides continuity across process restarts.

scheduled checks                  authenticated producers
      ↓                                   ↓
health snapshots                 events and heartbeats
      ↓ state changes                       ↓
      └──────────── events ─────────────────┘
                         ↓ dedupe key
                      incidents
                         ↓ policy
              dashboard and quiet alerts

Signals have different meanings

Health snapshots

A health snapshot is the result of one scheduled check at one point in time. The worker stores snapshots on every cycle, not only when something breaks. This creates a history for answering when a service changed state and whether the check runner itself was still active.

A state transition can create an event. Repeated identical health results remain useful history without generating repeated incident noise.

Events

An event is an atomic signal: one state change, producer observation, security condition, recovery, or watchdog result. Events are append-oriented evidence. They retain severity, domain, service context, timestamps, metadata, and a deduplication key.

External producers submit events through an authenticated intake boundary. They can also send heartbeats when there is no state-changing event to report.

Incidents

An incident is the operator-facing grouping of related events. The deduplication key joins repeated signals to one continuing problem instead of creating a new inbox item on every poll.

Incident state supports acknowledgment, snoozing, muting, notes, and resolution. Those actions change Ops Console’s record of operator intent; they do not execute fixes against the affected service.

Producer state

Producer state records the latest accepted event or heartbeat. This matters because silence can mean either “nothing changed” or “the producer stopped reporting.” The console keeps those cases separate.

Freshness watchdogs

Each registered producer can declare an expected cadence and a missing threshold. The watchdog compares the latest heartbeat or event with that policy.

producer cadence + last seen time
               ↓
         freshness check
          ↙           ↘
      current          stale
         ↓               ↓
 resolve prior      create or update
 missing incident   missing-producer incident

A producer that suppresses unchanged events must still send heartbeats. Otherwise a healthy but quiet monitored system is indistinguishable from a failed producer.

The worker also records its own heartbeat while storing snapshots. Self-health views combine worker freshness, producer coverage, notification failures, backup status, and application readiness so the cockpit does not present stale data as current without warning.

Quiet escalation

The default objective is durable evidence without constant interruption.

Notification policy applies several controls:

1. Events are grouped before alerting so repeated signals share one incident.
2. Maintenance windows suppress covered event noise while preserving the maintenance record.
3. Quiet hours suppress routine external delivery while allowing critical conditions through.
4. Flapping suppression quiets repeated firing notifications for unstable incidents.
5. Recovery messages and still-broken reminders retain distinct operator meaning and are not treated as ordinary repeated firing noise.
6. Every delivery attempt, including failure, is stored in notification history.

The dashboard remains the complete local record even when an external notification is suppressed. A quiet channel therefore does not imply an empty event history.

Read and mutation boundaries

Most of the console is observational: health, events, incidents, producer coverage, notification history, backups, audit records, and release identity can be read without changing monitored systems.

Allowed operator mutations are deliberately narrow:

• acknowledge, snooze, mute, resolve, or annotate incidents
• create or cancel maintenance windows
• request an Ops Console database backup
• activate or roll back a reviewed Ops Console application release through the deployment workflow

Incident, maintenance, and backup mutations are written to the audit history. Bulk incident operations are validated and bounded.

The console does not restart monitored services, alter network policy, execute arbitrary runbooks, or treat a dashboard action as remediation. It records and coordinates operator intent; system-specific tools retain authority over the systems they manage.

Persistence and recovery

SQLite runs in write-ahead logging mode and stores the operational history outside immutable application releases. Database backups use the online backup interface, verify database integrity, and retain independent recovery files. Restore verifies the selected backup and creates a safety copy before replacing the active database.

The database is authoritative for Ops Console’s own incident and audit history, but not for the underlying infrastructure state. Health snapshots and producer events are observations; live system inspection remains the final authority during diagnosis.

Release safety

Application releases follow an attended, immutable workflow:

1. Build an archive from a reviewed source commit.
2. Bind the archive, source identity, and managed service definitions in an integrity manifest.
3. Transfer the matching artifact pair.
4. Dry-run activation and verify integrity without changing the active release.
5. Activate through an atomic current-release pointer.
6. Restart services explicitly rather than as a hidden activation side effect.
7. Read back release identity, liveness, readiness, worker state, and the operator surface.

Runtime configuration, the database, producer state, and backups stay outside release artifacts.

Rollback uses the retained target-side release ledger and verifies the selected release before switching. Missing or mismatched integrity evidence fails closed; rollback is considered usable only after a controlled round trip and readback have succeeded.

Failure boundaries

The design makes silence inspectable. A healthy check, a quiet producer, a suppressed notification, and a dead monitoring path are represented as different states rather than collapsed into “no alert.”

Related documentation

• Monitoring — broader collection and alert-routing architecture
• RAG Pipeline — source-backed context available to operational agents
• Backup and Recovery — infrastructure recovery boundaries
• Lab Philosophy — authority and automation principles