Backup and Recovery Architecture

Backups are a recovery system, not part of the normal request path. Services can keep running while backup infrastructure is unavailable, but every missed backup increases the amount of state that could be lost after a failure.

This page describes the shared architecture and recovery boundaries. Detailed restore commands and host-specific dependencies remain in private runbooks beside the systems they protect.

Backup layers

The primary backup layer is Proxmox Backup Server (PBS). Stateful virtual machines and containers are captured by scheduled guest-level backup jobs to a local PBS datastore.

A second PBS datastore is backed by TrueNAS storage in the same rack. The primary datastore synchronizes to that copy on a daily schedule. This provides another restore source if the primary PBS server or datastore fails, but both copies remain inside the same physical failure domain.

stateful guests
      ↓ scheduled backups
primary PBS datastore
      ↓ daily synchronization
same-rack secondary PBS datastore on TrueNAS

The layers have different jobs:

A synchronized copy is still part of the same backup system and rack. It protects against failure of the primary PBS server or its local datastore, but not against rack-level or site-level loss. It does not replace a true off-site copy, restore testing, or independent configuration sources.

Retention intent

The current guest-backup policy keeps multiple recovery horizons:

• seven daily recovery points
• four weekly recovery points
• two monthly recovery points

The secondary datastore follows the PBS synchronization and pruning policy. Retention is intended to cover both recent operational mistakes and failures discovered after the most recent daily window.

Retention is not an archive guarantee. Important source code, infrastructure definitions, and other durable configuration also live in their own authoritative systems.

Guest state and bulk data are different

A guest backup protects the state stored inside that virtual machine or container: its boot disk, application configuration, local databases, and local volumes.

Large shared datasets are recovered through the storage platform that owns them. For example, restoring a media-service guest can recover its operating system and container definitions, but it does not recreate a separate shared media library. Recovery is complete only when both the guest and its external storage dependencies are healthy.

This distinction prevents a common false assumption: a successful VM restore does not prove that every dataset consumed by the application has also been restored.

Recovery prerequisites

Before starting a restore, the operator needs:

1. Compute capacity — the original virtualization host or a replacement capable of running the guest.
2. A reachable backup source — normally the primary PBS datastore, or the same-rack secondary copy if the primary is unavailable.
3. Network intent — the correct trust zone, addressing policy, DNS, and required routes for the restored service.
4. Storage dependencies — shared datasets must be available before dependent applications are declared recovered.
5. Configuration and secrets — retrieved from their secured sources of truth rather than copied from public documentation.
6. A known-good recovery point — selected for integrity and timing, not merely because it is the newest snapshot.

The restore should stop if these prerequisites are unclear. Guessing at network identity, storage ownership, or secret material can turn a recoverable outage into configuration drift.

Restore and validation flow

The common restore sequence is:

1. Select the latest known-good recovery point from the primary or secondary PBS datastore.
2. Restore the guest with its documented compute, memory, disk, and network profile.
3. Confirm the guest boots without storage, filesystem, or resource errors.
4. Reconnect required network and shared-storage dependencies.
5. Start foundational services before user-facing applications.
6. Verify application health directly, then verify the normal routed or user-facing path.
7. Check data recency, expected persistent state, and logs for incomplete startup or corruption.
8. Re-enable scheduled protection and confirm a subsequent backup completes.

A running guest is not sufficient evidence. Recovery is complete only when the service works through its normal path, its required data is present, and protection has resumed.

Failure boundaries

The architecture favors simple, inspectable recovery paths. Automation creates and copies recovery points; an operator still chooses the recovery point, controls the restore, and validates the result.

Related documentation

• TrueNAS storage server — storage platform context
• Monitoring — backup-result alerting and operational visibility
• Lab Philosophy — design principles and authority boundaries