# Backup and Recovery Architecture > How guest backups, same-rack secondary copies, retention, and restore validation fit together in the homelab. Status: active · Kind: reference · Last verified: 2026-07-19 · Date: 2026-07-19 Backups are a recovery system, not part of the normal request path. Services can keep running while backup infrastructure is unavailable, but every missed backup increases the amount of state that could be lost after a failure. This page describes the shared architecture and recovery boundaries. Detailed restore commands and host-specific dependencies remain in private runbooks beside the systems they protect. --- ## Backup layers The primary backup layer is Proxmox Backup Server (PBS). Stateful virtual machines and containers are captured by scheduled guest-level backup jobs to a local PBS datastore. A second PBS datastore is backed by TrueNAS storage in the same rack. The primary datastore synchronizes to that copy on a daily schedule. This provides another restore source if the primary PBS server or datastore fails, but both copies remain inside the same physical failure domain. ```text stateful guests ↓ scheduled backups primary PBS datastore ↓ daily synchronization same-rack secondary PBS datastore on TrueNAS ``` The layers have different jobs: | Layer | Purpose | |---|---| | Guest backup | Capture operating system, application configuration, databases, and local persistent volumes as one recoverable unit | | Local PBS datastore | Provide the normal fast restore path and retention history | | Secondary PBS datastore | Preserve another restore source on separate storage in the same rack | | Independent sources of truth | Rebuild configuration, credentials, and version-controlled projects when restoring an image is not the best option | A synchronized copy is still part of the same backup system and rack. It protects against failure of the primary PBS server or its local datastore, but not against rack-level or site-level loss. It does not replace a true off-site copy, restore testing, or independent configuration sources. --- ## Retention intent The current guest-backup policy keeps multiple recovery horizons: - seven daily recovery points - four weekly recovery points - two monthly recovery points The secondary datastore follows the PBS synchronization and pruning policy. Retention is intended to cover both recent operational mistakes and failures discovered after the most recent daily window. Retention is not an archive guarantee. Important source code, infrastructure definitions, and other durable configuration also live in their own authoritative systems. --- ## Guest state and bulk data are different A guest backup protects the state stored inside that virtual machine or container: its boot disk, application configuration, local databases, and local volumes. Large shared datasets are recovered through the storage platform that owns them. For example, restoring a media-service guest can recover its operating system and container definitions, but it does not recreate a separate shared media library. Recovery is complete only when both the guest and its external storage dependencies are healthy. This distinction prevents a common false assumption: a successful VM restore does not prove that every dataset consumed by the application has also been restored. --- ## Recovery prerequisites Before starting a restore, the operator needs: 1. **Compute capacity** — the original virtualization host or a replacement capable of running the guest. 2. **A reachable backup source** — normally the primary PBS datastore, or the same-rack secondary copy if the primary is unavailable. 3. **Network intent** — the correct trust zone, addressing policy, DNS, and required routes for the restored service. 4. **Storage dependencies** — shared datasets must be available before dependent applications are declared recovered. 5. **Configuration and secrets** — retrieved from their secured sources of truth rather than copied from public documentation. 6. **A known-good recovery point** — selected for integrity and timing, not merely because it is the newest snapshot. The restore should stop if these prerequisites are unclear. Guessing at network identity, storage ownership, or secret material can turn a recoverable outage into configuration drift. --- ## Restore and validation flow The common restore sequence is: 1. Select the latest known-good recovery point from the primary or secondary PBS datastore. 2. Restore the guest with its documented compute, memory, disk, and network profile. 3. Confirm the guest boots without storage, filesystem, or resource errors. 4. Reconnect required network and shared-storage dependencies. 5. Start foundational services before user-facing applications. 6. Verify application health directly, then verify the normal routed or user-facing path. 7. Check data recency, expected persistent state, and logs for incomplete startup or corruption. 8. Re-enable scheduled protection and confirm a subsequent backup completes. A running guest is not sufficient evidence. Recovery is complete only when the service works through its normal path, its required data is present, and protection has resumed. --- ## Failure boundaries | Failure | Expected recovery path | |---|---| | Individual guest failure | Restore from the primary PBS datastore | | Primary compute failure | Restore on replacement compute from PBS | | Primary backup-system failure | Use the secondary TrueNAS-backed PBS copy if the storage server remains available | | Rack-level or site-level failure | Both PBS copies may be lost; there is currently no true off-site copy | | Shared-storage failure | Recover through the storage platform; guest restore alone is insufficient | | Extended backup outage | Services may remain online, but the recovery point gets progressively older | | Lost configuration or credentials | Rebuild from the secured authoritative source rather than this page | The architecture favors simple, inspectable recovery paths. Automation creates and copies recovery points; an operator still chooses the recovery point, controls the restore, and validates the result. --- ## Related documentation - [TrueNAS storage server](/homelab/hardware/truenas-epyc-7232p) — storage platform context - [Monitoring](/homelab/software/monitoring) — backup-result alerting and operational visibility - [Lab Philosophy](/homelab/philosophy) — design principles and authority boundaries