01 What green was hiding
Every scheduled job reported success. What the job list does not show is the guests that are not in any job: six of them, including the single-sign-on host that every other service authenticates against and the monitoring server that would have been expected to notice. All 66 existing snapshots had a verify status of never.
The moment a metrics exporter was pointed at the backup server, a further nine guests surfaced with no backup for 75 to 77 days. None of this was visible from the hypervisor's own view of its backup jobs, which is the view everyone had been looking at.
02 Retention that never ran
The retention policy was roughly fourteen snapshots per group. The datastore held 2,622. Pruning had been failing for months on a missing permission, and failing quietly.
Before the live run, a dry run predicted 1,873 snapshots removed and 749 kept, with no group dropping below 12. The live run produced exactly 2,622 → 749. That the prediction and the result matched is the point — a prune that removes a surprising number is a prune you stop and investigate, not one you let finish.
A second root cause followed: the hypervisor's backup tool prunes only the groups it has just backed up, so guests that were deleted or migrated away are never pruned by anyone. Datastore-wide prune jobs on the backup server itself fix that. And garbage collection reporting “0 bytes removed” immediately afterwards is not a bug — chunks carry a 24-hour access-time grace period — which is the kind of thing worth knowing before spending an evening chasing it.
03 A hypervisor that cannot delete its own backups
If a hypervisor is compromised, the attacker holds whatever credential it uses to reach the backup server. That credential should not be able to destroy the backups. The hypervisor's token was stripped of prune and modify rights, then tested with a throwaway backup: delete refused, create and restore still work.
04 Restores, timed
- A 1.2 GiB container restored end to end in 48.7 seconds.
- A 2.5 TiB container backed up as a 25 GiB incremental.
- A failed NVMe boot drive recovered from the backup server rather than rebuilt by hand.