Skip to content
NetworkBound
All capabilities
Measured in production Proxmox Backup Server Proxmox VE ZFS Prometheus

Backup integrity audit

A backup you have not restored is a hypothesis.

We attempt a real restore from your real media before the answer matters, then fix the retention, the permissions and the gaps the green job list was hiding.

An audit of a backup estate whose job list was green. It found six guests that had never been backed up — including the single-sign-on host and the monitoring server — sixty-six snapshots never verified, and a retention policy that had silently failed for months on a missing permission.

  • 6

    guests never backed up, including SSO and monitoring

  • 2,622 → 749

    snapshots once retention actually ran; dry run predicted 749

  • 48.7 s

    to restore a 1.2 GiB container, timed

  • Delete refused

    for the hypervisor's own token; create and restore still work

Every figure was read from a live estate carrying real load. Measured in production, on live backup media. The restore figures are wall-clock timings of restores that were actually performed.

What you are handed

  • One restore performed end to end from real media, timed, before anything else is discussed
  • Guests absent from every job enumerated — the gap a job list structurally cannot show
  • Retention modelled, dry-run, then compared against the live result
  • Hypervisor credentials that can create and restore but not delete, proven with a throwaway backup
  • The failure domain written down, including whatever remains unfinished

Usually bought as

Assessment 3–5 days

A read of what you are running and what is currently untrue about it. Ends in a written findings list ordered by what it costs to fix versus what it costs to leave.

A written findings list, ranked by risk and effort, that you keep.

Start this one

01 What green was hiding

Every scheduled job reported success. What a job list does not show is the guests that are not in any job: six of them, including the single-sign-on host that every other service authenticates against and the monitoring server that would have been expected to notice. All 66 existing snapshots had a verify status of never.

The moment a metrics exporter was pointed at the backup server, a further nine guests surfaced with no backup for 75 to 77 days. None of this was visible from the hypervisor's own view of its backup jobs, which is the view everyone had been looking at.

02 Retention that never ran

The retention policy was roughly fourteen snapshots per group. The datastore held 2,622. Pruning had been failing for months on a missing permission, and failing quietly.

Before the live run, a dry run predicted 1,873 snapshots removed and 749 kept, with no group dropping below 12. The live run produced exactly 2,622 → 749. That the prediction and the result matched is the point — a prune that removes a surprising number is a prune you stop and investigate, not one you let finish.

A second root cause followed: a hypervisor's backup tool prunes only the groups it has just backed up, so guests that were deleted or migrated away are never pruned by anyone. Datastore-wide prune jobs on the backup server itself fix that. And garbage collection reporting “0 bytes removed” immediately afterwards is not a bug — chunks carry a 24-hour access-time grace period — which is the kind of thing worth knowing before spending an evening chasing it.

03 A hypervisor that cannot delete its own backups

If a hypervisor is compromised, the attacker holds whatever credential it uses to reach the backup server. That credential should not be able to destroy the backups. The hypervisor's token was stripped of prune and modify rights, then tested with a throwaway backup: delete refused, create and restore still work.

04 Restores, timed

  • A 1.2 GiB container restored end to end in 48.7 seconds.
  • A 2.5 TiB container backed up as a 25 GiB incremental.
  • A failed NVMe boot drive recovered from the backup server rather than rebuilt by hand.

Next

Want this one, scoped for your estate?

Start a conversation See everything else