Methodology

The Restore Drill That Never Ran

Nine storage tiers, thirteen findings, one punchline: the restore drill had never fired. A perimeter that has never been restored from is a hypothesis, not a backup.

Nine storage tiers, years of accumulated backups, and a documented restore path — all of it unproven. The drill that would have proved it had never once been fired.

On 2026-07-28 we ran a single-session audit of our own backup perimeter: nine storage tiers measured against a three-leg standard, producing thirteen findings — five Critical, six Important, two Advisory. Eleven were remediated inside the same session; all thirteen were remediated, resolved or recorded as a deliberate exemption by the same-day follow-up. This is our infrastructure, not a client's.

The finding that reframed the exercise was not one of the five Criticals. It was filed Important, and it reads: the restore drill had never fired. Every claim the perimeter made about itself — that the copies existed, that they were readable, that the documented path worked — was an argument rather than an observation. The whole perimeter was a hypothesis. We fired the drill manually at 10:53 that morning and it passed with zero failures, which is worth exactly one thing: it is the first evidence the system had ever produced about itself.

The standard that came out of the session is Three Legs and a Drill: every tier gets three independent copies; every push claim is verified against the remote rather than the local log; every supervisor check parses rather than string-matches; and a perimeter that has never been restored from is a hypothesis, not a backup.

The method, because a backup audit that trusts its own tooling is theatre

Five instruments ran on 2026-07-28: an eleven-agent adversarially-verified secret scan across eleven trees, an eight-agent recovery-avenue miner, storage-size and snapshot probes on every tier, a remote-reference check for every push claim, and the full test suite for every supervisor script touched.

Adversarial verification means a second pass whose job is to disagree with the first — not decoration here, since two findings below are cases where the obvious instrument reported a comfortable answer and a second look contradicted it.

Nine tiers, three legs, one overdue drill A storage tier fans out to three independent copy legs, with a counter showing offsite legs growing from two to seven. A findings strip records thirteen findings, five critical — including two live production workers with zero durable copies and 4,312 untracked files in an archive repository. Two closing panels: the restore drill had never fired, making the whole perimeter a hypothesis — and its first manual firing on 2026-07-28 passed with zero failures. THREE LEGS AND A DRILLA storage tier(× 9 audited)Leg 1 — local working copyLeg 2 — offsite copyLeg 3 — second, independent offsiteoffsite legs2 → 713 findings · 5 Critical — incl. two live production workers with ZERO durable copiesand 4,312 untracked files in an archive repo with no ignore file ever authored.F-10: the restore drill had never fired.The whole perimeter was a hypothesis.Fired 2026-07-28 10:53PASS — 0 failures
One-session backup-perimeter audit, 2026-07-28; eleven of thirteen findings remediated in-session. A perimeter that has never been restored from is a hypothesis, not a backup.

Leg one — three independent copies, and what "independent" excludes

Four Criticals from the 2026-07-28 session were plain absence of copies, each a different route to the same place.

Untracked content in an archive repository. 4,312 files across 38 entries had never been committed, in a repository with no ignore file ever authored — so nothing had ever been deliberately excluded, and nothing deliberately included either. Fixed in eight batched commits, one producing a 96 MB pack that needed a buffer bump to push, verified against the remote.

Two live production workers whose source had zero durable copies. Both were running and serving traffic. Neither existed anywhere a failure could not reach. Running is not a backup.

No offsite leg for the main working tree. The record's own sentence is the one to remember: the repository sat on a removable drive that was unplugged for 71 hours during the 07-24 failure. A copy physically disconnected during the exact event you are protecting against is not a second leg — it is the first leg with extra steps.

By the end of the session, offsite legs had gone from 2 to 7, adding roughly 13 GB of offsite data, each in a dated archive directory. An undated archive tells you a copy exists, not when it stopped being current.

Leg two — verify every push claim against the remote

A local version-control log will happily report a clean, committed history for work that has never left the machine. It is describing your disk; a backup audit asks about the other end.

Every push claim in the session was checked against the remote reference — one extra command per repository, and the difference between "I committed it" and "it exists somewhere my hardware failure cannot reach." The eight batched commits counted as remediated only after that check returned.

Leg three — every supervisor check parses, never string-matches

Read on 2026-07-28, the supervisor process that watches the backup legs and pages on failure was doing so with a literal string match. Any leg producing the expected text was healthy; anything else was, in practice, invisible. The consequence, stated plainly in the finding: a failure in any new leg would never page. The monitor was not watching the system. It was watching a sentence that a subset of the system happened to emit.

This is the most portable defect in the audit, because the shape recurs in log alerting, uptime checks, CI gates and health endpoints. A check that matches a literal is pinned to the system as it stood on the day it was written, and every leg added afterwards sits silently outside its coverage. Coverage that shrinks as the system grows is worse than none, because it arrives with a green light attached.

The repair regex-parses every leg's output and carries a golden paging contract in the tests: an explicit assertion of which conditions must page, so a change that silences an alarm fails a test rather than passing quietly.

The 629 GB that was not there

An entire cloud drive had been absent from every prior audit. Probed on 2026-07-28, it returned roughly 629 GB in use, which looked like a tier comfortably accounted for.

It was a virtual filesystem echoing the local drive. The number was real in the sense that a size probe returned it; it described bytes already counted elsewhere, presented through a mount that made them look like an independent copy. A tier that mirrors a tier is one tier.

Read correctly, the drive gave up three sole-copy hazards, all fixed in session, plus roughly 40 GB of raw video recorded as a deliberate exemption. Exemptions matter as much as fixes: an exemption on the record is a decision, while an omission is an accident that looks identical from outside.

The general form: a size figure is not evidence of independence. Before a tier counts as a leg, something has to establish that its bytes are not the bytes you already have.

Recovery archaeology, and the finding nobody was looking for

Mining the recovery avenues meant reading artifacts that exist only because a process died — killed session transcripts, partial logs, orphaned working state. The point was to learn what could be recovered after a failure. It found something else.

One transcript recorded a client payment that had been received, was described in an internal note as booked, and had never been marked paid in the books. A live, client-facing accounting error, sitting in production, surfaced by a backup audit that was not looking for it. Resolved the same evening. The client and the amount are not the point and are not published; the class is.

Two things generalize. Recovery archaeology reads the artifacts nobody reads, which is exactly where errors that survived every review live. And the defect had passed through the normal financial process without tripping anything — so that process had no control capable of noticing a disagreement between a note and a ledger.

The severity framework, and its one strange choice

The transferable asset here is not any individual finding. It is the definition we classified against:

Critical = a tier with zero durable copies, a single-point-of-failure copy on removable media, or a live client-facing money defect.

That third clause does not belong in a backup severity scale by any conventional reading. Backup severity is about data; a billing error is a finance issue, on a different board, with a different owner.

We put it there deliberately, and the reasoning is worth more than the classification. A severity scale should rank by harm to the person on the other side of the system, not by which internal team owns the artifact. From the client's position, a lost tier and an unrecorded payment are the same event: something in our system is wrong, they are worse off for it, and they cannot see it. Both are silent, both survive indefinitely if nothing goes looking, and both are found by audit rather than by complaint.

Scales that sort by artifact type produce the failure everyone has seen — a defect triaged into whichever queue matches its file extension, so a money error lands behind a UI ticket because it arrived through a spreadsheet. Sorting by harm puts the accounting error above the archive repository, which is why it was fixed the same evening rather than filed.

One result belongs on the record because it is the boring kind: the secret scan over eleven trees returned zero critical and zero high findings, with one medium — a token value in a local settings file, now ignored. Publishing the clean result alongside the bad ones is what keeps the bad ones believable.

What would prove this wrong

  • A second and third firing of the same drill failing after the first passed. One PASS would close nothing — our own argument applied to our own evidence, and we would publish the failures.
  • A tier with two legs recovering as reliably as a tier with three, across repeated drills. Then the third leg is ceremony.
  • The string-match health check paging correctly against a newly added leg. The finding predicts it cannot; one counter-example overturns it.
  • An independent size probe from a second host returning the same 629 GB against genuinely distinct bytes. Then the mirror explanation is wrong and the tier really was a leg.
  • A financial control that catches the booked-but-unpaid class before an audit does. That defect would stop being audit-only and the severity argument weakens.

The date nobody can name

A backup you have never restored from is a belief. Three copies, a remote-verified push, a parsing health check and a drill that actually fires convert it into a measurement — and the drill is the step people skip, because until it fails it feels redundant and after it fails it is the only thing that mattered. If nobody at your organization can name the date the last restore drill ran, that date is the finding. We will compare perimeters with any team that fires theirs and publishes what came back.

  • #blog
  • #backups
  • #disaster-recovery
  • #restore-drill
  • #infrastructure
  • #audit
Share X LinkedIn
Build it yourself?

Get the kit, not just the theory.

We'll send the build checklist behind this post — and the next pillar when it ships. One email, no drip sequence. Unsubscribe in one click.

Want this built for you?

Book a discovery call. We'll walk your numbers.

20 minutes. Tell us what's broken, hear what we'd ship in the next 90 days. No pitch deck.