Approach
Most of what makes infrastructure trustworthy is refusing to skip the boring step.
There is very little novel engineering in a reliable estate. There is a great deal of verifying claims that everyone assumed were true, and writing down what was found so the next person does not have to assume them again.
01 Principles
Five commitments
They shape every engagement, whatever its size. Each one is a refusal to assume something that is usually assumed.
- 01
Verify before changing
The first deliverable on any engagement is usually a list of things that are not true: hosts reported down that are running fine, checks configured against addresses that moved, alerts that cannot clear. On one recent audit that list had sixteen numbered entries. Fixing those costs little and changes how much of the rest is trusted.
- 02
Make failure modes explicit
Every automated change is reversible, every destructive one asks first, and anything that could contact a real person during testing is disabled by default. Tooling that quietly does the wrong thing at scale is worse than tooling that refuses.
- 03
Test the thing you are relying on
A failover that has never been performed is a plan, not a capability. A backup that has never been restored is a hypothesis. This work includes stopping the primary and watching what actually happens — including whether the collectors follow, which is the part usually missed. The last time, the role moved in under ten seconds and the restore took 48.7 seconds. Both are written down.
- 04
A clean result has to mean something
Tools that call every difference a problem produce a wall of findings nobody reads. Cases the system genuinely cannot judge are reported as unverifiable rather than counted as healthy, because the value of a green result is entirely in what it rules out.
- 05
Leave it maintainable
Configuration in version control, runbooks written for someone who was not there, and a documented reason for each non-obvious decision. Runbooks that have not been tested are marked as such rather than counted. The measure of the work is whether it survives without the person who did it.
02 Delivery
What an engagement looks like
Five phases. Each ends in something you can hold, which is what stops a project from being 80% done for three months.
- 01
Read what exists
Inventory, access, current state. This is where the list of things that are not true gets written: hosts reported down that are running fine, checks pointed at addresses that moved, backup jobs green against a datastore nobody has restored from.
Deliverable Findings list, ordered by cost to fix against cost to leave.
Inventory, access, current state. This is where the list of things that are not true gets written: hosts reported down that are running fine, checks pointed at addresses that moved, backup jobs green against a datastore nobody has restored from.
- 02
Agree the shape
What gets built, what deliberately does not, and what the finished thing has to be able to demonstrate. Scope written as verifiable outcomes rather than as a component list, because a component list can be delivered without the system working.
Deliverable Written scope with acceptance criteria you can test.
What gets built, what deliberately does not, and what the finished thing has to be able to demonstrate. Scope written as verifiable outcomes rather than as a component list, because a component list can be delivered without the system working.
- 03
Build against a test instance
Changes are developed somewhere that cannot page anyone and cannot overload production. Configuration goes into version control from the first commit, not as a tidy-up at the end when the shape has been forgotten.
Deliverable Reviewable change history with a promotion path.
Changes are developed somewhere that cannot page anyone and cannot overload production. Configuration goes into version control from the first commit, not as a tidy-up at the end when the shape has been forgotten.
- 04
Prove it under failure
The primary gets stopped. The restore gets performed. The alert path gets fired end to end to a real destination. Anything that has never been exercised is recorded as untested rather than quietly counted as working.
Deliverable Measured recovery times and a tested alert path.
The primary gets stopped. The restore gets performed. The alert path gets fired end to end to a real destination. Anything that has never been exercised is recorded as untested rather than quietly counted as working.
- 05
Hand it over properly
Runbooks, architecture notes, and a walkthrough with whoever will hold it next. The measure of the work is whether the estate survives without the person who built it — so handover is a deliverable, not a courtesy at the end.
Deliverable Documentation, runbooks, and a handover session.
Runbooks, architecture notes, and a walkthrough with whoever will hold it next. The measure of the work is whether the estate survives without the person who built it — so handover is a deliverable, not a courtesy at the end.
03 Working together
What you can hold us to
Six commitments about conduct rather than technology. They are the terms an engagement is run on, and they do not change with its size.
- 01
Nothing destructive runs without asking first, and nothing runs against production that has not run against a test instance.
- 02
If a finding turns out to be wrong, it gets corrected in writing rather than quietly dropped from the next report.
- 03
Estimates come with the assumption they rest on, so you can see which one broke when they move.
- 04
Work that cannot be finished gets said out loud while there is still time to do something about it.
- 05
Every access credential issued is recorded, scoped to the task, and revoked at handover.
- 06
No client estate, address or hostname appears in anything published — including this site.