Monitoring drift reconciliation

01 Monitoring lies quietly

A container gets renumbered. The monitor keeps polling the old address. The nodata trigger fires, the host goes HIGH, and the alert that lands in your inbox is indistinguishable from a real outage.

So you investigate it once. You find the service is fine. And the next time that alert appears, you skim past it — which is precisely the failure mode monitoring exists to prevent. The system is still green on its own dashboard the whole time, because nothing in it knows it is polling an address that no longer belongs to anything.

Meanwhile the inverse is quieter still: a guest nobody added is not monitored at all, and nothing anywhere says so.

02 Reconciling two sources of truth

The hypervisor knows what guests exist and what addresses they have. The monitor knows what it is polling. Nothing compares the two, so they drift, and the drift is invisible until it costs you an incident.

reconcile.py pulls both inventories and reports five classes of disagreement.

03 What a first run finds

The tool exists because a single container was renumbered and monitoring failed to notice. On a first run against a live estate it found nine hosts whose monitoring interface was 0.0.0.0, plus one polling an address the host had not held for weeks.

That is the whole argument for running it. Every one had been raising a high-severity unreachable alert for days against a service answering perfectly well on its own port.

04 Being careful about what it does not know

A tool that reports drift is only useful if a clean result means something. The easy version compares two lists and calls every difference a problem, which produces a wall of findings nobody reads.

So the awkward cases are handled explicitly rather than swept into a bucket. A DHCP container, or a virtual machine with no guest agent, has no address the hypervisor can vouch for — those are reported as unverifiable, never counted as clean. A host polled by hostname rather than by address is not compared at all, because comparing a name to an address is meaningless.

Equipment that is legitimately not a virtualisation guest — switches, access points, a UPS, the hypervisors themselves — will always look orphaned, so groups can be excluded. That exclusion caused a bug worth keeping: filtering hosts out of the comparison also removed them from the name index, so their perfectly-monitored guests started reporting as unmonitored. There is now a regression test named after it.

05 The rest of the toolkit

• zbx.py — the Zabbix 7.x client the others share. Version 7.0 moved API authentication to an Authorization: Bearer header; the old auth body field is now ignored rather than rejected, so scripts written against 6.x fail with a permission error that never mentions authentication.
• problems.py — bulk triage. Every close is a dry run until --apply, and triggers without manual_close are reported as skipped rather than counted as done, because the server will not close those and pretending otherwise hides work.
• inventory.py — host export plus a forward and reverse DNS audit.
• templates.py — templates under version control. Configuration that only exists inside a database is configuration you cannot diff, review, or roll back.
• install-agent.sh / bulk-install-agents.sh — agent rollout through the hypervisor, so no credentials are ever needed inside the guests.

06 Running it on a schedule

Drift is worth catching every morning, not whenever someone remembers. But there is a constraint that shapes the whole design: the monitoring and hypervisor APIs sit on a private network, and a hosted runner has no route to them.

So the work splits. Lint, tests and a secret scan run anywhere — the tools keep their comparison logic separate from their I/O precisely so the interesting parts are testable with no monitoring server in reach, which is what makes 41 tests runnable on a hosted runner. The live reconcile runs on a self-hosted runner inside the network.

Both workflow files live in .github/workflows/, which Gitea Actions also reads — so one set serves a hosted forge and a self-hosted one equally.