Zabbix drift tooling

Monitoring lies quietly

A container gets renumbered. Zabbix keeps polling the old address. The nodata trigger fires, the host goes HIGH, and the alert that lands in your inbox is indistinguishable from a real outage.

So you investigate it once. You find the service is fine. And the next time that alert appears, you skim past it — which is precisely the failure mode monitoring exists to prevent. The system is still green on its own dashboard the whole time, because nothing in Zabbix knows it is polling an address that no longer belongs to anything.

Meanwhile the inverse is quieter still: a guest nobody added is not monitored at all, and nothing anywhere says so.

Reconciling two sources of truth

Proxmox knows what guests exist and what addresses they have. Zabbix knows what it is polling. Nothing compares the two, so they drift, and the drift is invisible until it costs you an incident.

reconcile.py pulls both inventories and reports five classes of disagreement.

What the first run found

It was written after a single container was renumbered and monitoring failed to notice. On its first run against a production estate it found nine hosts whose Zabbix interface was 0.0.0.0, plus one polling an address the host had not held for weeks.

That is the whole argument for the tool. Every one had been raising a high-severity unreachable alert for days against a service answering perfectly well on its own port.

Being careful about what it does not know

A tool that reports drift is only useful if a clean result means something. The easy version of this compares two lists and calls every difference a problem, which produces a wall of findings nobody reads.

So the awkward cases are handled explicitly rather than swept into a bucket. A DHCP container, or a VM with no guest agent, has no address Proxmox can vouch for — those are reported as unverifiable, never counted as clean. A host polled by hostname rather than by IP is not compared at all, because comparing a name to an address is meaningless.

Equipment that is legitimately not a virtualisation guest — switches, access points, a UPS, the hypervisors themselves — will always look orphaned, so groups can be excluded. That exclusion caused a bug worth keeping: filtering hosts out of the comparison also removed them from the name index, so their perfectly-monitored guests started reporting as unmonitored. There is now a regression test named after it.

The rest of the toolkit

• zbx.py — the Zabbix 7.x client the others share. Version 7.0 moved API authentication to an Authorization: Bearer header; the old auth body field is now ignored rather than rejected, so scripts written against 6.x fail with a permission error that never mentions authentication.
• problems.py — bulk triage. Every close is a dry run until --apply, and triggers without manual_close are reported as skipped rather than counted as done, because Zabbix will not close those and pretending otherwise hides work.
• inventory.py — host export plus a forward and reverse DNS audit.
• templates.py — templates under version control. Configuration that only exists inside a database is configuration you cannot diff, review, or roll back.
• install-agent.sh / bulk-install-agents.sh — agent rollout via pct exec, so no credentials are ever needed inside the guests.

Running it on a schedule

Drift is worth catching every morning, not whenever someone remembers. But there is a constraint that shapes the whole CI design: the Zabbix and Proxmox APIs are on a private network, and a hosted runner has no route to them.

So the work splits. Lint, tests and a secret scan run anywhere — the tools keep their comparison logic separate from their I/O precisely so the interesting parts are testable with no Zabbix in reach, which is what makes 41 tests runnable on a hosted runner. The live reconcile runs on a self-hosted runner inside the network.

Both workflow files live in .github/workflows/, which Gitea Actions also reads — so one set serves GitHub and a self-hosted Gitea equally.