Skip to content
NetworkBound
All work
In production Zabbix 7 PostgreSQL 16 TimescaleDB MariaDB pgloader keepalived

Zabbix at scale: live migration and HA

39.7 million rows moved on a live server. Then the primary was switched off on purpose.

Two pieces of work on one production Zabbix estate: moving the backend from MariaDB to PostgreSQL 16 with TimescaleDB while the server kept polling, then clustering it across two hypervisors with a floating address — and stopping the primary to prove the proxies followed.

  • 39.7 M

    history rows back-filled in 13 min 39 s

  • 0 errors

    across 2.08 M configuration rows, counts matched

  • < 10 s

    role move when the primary was stopped

  • 112

    hosts — 100 enabled, 0 unavailable — 13,093 items

01 Why move a database that works

Zabbix history tables grow without limit and dominate everything: disk, backup windows, housekeeper run time, the speed of every graph. TimescaleDB gives Zabbix native compression and time-partitioned history, and it runs on PostgreSQL. The estate was on MariaDB.

Migrating a monitoring backend is uncomfortable because the monitoring is the thing that would tell you if the migration went wrong. So the plan was built around keeping a way back open at every step until the new database had been proven.

02 The migration, with the exit held open

  • 2.08 million configuration rows moved with pgloader — 0 errors, and row counts matched table by table before anything else happened.
  • 39.7 million history rows back-filled in 13 minutes 39 seconds, then TimescaleDB compression enabled on the history hypertables.
  • A storage snapshot of the pre-migration server held as the fallback until the new backend had been verified serving live polling.
  • The old database dumped, its checksum recorded, and only then dropped.

03 The gotchas, recorded

Four things bit during the migration and are now in the runbook so they bite nobody else: a schema-rename step the documentation glosses over; a TimescaleDB version flag the Zabbix release did not yet accept; the frontend needing a different database driver package than the server; and a macro edited in the leftover database that appeared to “not take” because it had been changed in the wrong backend.

04 Then the failover

With the backend settled, the server was clustered using Zabbix's native high availability across two hypervisors, with keepalived holding a floating address in front. Then the primary was stopped.

The active role moved in under ten seconds. The floating address followed. The proxy reconnected. Fail-back was clean. All 55 agents were re-pointed at the floating address, and both nodes were version-pinned so a package upgrade on one cannot split the pair.

  • A proxy's server setting takes one address. Without the floating address, a perfectly healthy failover would have blinded roughly 70 proxy-monitored hosts — the server would be up, and the proxy would be talking to the wrong node.
  • keepalived's nopreempt silently breaks failover in this configuration. It reads as a sensible setting and produces a floating address that does not move.

05 What this HA is, and is not

This is high availability across two hypervisors. The two nodes share one database, so the database remains a single point of failure, and it is written down as one. The architecture is designed for a second site, but the standby site has no hardware yet. It is not geo-redundant and this page does not call it that.

A failover that has been performed is a capability. A second site that has been designed is a plan. Both are useful; they are not the same thing.

06 The estate it holds up

The server grew from 45 to 75 to 112 hosts — 100 enabled, 0 unavailable — carrying 13,093 items and 5,008 triggers. Changes are rehearsed on a staging instance first: a cloner refuses any destination that lacks an environment marker macro, and disables every action and media channel after import, so the copy cannot page anyone. The staging instance carries 7 templates, 9,266 items and 3,396 triggers with 0 enabled actions.