Skip to content
NetworkBound
All capabilities
Measured in production Zabbix 7 PostgreSQL 16 TimescaleDB MariaDB pgloader keepalived

Live database migration and monitoring HA

39.7 million rows moved on a live server. Then the primary was switched off on purpose.

We move your monitoring backend to PostgreSQL and TimescaleDB without stopping collection, then cluster it and stop the primary in front of you to show the collectors follow.

Two pieces of work that belong together: moving a monitoring backend from MariaDB to PostgreSQL 16 with TimescaleDB while the server keeps polling, and clustering it behind a floating address — then stopping the primary, because a failover nobody has performed is a plan rather than a capability.

  • 39.7 M

    history rows back-filled in 13 min 39 s

  • 0 errors

    across 2.08 M configuration rows, counts matched

  • < 10 s

    role move when the primary was stopped

  • 112

    hosts — 100 enabled, 0 unavailable — 13,093 items

Every figure was read from a live estate carrying real load. Measured in production, on a live estate that kept collecting throughout. The fallback path was held open at every step until the new backend had been proven.

What you are handed

  • The migration plan with the rollback point named at each step, agreed before anything moves
  • Row-count reconciliation table by table, recorded before cut-over
  • A clustered pair behind a floating address, with the failover performed and timed in front of you
  • A runbook carrying the four things that bit during the migration, so they bite nobody else
  • The failure domain written down — including the parts that remain single points of failure

Usually bought as

Build 2–6 weeks

Design and delivery of a defined system — a monitoring estate, an alarm-to-ticket integration, an access layer, a backup regime, a hosting platform — including the documentation and the handover.

A working system you can operate without us.

Start this one

01 Why move a database that works

Monitoring history tables grow without limit and dominate everything: disk, backup windows, housekeeper run time, the speed of every graph. TimescaleDB gives Zabbix native compression and time-partitioned history, and it runs on PostgreSQL. The estate in question was on MariaDB.

Migrating a monitoring backend is uncomfortable because the monitoring is the thing that would tell you if the migration went wrong. So the plan is built around keeping a way back open at every step until the new database has been proven.

02 The migration, with the exit held open

  • 2.08 million configuration rows moved with pgloader — 0 errors, and row counts matched table by table before anything else happened.
  • 39.7 million history rows back-filled in 13 minutes 39 seconds, then TimescaleDB compression enabled on the history hypertables.
  • A storage snapshot of the pre-migration server held as the fallback until the new backend had been verified serving live polling.
  • The old database dumped, its checksum recorded, and only then dropped.

03 The gotchas, recorded

Four things bit during the migration and are now in the runbook you are handed: a schema-rename step the documentation glosses over; a TimescaleDB version flag the Zabbix release did not yet accept; the frontend needing a different database driver package than the server; and a macro edited in the leftover database that appeared to “not take” because it had been changed in the wrong backend.

04 Then the failover

With the backend settled, the server was clustered using Zabbix's native high availability across two hypervisors, with keepalived holding a floating address in front. Then the primary was stopped.

The active role moved in under ten seconds. The floating address followed. The proxy reconnected. Fail-back was clean. All 55 agents were re-pointed at the floating address, and both nodes were version-pinned so a package upgrade on one cannot split the pair.

  • A proxy's server setting takes one address. Without the floating address, a perfectly healthy failover would have blinded roughly 70 proxy-monitored hosts — the server up, the proxy talking to the wrong node.
  • keepalived's nopreempt silently breaks failover in this configuration. It reads as a sensible setting and produces a floating address that does not move.

05 What this high availability is, and is not

This is high availability across two hypervisors. The two nodes share one database, so the database remains a single point of failure, and it is written down as one. The architecture accommodates a second site, but the standby site has no hardware in it. It is not geo-redundant, and nothing here will call it that.

A failover that has been performed is a capability. A second site that has been designed is a plan. Both are useful; they are not the same thing, and we will not let a proposal blur them.

06 The scale it holds up

The server carried 112 hosts — 100 enabled, 0 unavailable — with 13,093 items and 5,008 triggers, having grown from 45 through 75. Changes are rehearsed on a staging instance first: a cloner refuses any destination that lacks an environment marker macro, and disables every action and media channel after import, so the copy cannot page anyone. The staging instance carried 7 templates, 9,266 items and 3,396 triggers with 0 enabled actions.

Next

Want this one, scoped for your estate?

Start a conversation See everything else