Alarm-to-ticket roll-up

01 The twenty-ticket problem

A fibre span carries many circuits. When it is cut, the monitor is right to raise an alarm for each one — but the ITSM is wrong to open a ticket for each one, and the operator is worse off still if the alternative is a single ticket whose affected-circuit list was typed by hand at two in the morning.

What the NOC actually wants is one incident per cause, opened from the screen the operator is already looking at, with every affected circuit attached as a configuration item so the customer-impact and notification workflows downstream can do their job without anyone re-keying anything.

02 Grouping where the operator already is

The grouping is carried by Zabbix's native cause-and-symptom ranking rather than by anything bolted on. That matters for a mundane reason: a manual event action only ever receives one event ID. If the roll-up lived in a script's memory, it would vanish on navigation and be invisible to the next shift. As a first-class Zabbix feature, the ranking survives a page reload, appears in the problem view for everyone, and is what the webhook reads when it fires.

The operator marks the span alarm as the cause, marks the circuit alarms as its symptoms, and runs one action against the cause. Everything after that is the integration's job.

03 What one run produces

The drill simulates an optical span cut: one master alarm, twenty customer circuits down, and four circuits on a diverse path deliberately left up as a control group. The control group is the important part — an integration that attaches every circuit it can find is not hard to write, and is wrong.

04 Refusing to do the wrong thing

Most of the engineering is in what the integration declines to do. Re-running the action on an incident that already exists returns that incident rather than opening a second. Running it on a symptom alarm is refused, and the refusal names the cause the operator should have run it on instead. A circuit that the CMDB has never heard of is not silently dropped from the ticket: it is listed anyway and called out in a work note, because a missing CMDB record is itself a finding the NOC needs to see.

05 Testing against something ServiceNow-shaped

The integration was built against a staging instance that behaves like ServiceNow where it matters: the real Table API query dialect — equality, negation, LIKE, STARTSWITH, ENDSWITH, IN, ISEMPTY, the ^ and ^OR conjunctions and ORDERBY — stock incident numbering, and the impact-by-urgency priority matrix. It was also exercised in stock Table-API mode with zero server-side customisation, so the pattern ports to a real instance without requiring anyone to install anything on it.

On the monitoring side, the lab Zabbix carried 68 cloned production hosts. Rather than mass-disabling them — which changes the configuration under test — every poller was zeroed through a drop-in, then verified: no poller process running, zero connections to any agent port.

06 Three things the webhook runtime will do to you

• Zabbix caps HttpRequest allocations per webhook execution. A script that creates a fresh request object per call fails at around 23 calls, which a twenty-circuit roll-up reaches easily. Reuse one object and clear its headers between calls.
• {EVENT.SEVERITY} resolves to a name, not a number. Pass it through parseInt and you get NaN, which quietly maps every incident to the lowest priority. {EVENT.NSEVERITY} is the numeric macro, and the one to use.
• Global secret macros are served from the server's configuration cache. Rotate a token and the webhook keeps failing with “Not authorized” — the new value is correct in the database and irrelevant to the running server — until the cache is reloaded.