Alarm-to-ticket roll-up

01 The twenty-ticket problem

A fibre span carries many circuits. When it is cut, the monitor is right to raise an alarm for each one — but the ITSM is wrong to open a ticket for each one, and the operator is worse off still if the alternative is a single ticket whose affected-circuit list was typed by hand at two in the morning.

What a NOC actually wants is one incident per cause, opened from the screen the operator is already looking at, with every affected circuit attached as a configuration item so the customer-impact and notification workflows downstream can do their job without anyone re-keying anything. That is the thing we build.

02 Grouping where the operator already is

We carry the grouping on the monitor's native cause-and-symptom ranking rather than on anything bolted alongside it. That matters for a mundane reason: a manual event action only ever receives one event ID. If the roll-up lived in a script's memory it would vanish on navigation and be invisible to the next shift. As a first-class feature of the monitor, the ranking survives a page reload, appears in the problem view for everyone, and is what the webhook reads when it fires.

The operator marks the span alarm as the cause, marks the circuit alarms as its symptoms, and runs one action against the cause. Everything after that is the integration's job.

03 What one run produces

The drill simulates an optical span cut: one master alarm, twenty customer circuits down, and four circuits on a diverse path deliberately left up as a control group. The control group is the important part — an integration that attaches every circuit it can find is not hard to write, and is wrong.

04 Refusing to do the wrong thing

Most of the engineering is in what the integration declines to do. Re-running the action on an incident that already exists returns that incident rather than opening a second. Running it on a symptom alarm is refused, and the refusal names the cause the operator should have run it on instead. A circuit the CMDB has never heard of is not silently dropped from the ticket: it is listed anyway and called out in a work note, because a missing CMDB record is itself a finding the NOC needs to see.

05 Proving it without touching your instance

The work is built against a staging instance that behaves like the real thing where it matters: the actual Table API query dialect — equality, negation, LIKE, STARTSWITH, ENDSWITH, IN, ISEMPTY, the ^ and ^OR conjunctions and ORDERBY — stock incident numbering, and the impact-by-urgency priority matrix. It is also exercised in stock Table-API mode with zero server-side customisation, so the pattern ports to a real instance without anyone having to install something on it first.

On the monitoring side the replica carries 68 cloned hosts. Rather than mass-disabling them — which changes the configuration under test — every poller is zeroed through a drop-in, then verified: no poller process running, zero connections to any agent port. You get a rehearsal that is faithful and an estate that was never at risk.

06 Three things the webhook runtime will do to you

• Zabbix caps HttpRequest allocations per webhook execution. A script that creates a fresh request object per call fails at around 23 calls, which a twenty-circuit roll-up reaches easily. Reuse one object and clear its headers between calls.
• {EVENT.SEVERITY} resolves to a name, not a number. Pass it through parseInt and you get NaN, which quietly maps every incident to the lowest priority. {EVENT.NSEVERITY} is the numeric macro, and the one to use.
• Global secret macros are served from the server's configuration cache. Rotate a token and the webhook keeps failing with “Not authorized” — the new value is correct in the database and irrelevant to the running server — until the cache is reloaded.