01 Verify from the receiver, not the sender
The monitoring server showed its notifications as sent. Reading the receivers' logs instead — the mail server, the chat gateway, the webhook endpoint — showed that over seven days, 13 of 10,388 notification attempts had actually arrived. The dashboard had been green the whole time.
There were four causes, all independent, none of them a threshold: service accounts with no media configured sitting in the target group; a webhook whose token had been revoked; an endpoint that no longer existed; and an SMTP authentication mode the relay had stopped accepting. Any one of them alone would have been found quickly. Four at once looked like silence, and silence looks like health.
The same pass found a stock “High CPU” trigger on three templates reading the idle percentage — an alert that could only fire when the machine was doing nothing.
02 Green since April, never installed
The uptime monitor had 96 checks. 50 of them were ping-only, and ping answers whether a kernel is up, not whether a service is. One container had shown UP since April while the service it was supposed to be running had never been installed.
The fix is to make each check ask something only a working service could answer, and to cut the paging scope from all 96 monitors to the 12 that are customer-impacting — because a page that goes to a phone for a lab container is a page that trains the phone to be ignored.
03 Loops
Once alerts can reach people, some of them will not stop. One resend loop produced 1,020 emails in six hours from 17 distinct events; it was root-caused to three compounding deduplication bugs and reduced to zero. A chat alerter was re-sending one scanner's finding — 362 signatures — every five minutes indefinitely; after the fix it produced zero alerts over consecutive real-log cycles.
Three separate cooldown and cap bugs across different tools turned out to be one pattern: UTC ISO strings compared against local-time database timestamps. Every limit failed open.
04 Page, record, or count
Not every signal deserves a person. The policy that comes out of this work has three tiers — urgent, recorded, counted — and every alert source is placed in one deliberately.
- 2,300 intrusion bans across 1,671 addresses back-filled into the policy: 83 % were one-off drive-bys, now counted rather than paged.
- A paging budget of 20 per day was found exhausted by 06:06 with 0 genuine threats inside it. It was replaced with severity-aware budgets.
- Two intrusion-detection signatures accounted for 78.7 % of all alerts — 4,501 of 5,720 — and were suppressed. The ruleset then ran roughly 68,700 rules with 52,700 enabled and 0 kernel drops.