The problem with raw alerts
A raw alert stream has no memory. It cannot tell a new outage from the tenth message about the same one, so people learn to ignore it, and that is when a real outage gets missed.
The pattern
- Read the monitoring source on a schedule, every minute in our build.
- Give each alert a key made from the device and the alert rule.
- If no open incident has that key, open one and notify the on-call person.
- If one is open, attach the alert to it and stay quiet.
- When the monitor reports recovery, close the incident and record how long it lasted.
What we built
Five n8n workflows handle polling, opening, merging, recovery and notifications. A small incident service holds the state, so a workflow that runs twice cannot open the same incident twice.
We tested it in a lab with a real LibreNMS install and a virtual MikroTik router, causing outages on purpose and checking every step. 23 behaviour tests pass. No customer data is used in the lab.
What to decide before you build
- Which alerts count as the same incident.
- Who is notified first, and who next if nobody responds.
- How long a recovery has to hold before the incident closes.
- Where incidents live: a ticket tool, a sheet or a small database.
Drowning in alerts?
Send us your monitoring tool and where incidents should go.