Skip to content
OQVIAN

How to turn network alerts into tracked incidents

When a router goes down, most monitoring tools send an alert, then another, then another. The fix is to treat the first alert as the start of an incident, attach every repeat to it, and close it automatically when the device recovers. OQVIAN built exactly this for Jackson Electric Cooperative.

6 min read

The problem with raw alerts

A raw alert stream has no memory. It cannot tell a new outage from the tenth message about the same one, so people learn to ignore it, and that is when a real outage gets missed.

The pattern

  • Read the monitoring source on a schedule, every minute in our build.
  • Give each alert a key made from the device and the alert rule.
  • If no open incident has that key, open one and notify the on-call person.
  • If one is open, attach the alert to it and stay quiet.
  • When the monitor reports recovery, close the incident and record how long it lasted.

What we built

Five n8n workflows handle polling, opening, merging, recovery and notifications. A small incident service holds the state, so a workflow that runs twice cannot open the same incident twice.

We tested it in a lab with a real LibreNMS install and a virtual MikroTik router, causing outages on purpose and checking every step. 23 behaviour tests pass. No customer data is used in the lab.

What to decide before you build

  • Which alerts count as the same incident.
  • Who is notified first, and who next if nobody responds.
  • How long a recovery has to hold before the incident closes.
  • Where incidents live: a ticket tool, a sheet or a small database.

Drowning in alerts?

Send us your monitoring tool and where incidents should go.

Talk to OQVIAN Automate