Skip to content

Alert fatigue: debounce, quiet hours and escalation

Alert fatigue sets in when most notifications need no action, so the important one is overlooked. The fix is not fewer checks but stricter rules for what may notify a person: confirm failures before alerting, bundle related failures into one incident, send a recovery notice, keep non-urgent messages out of the night, escalate what nobody acknowledges, and alert on consequences rather than raw metrics.

Alert fatigue sets in when most notifications from your monitoring need no action. People stop reading them, and the one that matters is overlooked. The fix is not to monitor less. It is to be strict about what may interrupt a person: confirm a failure before alerting, turn related failures into one incident, always send the all-clear, keep non-urgent messages out of the night, escalate what nobody picks up, and alert on consequences instead of raw metrics. This article goes through these six rules with concrete examples.

Where does the noise come from?

Count the notifications of one ordinary week and sort them into three piles: needed immediate action, needed action at some point, needed nothing. In most setups the third pile is by far the largest. It typically contains:

  • Spikes. CPU at 95 % for one minute during a backup, one slow response during a deployment.
  • Echoes. A server goes down and every check on it reports separately: ping, HTTP, database, three websites, the cron heartbeat.
  • Flapping. A value sitting right at its threshold crosses it back and forth and reports "problem" and "recovered" every few minutes.
  • Known states. Planned maintenance, a test server that is always nearly full, a job that fails every Sunday and that everyone knows about.
  • Messages without consequence. "Load is 4.2" says nothing unless you know the number of cores and whether anyone was affected; see what the Linux load average means.

Each of these has a specific remedy.

Rule 1: Confirm before you alert (debounce)

A single failed measurement is a data point, not an incident. Debouncing means a check must fail several times in a row before anyone is notified:

12:00  ok
12:01  FAIL   (1 of 2)  no notification
12:02  ok               counter reset, nothing happened
12:07  FAIL   (1 of 2)  no notification
12:08  FAIL   (2 of 2)  incident opens, one notification

With one-minute checks, two or three consecutive failures cost one or two minutes of detection time and remove most spikes. For outside checks, add a second dimension: confirmation by several locations, as explained in uptime monitoring from multiple locations.

The same applies in the other direction. Close the incident only after the check has been healthy for several consecutive measurements. Otherwise a flapping check produces a stream of alerts and all-clears. If a check keeps changing state within a short window, treat it as one "unstable" incident rather than many.

Rule 2: One incident, not twenty messages

When a host dies, everything that depends on it fails within the same minute. Twenty notifications carry the information of one. Good alerting groups them:

  • By cause. If the server itself is unreachable, the failing services on it are consequences and belong under the same incident.
  • By dependency. Virtual machines on a failed host, websites behind a failed database.
  • By time. Whatever is sent, one notification per incident and stage is enough. Repeating the same message every five minutes trains people to mute the channel.

The notification should lead with the probable cause and list the consequences below it, not the other way round.

Rule 3: Always send the all-clear

An alert without a recovery notice leaves a question open: is it still broken? Someone has to log in and look, even at three in the morning. A recovery message that says "resolved after 4 minutes" answers the question and tells you something about severity. A four-minute blip during a deployment and a four-hour outage are different events and should be recognisable as such in the inbox.

Rule 4: Quiet hours for what can wait

Not every finding is worth waking someone. A disk that will be full in nine days, a certificate with twelve days left, a pending security update: all of these are important and none is urgent at 03:00. Quiet hours hold such notifications back until the morning.

Two conditions make quiet hours safe:

  • Measurement continues. Quiet hours must suppress the message, never the check. The history stays complete, and what is still broken at the end of the window is reported then.
  • Urgent things are routed differently. Decide per severity or per channel what may break the silence. A shop that is down at night is a different matter from a warning.

Planned maintenance works the same way: announce a window, mute alerts for the affected systems during it, and let measurement run so you can see afterwards what happened.

Rule 5: Escalate what nobody picks up

A notification is not the same as a reaction. Phones are on silent, people are on trains, messages are delivered to a group in which everyone assumes someone else is on it. Escalation closes that gap with a simple time rule:

StageWhenWhoChannel
1immediatelyperson on dutypush or messenger
2still open after a set timesecond personmessenger and email
3still open after a further delayteam lead or whole teamall channels

The delays depend on what an outage costs you; for example 15 minutes until stage 2 and 30 minutes until stage 3 as a starting point that you then adjust. Escalation only works together with rule 1 and rule 2. Escalating noise makes it worse.

Rule 6: Alert on consequences, not on causes

The most effective change is often to delete alerts. A useful test for every rule: if this fires, what will the person do? If the honest answer is "look, shrug, close", it should be a chart, not an alert.

Instead of alerting onAlert on
CPU above 90 %response time, a full queue, load per core sustained over time
disk above 80 %disk full in a few days at the current growth rate, or above a hard limit
memory above 85 %swap growing steadily, the OOM killer becoming active
one error in the logerror rate rising, a new kind of fatal error
a cron job's mail outputthe job not reporting success within its schedule

The right column describes things that hurt users or will do so soon. The left column describes states that are frequently normal. A disk at 85 % that has been at 85 % for a year needs no attention; one at 60 % that grows by 5 % a day does. If the disk alert does fire, how to find what is filling a Linux disk describes the next step.

Common mistakes

  • Raising thresholds until it is quiet. That removes noise and real warnings alike. Fix the rule, not the number.
  • Sending everything to one group chat. Without an owner, every message belongs to nobody.
  • Sending warnings and critical alerts through the same channel with the same sound.
  • Using quiet hours that stop measurement. The gap in the data is exactly where you will later want to look.
  • Never reviewing. Alert rules age. Once a month, look at which alerts fired most often and what was done in response.
  • Keeping alerts nobody can act on, for example for systems the recipient has no access to.

Checklist

  1. Count last week's notifications and mark those that led to an action.
  2. Require at least two consecutive failures before any notification.
  3. Group failures of the same server into one incident.
  4. Make sure every alert is followed by a recovery notice with duration.
  5. Define quiet hours for warnings and decide what may break them.
  6. Set up at least a second escalation stage with a different person.
  7. Replace metric thresholds with consequence-based rules where possible.
  8. Review the most frequent alerts once a month.

How Lynx Heartbeat helps

Lynx Heartbeat has these rules built in. Failures are debounced, related failures on a server are bundled into one incident with the cause on top, flapping checks are recognised, and the recovery notice includes the duration. Quiet hours and maintenance windows hold back the notification while measurement continues, and escalation in stages reaches further people when nobody reacts. Forecasts such as "disk full in ≈ 9 days" for disks, inodes and memory cover the consequence-based side. Channels and alert options are described on the features page.

Frequently asked questions

What is alert fatigue?

Alert fatigue is the habit of ignoring monitoring notifications because most of them turned out to be unimportant. It is dangerous because the one alert that matters is treated like all the others.

What does debouncing an alert mean?

Debouncing means that a single failed measurement does not notify anyone. Only when the failure is confirmed by further consecutive measurements does an incident open. Short spikes, restarts and deployments then stay silent.

Should monitoring keep measuring during quiet hours?

Yes. Quiet hours should hold back the notification, not the measurement. The check keeps running and the incident is recorded; what is still broken when quiet hours end is reported then.

What is alert escalation?

Escalation means that an alert nobody reacts to is sent again after a set time, to a further person or through a louder channel. It makes sure an incident is not lost because one person is asleep, on holiday or without reception.

Should I alert on CPU usage?

Usually not on its own. High CPU usage is a cause that may or may not hurt anyone. Alert on consequences such as slow responses, a full worker pool or a disk that will be full in a few days, and keep CPU as a chart for diagnosis.

Monitoring

Related articles

All articles