Lorixen.
EN

Alert fatigue: cut the noise so real problems get noticed

Short answer: When your team gets dozens of alerts a day, from the hosting dashboard, uptime checks, error trackers and server monitoring, people stop reading them, and the one alert that matters gets missed. Cut the noise deliberately: list every alert, delete the ones nobody acts on, alert on what customers experience rather than every internal cause, send only urgent, actionable problems to phones, group related alerts into one message, mute during planned work, and review the list monthly.

How alert fatigue happens

Alerts accumulate. Each incident adds one more rule "so we catch it next time". New tools arrive with their default notifications switched on. Nobody removes anything. Before long, a small team has a chat channel scrolling with warnings about CPU spikes, slow responses that recovered on their own, test environments and certificate renewals that will happen automatically.

Google's Site Reliability Engineering book, in its chapter on monitoring, describes the result: when pages occur too often, people second-guess, skim or even ignore incoming alerts, sometimes ignoring a real page masked by the noise, and outages last longer because noise interferes with diagnosis. It concludes that effective alerting systems have good signal and very low noise.

For a business, that means a checkout outage noticed by customers before staff.

Step one: list every alert

Gather every alert source in one document: monitoring tools, hosting providers, error trackers, uptime services, database alerts, cloud cost alerts, security tools. For each alert, note:

  • What triggers it.
  • Where it goes: phone, email, chat channel, ticket.
  • How often it fired in the last month.
  • What someone actually did when it fired.

That last column is the most revealing. Alerts that fired many times with no action are the first to go.

Step two: apply a simple test

The SRE book offers questions for every alerting rule, including whether it detects an otherwise undetected condition that is urgent, actionable and actively or imminently visible to users, and whether you will ever be able to ignore it, knowing it is benign. It adds that every page should be actionable and require intelligence; if a page merits only a robotic response, it should not be a page.

Use that test to sort each alert into one of three groups, following the book's own distinction between pages, tickets and email alerts:

  • Page: someone must act now, at any hour, because customers are affected or soon will be.
  • Ticket or task: needs attention during working hours, such as a disk that will fill in two weeks.
  • Dashboard or log only: useful for investigation, not worth interrupting anyone.

Delete the rest.

Step three: alert on symptoms

The SRE book separates "what's broken", the symptom, from "why", the cause. Prometheus's alerting guidance summarises the same approach: keep alerting simple, alert on symptoms, have good consoles to pinpoint causes, and avoid pages where there is nothing to do.

For a business website or app, symptom alerts are things like:

  • The checkout or booking flow is failing for a significant share of attempts.
  • The site is not loading from outside your network.
  • Error rates on key pages are well above normal.
  • Orders have stopped arriving during normal trading hours.

High CPU on one server, a single slow request or a restarted process are causes, worth seeing on a dashboard but rarely worth a page on their own.

Step four: group, wait and mute

Even good alerts can arrive in floods. Prometheus's Alertmanager documentation describes useful concepts that most alerting tools share in some form:

  • Grouping: combining alerts of a similar nature into a single notification, so that a database outage affecting many services sends one compact message rather than hundreds.
  • Inhibition: suppressing related alerts when a larger one is already firing, such as muting individual service alerts when the whole cluster is unreachable.
  • Silences: muting alerts for a set time, useful during planned maintenance.

Also add a short waiting period to alerts that flicker, so that a problem must last a few minutes before anyone is notified, and send a recovery notice so people know when it is over.

Step five: route to the right place

  • Pages go to the person on call by phone, with escalation if not acknowledged.
  • Tickets go into your task tracker, where they are prioritised like other work.
  • Informational alerts go to a low-traffic channel that people can mute.
  • Keep test and staging alerts completely separate from production.

Keep it clean

Review alerts monthly, or after any noisy week: what fired, what was acted on, what could be removed or tuned. Each new alert should come with a note explaining what to do when it fires; if nobody can write that note, the alert is probably not needed. Watch out for tools that switch notifications back on after updates, and for new services added with their default alerts enabled; both quietly undo the clean-up.

A worked example

Picture a hypothetical Bristol online bookshop whose two developers received around a hundred alerts a day by email and chat. When the payment page failed one Saturday, the relevant alert sat unnoticed among dozens of CPU and disk warnings for hours.

The developers listed every alert, removed those that had never led to action, and converted many into a weekly ticket review. They added symptom alerts for failed checkouts and site availability from outside, sent only those to phones, grouped related alerts, and set silences for planned maintenance. Daily alerts dropped to a handful, and the next payment problem was noticed and fixed within minutes.

FAQ

Is it risky to delete alerts?

Less risky than having alerts nobody reads. Keep the data on dashboards so you can still investigate.

Should warnings go to email?

Only if someone reads that email regularly. A ticket queue is usually better.

How many pages a week is reasonable?

Ideally very few. Each one should be a real problem someone needed to act on.

If your team's phones buzz all day with alerts nobody acts on, tell us which tools send them.

Sources

WhatsAppBook a call