Alerts That Don't Annoy

Alerts that don't annoy are about making sure every notification you receive is meaningful, actionable, and worth interrupting someone for. In observability, an alert is a signal that something needs attention. But not every signal deserves an alert. The goal is to design alerts that fire only when there is a real problem that requires human intervention. This concept is often called "alerting on symptoms, not causes" or "alerting on what matters." It's about shifting from a mindset of "let's alert on everything" to "let's alert only on what truly impacts users or business outcomes."

When you alert on real problems, you reduce noise and increase trust. If every alert is a genuine issue, people respond quickly and thoughtfully. If alerts fire for trivial or transient conditions, people start to ignore them. That's alert fatigue: a state where responders become desensitized and may miss critical issues. Avoiding alert fatigue is not just about fewer alerts; it's about better alerts. It's about understanding which metrics, logs, and traces indicate a problem that a human can fix, and which are just informational. This concept is central to observability because it forces you to think about what "healthy" looks like and what "broken" looks like from the user's perspective.

Alert fatigue is one of the most common reasons observability efforts fail. When teams are bombarded with alerts, they either ignore them, silence them, or spend so much time triaging false positives that they miss the real issues. This leads to slower incident response, higher stress, and eventually, a culture where alerts are seen as noise rather than signals. In contrast, well-designed alerts create a feedback loop: responders trust the alerts, act quickly, and learn from each incident. That trust is essential for maintaining uptime and reliability.

Moreover, alerts that don't annoy help you focus on what matters. In a complex system, there are countless things you could monitor. But only a few of them directly affect users or business goals. By alerting on real problems, you align your monitoring with your service level objectives (SLOs) or key performance indicators (KPIs). This alignment ensures that when an alert fires, it's because something that matters is broken. It also helps you prioritize improvements: if an alert fires often, it might indicate a systemic issue that needs a permanent fix, not just a quick patch. Ultimately, useful alerts are a cornerstone of a healthy observability culture, where data drives action and action drives reliability.

The most common mistake is alerting on causes rather than symptoms. For example, you might set an alert for high CPU usage on a server. But high CPU might be normal during peak load, or it might not affect users at all. If you alert on every cause, you'll get flooded with notifications that don't require action. Another mistake is setting thresholds too low or too high without understanding normal behavior. A threshold that's too low creates false positives; one that's too high misses real problems. Both lead to alert fatigue.

Another frequent error is not having a clear owner or runbook for each alert. When an alert fires, the responder should know what to do. If they don't, the alert is just noise. Similarly, many teams forget to review and tune alerts regularly. Systems change, traffic patterns shift, and what was once a critical alert may become irrelevant. Without periodic review, alerts accumulate and become stale. Finally, some teams use alerts as a substitute for dashboards or logs. Alerts should be for urgent, actionable issues; everything else can be explored in dashboards or logs. Mixing the two creates confusion and fatigue. Avoiding these mistakes requires discipline and a willingness to question every alert: does this truly indicate a problem that needs immediate human attention?

A concrete, repeatable approach to learning useful alerts is to start with a single service and define its service level objectives (SLOs). An SLO is a target for how reliable a service should be from the user's perspective, such as "99.9% of requests succeed" or "95% of requests complete within 300 milliseconds." Once you have an SLO, you can derive alerts from it. For example, you might alert when the error budget burn rate is too high, meaning you're consuming your allowed failures faster than planned. This approach ensures you're alerting on user-impacting symptoms, not internal causes.

To practice, pick a simple application you control. Write down what "healthy" looks like for its users. Then, for each SLO, create a corresponding alert that fires only when the SLO is at risk. Test the alert by simulating failures or using historical data. Ask yourself: if this alert fires at 3 a.m., would I want to be woken up? If not, it's not a good alert. Iterate: remove alerts that don't lead to action, and refine thresholds based on observed behavior. Over time, you'll develop a sense for what constitutes a real problem. This method is repeatable and can be applied to any service, regardless of technology stack. It's a practical way to internalize the principle of alerting on real problems and avoiding alert fatigue.

Useful alerts work best when paired with other observability practices. First, pair alerts with dashboards that provide context. When an alert fires, responders should be able to quickly see related metrics, logs, and traces to understand the scope and impact. This reduces the time to diagnose and fix. Second, pair alerts with runbooks or playbooks that outline steps to mitigate the issue. A good runbook turns an alert into a guided action, reducing stress and errors. Third, pair alerts with a clear incident response process. This includes defining severity levels, escalation paths, and post-incident reviews. Alerts are just the beginning; how you respond matters.

Additionally, pair alerts with regular reviews of their effectiveness. Track metrics like alert volume, false positive rate, and time to acknowledge. Use these to tune and remove alerts. Pair alerts with a culture of blameless postmortems, where the goal is to learn and improve, not to punish. Finally, pair alerts with automation where possible. For example, if an alert indicates a known, safe-to-restart service, you might automate a restart and only alert if the automation fails. This reduces human toil and keeps alerts meaningful. By integrating alerts into a broader observability and response framework, you ensure they remain useful and actionable.

Not every system needs a sophisticated alerting strategy. For small applications, internal tools, or early-stage projects, you can keep it simple: alert on a few key metrics like uptime and error rate, and use a single notification channel. The goal is to avoid over-engineering. If you have a handful of users and a simple architecture, you don't need complex SLOs or burn-rate alerts. Start with basic health checks and expand as your system grows. Simplicity reduces the risk of alert fatigue because you're only alerted when something is clearly broken.

Also, keep it simple when your team is small. With limited people, you can't afford to triage many alerts. Focus on the most critical user journeys and alert only on those. Use manual checks or periodic reviews for less critical components. As your team and system scale, you can introduce more sophisticated alerting. Remember, the goal is not to have the most alerts, but to have the right alerts. Sometimes, a simple threshold on a key metric is enough. Don't add complexity unless it solves a real problem. The principle of alerting on real problems applies at any scale: if an alert doesn't lead to action, it's not useful, no matter how simple or complex your setup.

Q: How do I know if an alert is useful? A: Ask yourself: if this alert fires, will someone need to take action? If the answer is no, it's not useful. Also, track how often alerts lead to meaningful action versus being ignored or silenced. If an alert consistently doesn't require action, consider removing or refining it.

Q: What's the difference between alerting on symptoms and causes? A: Symptoms are what users experience, like slow responses or errors. Causes are internal conditions, like high CPU or disk space. Alerting on symptoms ensures you're focusing on real user impact. Causes can be useful for diagnosis, but they shouldn't wake someone up unless they directly affect users.

Q: How many alerts should I have? A: There's no magic number. The right number is the minimum needed to detect and respond to real problems. If your team feels overwhelmed, you have too many. If you're missing critical issues, you might need more. Regularly review and tune. The goal is to have alerts that are trusted and actionable, not to hit a specific count.