On-call teams, postmortems, and costs of downtime are well-covered topics of DevOps. What's not spoken of is the costs of false alarms in your alerting. The team's ability to effectively handle true issues is hindered by this noise. What are these hidden costs, and how do you eliminate false alarms?
While you're at LISA17, how many monitoring emails do you expect to receive? 50? 100? How many of those need someone's intervention? Odds are you won't need to go off into a corner with your laptop to fix something critical on all of those emails.
Noisy monitoring system defaults and un-tuned alerts barrage us with information that isn't necessary. Those false alerts have a cost, even if it's not directly attributable to payroll. We'll walk through some of these costs, their dollar impacts on companies, and strategies to reduce the false alarms.