Server monitoring tools collect hundreds of metrics. Alerting on all of them creates alert fatigue and makes real problems easier to miss. Prioritize the golden signals.
The four golden signals
- Latency: how long requests take
- Traffic: how much demand the system receives
- Errors: the proportion of failed requests
- Saturation: how fully resources such as CPU, RAM and disk I/O are utilized
Alerting rules
Alert on conditions that affect users rather than on every metric. For example, consider an alert when P95 latency exceeds two seconds instead of simply when CPU usage reaches 90%. If users will feel the impact, an alert is likely justified.
Tool options
Datadog, Dynatrace and New Relic are prominent enterprise options. Prometheus, Grafana and Alertmanager are a common open-source combination. Zabbix and Nagios remain useful for traditional infrastructure-focused monitoring.
Keep reading