10 Prometheus Alerts That Wake Up Admins at Night, and What to Do With Each

Your phone is quiet all evening, then at 3 a.m. it starts buzzing. Sound familiar? Below are ten alerts from the Prometheus Alerts Pack v1.0 rule set, with concrete steps for each one: what the alert means, where to start the diagnostics, how to fix it, and when it is acceptable to silence it.

A fair disclaimer: this is a selection of typical scenarios, not measured statistics. We did not count which alerts actually wake people up most on your servers. The thresholds and commands below come from the runbook. If your thresholds differ, follow your own rules.

One rule covers all of them: first find out what is broken and whether users are affected, then fix it. And when you silence an alert in Alertmanager, always add a comment and an expiry time. A silence with no expiry is monitoring that has been switched off.

1. NodeDown

Prometheus has been unable to scrape node-exporter for 3 minutes (up == 0). This is not necessarily a server outage. Only the exporter or the network path to port 9100 may be down.

Start with a check:

ping -c 3 <host> curl -sS -m 5 http://<host>:9100/metrics | head ssh <host> 'systemctl status node_exporter --no-pager'

The unit name depends on how the exporter was installed: it can be node_exporter or prometheus-node-exporter. If the server is unreachable, deal with the hosting provider or the hypervisor console. If only the exporter is down, restart it (systemctl restart node_exporter), then confirm that port 9100 is listening (ss -ltnp | grep 9100) and not blocked by the firewall.

It is acceptable to silence this during planned work or when decommissioning the host. Even better, remove the target from the Prometheus configuration.

2. NodeDiskSpaceCritical

Less than 5% free space is left on the partition. Writes may start failing at any moment, so this is no longer a warning. It is a task for right now.

df -hT du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -n 20 ls -lhS /var/log | head journalctl --disk-usage docker system df

To free space quickly, truncate the largest log with : > /var/log/<file> rather than rm, because a process may still hold the deleted file open. Shrink the journal with journalctl --vacuum-size=200M, and delete old backups. Run docker system prune only after you have read carefully what it will remove. Afterwards, find the source of the growth and extend the volume.

Do not silence this until space has actually been freed.

3. NodeDiskWillFill

This alert is about the trend, not the current state. A linear extrapolation over the last 6 hours says the partition will run out of space within the next 24 hours. It fires only when less than 40% is free, so large, mostly empty disks don't generate noise.

df -h <mountpoint> du -xh --max-depth=2 <mountpoint> 2>/dev/null | sort -h | tail -n 20 find <mountpoint> -xdev -type f -mmin -60 -size +100M -exec ls -lh {} +

Look at the node_filesystem_avail_bytes graph for the last 6 to 24 hours. Steady growth usually points to logs or a database. A sudden jump usually means an import, an export or a backup. If it is a one-off operation and there is enough space to finish it, a silence of a few hours is fine, but not days.

4. NodeMemoryCritical

Less than 4% of memory is available. The kernel will soon start killing processes through the OOM killer, and the system will swap heavily and slow down.

free -h ps aux --sort=-%mem | head -n 10 dmesg -T | grep -i -E 'out of memory|killed process' | tail vmstat 1 5

The causes are the same as for NodeMemoryLow, only more acute: a sharp leak or an unexpected load spike. Stop or restart the main memory consumer, and reduce the load temporarily if needed. Adding RAM or swap is a valid stopgap, but you still need to find the root cause. Do not silence this until the system has stabilised.

5. NodeCpuHigh

Average CPU load across all cores has stayed above 90% for 15 minutes. Common causes are a traffic peak, a runaway process, or a heavy backup or build.

top -b -n 1 | head -n 15 ps aux --sort=-%cpu | head -n 10 pidstat 1 5 docker stats --no-stream

If an unknown process is eating CPU with no obvious reason, check it for signs of compromise. Then limit the culprit with nice, cpulimit, or cgroup/Docker limits, optimise queries, or add cores. Silencing is reasonable on batch-compute hosts where 100% CPU is normal. For those, exclude the host from the rule or raise the threshold.

6. NodeClockSkew

The system clock differs from the reference time by more than 0.05 seconds for more than 10 minutes. It sounds minor, but fix it rather than silence it.

timedatectl status chronyc tracking && chronyc sources -v systemctl status chronyd systemd-timesyncd --no-pager journalctl -u chronyd -u systemd-timesyncd -n 50 --no-pager

The usual cause is simple: the time sync service is not running, or the NTP servers are unreachable, for example because UDP/123 is blocked. Start and enable the service (systemctl enable --now chronyd or systemd-timesyncd). To force an immediate correction, use chronyc makestep.

7. DockerContainerRestartLoop

In the last 15 minutes the container has started again three or more times. The process crashes right after starting, Docker restarts it, and the cycle repeats.

docker ps -a --filter name=<name> docker inspect <name> --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.RestartCount}}' docker logs --tail 200 <name> docker events --since 15m --filter container=<name>

Pay attention to the exit code. Exit code 137 usually means SIGKILL, including an OOM kill, while exit code 1 means an error in the application itself. Common causes are a broken configuration or environment variables, an unavailable dependency such as a database, a wrong image or command, or a missing file or volume. If memory is the problem, move on to the DockerContainerOomKilled alert. Silencing this one is acceptable only on a test setup, never in production.

8. KubePodCrashLooping

A container has been in CrashLoopBackOff for 10 minutes. It starts, crashes, and Kubernetes restarts it with a growing delay.

kubectl -n <ns> get pod <pod> kubectl -n <ns> describe pod <pod> kubectl -n <ns> logs <pod> -c <container> --previous kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -n 20

The key tool here is the --previous flag, which shows the log of the previous, crashed run. The Events and Last State sections of describe show why the container terminated. Fix the cause shown in the log: the configuration, a secret, or a dependency. If a liveness probe is failing, increase initialDelaySeconds or failureThreshold. After the fix, restart the deployment: kubectl -n <ns> rollout restart deployment/<name>.

9. BlackboxProbeFailed

A probe has been failing for 3 minutes in a row (probe_success == 0). The site or service is unavailable, or it responds differently from what the probe module expects. The causes vary: the service is down, DNS is broken, a port is closed, the TLS certificate has expired, or the probe timed out.

curl -sS -v -m 10 -o /dev/null https://example.com dig +short example.com curl -sS -m 10 'http://<exporter>:9115/probe?module=http_2xx&target=https://example.com&debug=true'

The last request returns a detailed probe log, which often shows exactly which step failed. If this is a false positive caused by a timeout, increase timeout in the blackbox module. A silence is acceptable during planned maintenance or a site migration.

10. BlackboxSslExpiresCritical

Less than 3 days remain before the certificate expires, or it has already expired. Visitors will soon see a browser error, and API clients will start refusing the connection.

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -enddate ls -l /etc/letsencrypt/live/<domain>/ 2>/dev/null nginx -t && systemctl status nginx --no-pager

The order of work is: renew or issue the certificate manually (for example, certbot renew), reload the web server or load balancer, then verify the result with openssl s_client. Sometimes the new file is on disk but the server still serves the old one because it was never reloaded. Silence this only for certificates you are deliberately not renewing, such as a decommissioned domain. In every other case, first find out why automatic renewal failed: certbot, cert-manager, or access to the DNS API.

What these alerts have in common

Almost every one of them follows the same pattern: check what the logs say, find the source, and fix the cause rather than the symptom. A silence with no expiry and no comment just postpones the incident.

The full sections, with thresholds, causes and "when it is OK to silence" notes, are in the runbook. Pages for each alert: