Day 25: Service Monitoring Scripts That Cannot Lie to You

A queue worker on one of our hosts died at 02:14 on a Saturday. We found out at 09:40 when a customer emailed. The host knew the whole time -- systemctl is-active would have told us instantly -- but nothing on the box was asking the question on a schedule. The only "monitoring" was a dashboard nobody had open at 2 a.m.

This is the Bash I run now. It is deliberately boring, and every design choice maps to a way the previous setup failed.

The full health check, with honest exit codes 

#!/usr/bin/env bash set -euo pipefail unit="${UNIT:-payments-worker.service}" port="${PORT:-8081}" url="${HEALTH_URL:-http://127.0.0.1:8081/healthz}" fail() { printf 'CRIT %s\n' "$1" >&2; exit 2; } # Layer 1: systemd's view of the unit. systemctl is-active --quiet "$unit" \ || fail "unit ${unit} is $(systemctl is-active "$unit")" # Layer 2: is the port actually accepting connections? ss -ltn "sport = :${port}" | grep -q LISTEN \ || fail "unit ${unit} active but nothing LISTENING on ${port}" # Layer 3: does the app's own health endpoint return 2xx? curl -fsS --max-time 5 "$url" >/dev/null \ || fail "health endpoint ${url} did not return 2xx" printf 'OK %s active, listening, healthy\n' "$unit"

Three layers, each able to fail independently, each exiting non-zero. The exit code is the contract: a scheduler or alerting layer keys off it. A monitor that only prints is a monitor nothing acts on.

Why each layer earns its place

  • systemctl is-active --quiet is the cheapest signal and the first thing to check. If systemd has given up restarting the unit, you are done -- no point probing further.
  • ss -ltn is the layer most checks skip, and it is the one that caught us. Our unit reported active while the listener thread inside the process had crashed. "The process exists" and "the service accepts connections" are different facts.
  • curl -fsS --max-time 5 is the application's own opinion. The -f is non-negotiable: without it, curl treats an HTTP 500 as a successful transfer and your check goes green on a broken backend. --max-time stops a hung endpoint from holding the check open until runs overlap.

Wrap it so it never trips over itself

Running a check every minute introduces two new failure modes: overlapping runs, and an ever-growing log. Both are handled in the wrapper, not the check:

#!/usr/bin/env bash set -euo pipefail exec 9>/run/svc-health.lock flock -n 9 || { logger -t svc-health "still running, skip"; exit 0; } logdir=/var/log/svc-health mkdir -p "$logdir" if /usr/local/bin/check-service.sh >>"$logdir/check.log" 2>&1; then exit 0 fi rc=$? logger -t svc-health "service unhealthy rc=${rc}" printf 'ALERT %s rc=%d\n' "$(date -Is)" "$rc" >>"$logdir/alerts.log" exit "$rc"

The flock -n means a second invocation exits immediately rather than stacking. logger -t routes the meaningful events into journald, which is already size-capped, so the noisy minute-by-minute lines never become a disk-fill incident.

Schedule it where it survives a reboot

A while true; sleep 30 loop dies on reboot and leaves no trace. Use a systemd timer (oneshot service + timer, OnBootSec to arm after boot, visible in systemctl list-timers) or, on a non-systemd host, a single cron line:

# crontab -e -- run the wrapper every minute, discard stdout (it logs itself) * * * * * /usr/local/bin/svc-health-run.sh >/dev/null 2>&1

The logrotate snippet so the monitor never fills the disk 

# /etc/logrotate.d/svc-health /var/log/svc-health/alerts.log /var/log/svc-health/check.log { size 5M rotate 5 compress delaycompress missingok notifempty copytruncate su root root }

I rotate on size 5M rather than daily here so a sudden flood of alert lines triggers a rotation immediately instead of waiting for midnight, and copytruncate truncates each file in place so open handles keep writing without a restart.

The takeaway

The outage was not a hard problem. The host had the answer at 02:14. What was missing was a small script asking three honest questions on a schedule that survives a reboot, with an exit code something acts on, a lock so it never overlaps, and rotation so it never becomes the next incident. Boring, owned, and repeatable beats a dashboard nobody is watching at 2 a.m.