Part 6 of 6 in Building and Operating an HTCondor Pool for Research Computing.
An HTCondor worker can disappear from the pool. A GPU worker can stay online while its VRAM collector stops writing. In the second case, Node Exporter may keep serving the same value on every scrape. Our lab guide routes both alerts to one Slack channel and identifies the worker and the failed check in each message.
This walkthrough starts with the pool from Part 1 and the GPU textfile writer from Part 5. On the monitoring host, the example queries HTCondor and Prometheus once per poll. A local exercise reproduces the alert and recovery decisions without contacting the pool or Slack.
Choose what to monitor
Keep two inventories on the monitoring host. Populate them from the machines you expect to serve and keep them across polls.
| Inventory | Identifier | What one entry covers |
|---|---|---|
| Execute nodes | Exact HTCondor Machine name, such as worker.example.org | All visible slot ads for that machine. One remaining ad is enough for the presence check. |
| GPU workers | Exact Prometheus instance label, such as 192.0.2.10:9100 | The worker’s gpu_vram.prom file. The file can hold readings for several GPU UUIDs. |
The lab guide adds newly observed targets every ten minutes. A worker that has never reported needs a manual entry. When retiring a target, stop its advertisement or scrape registration first so discovery cannot add it back. The example below uses two manually maintained files. Using only the latest query result as the inventory would silently drop the worker you want to detect.
Execute nodes
Run condor_status -af Machine | sort -u on the monitoring host. If a successful query omits an expected name, start counting at the first poll that observes the absence. Collector ads can outlive a lost connection. An alert can therefore arrive more than 15 minutes after the workstation lost power. A nonzero exit or timeout makes the observation unknown. HTCondor ClassAd lifetime
The lab implementation has an extra step. A one-minute writer exports htcondor_execute_node_up and retains the last good textfile if condor_status fails. The lab notifier reads that file without checking its age. A retained zero can prolong an apparent outage, and a retained one can hide one. To diagnose that version, compare the file’s mtime and contents with the command’s current output:
status_file=/var/lib/node_exporter/textfile_collector/htcondor_execute_nodes.prom stat -c '%y %n' "$status_file" cat "$status_file" condor_status -af Machine The example queries condor_status directly. It needs neither this execute-node textfile nor the HTCondor exporter on port 9888.
GPU file updates
On the GPU worker, Part 5’s writer refreshes gpu_vram.prom. Node Exporter exposes the modification time of a successfully read file as node_textfile_mtime_seconds. In Prometheus, run this instant query. Use the job label from your scrape configuration in both selectors and in the script below. Node Exporter textfile collector
node_textfile_mtime_seconds{job="node", file=~"(.*/)?gpu_vram[.]prom"} and on (job, instance) (up{job="node"} == 1) Use the sample value as the file mtime. In the HTTP API’s [timestamp, "value"] pair, using the first element would make a newly queried old file appear fresh. Use one matching file per target. Prometheus instant-query results
| What the monitor receives | How it treats that observation |
|---|---|
| A timestamp at least 900 seconds old | The file already meets the alert threshold on this poll. |
| A successful vector with no row for an expected worker | Start or continue the absence interval. A failed scrape, a removed target, or an unreadable file can all produce this result. |
| A failed API request, a warning, or an invalid result structure | Mark the GPU query unknown for this poll. The execute-node check still has its own result. |
| Duplicate rows or an invalid timestamp for an identifiable instance | Mark only that worker unknown and log the problem. Other valid rows can still be evaluated. |
A zero-byte free-VRAM reading is valid. The check follows the file’s update time. A current file can omit a card, so use the UUIDs from Part 5 to investigate an individual GPU.
Decide when to alert and when to recover
The lab guide uses 15 minutes for failure and 10 minutes for recovery. Its GPU check treats any file younger than 15 minutes as healthy. A writer could update once at 14:00, stop immediately, and still satisfy that test at 14:10. The example adds a tighter freshness requirement and checks for another write.
| Transition | Rule in the example |
|---|---|
| Execute node absent, or GPU file timestamp missing | Alert after 15 minutes of observed absence. Minute 0 is the first omission in a successful query. |
| GPU file present but stale | Alert on the first valid observation with file age ≥ 15 minutes. For example, a 14:00 mtime qualifies at 14:16 even if 14:16 is the first poll. |
| Execute node present again | Recover after 10 minutes of valid polls that see the expected machine name. |
| GPU file updating again | Recover after 10 minutes during which every observed file age is 0–120 seconds and the final mtime is later than the first. |
For either check, an unknown observation, a backward step in the poll clock, or a gap greater than three minutes breaks the pending time window. An open incident stays open. A single fresh write can start the recovery window. If the file ages past two minutes, that window resets. Keep the monitoring host, Prometheus host, and worker clocks synchronized. The example treats future-dated files as unknown.
The display and notification thresholds serve different purposes. Part 5 hides VRAM at 90 seconds. At 100 seconds, the graph can already show a gap while the file still qualifies for this recovery check. 15 minutes controls when a stale file warrants a Slack alert.
Set up the monitoring host
The example uses Python 3’s standard library on Linux. For both checks, use an account that can query the intended HTCondor pool and reach the Prometheus API. By default the program uses http://127.0.0.1:9090; change the endpoint in the script if Prometheus is elsewhere. Grafana is optional. Run these checks under the same account that will own the timer:
set -o pipefail condor_status -af Machine | sort -u curl -fsSG http://127.0.0.1:9090/api/v1/query \ --data-urlencode 'query=node_textfile_mtime_seconds{job="node",file=~"(.*/)?gpu_vram[.]prom"} and on(job,instance) (up{job="node"} == 1)' mkdir -p ~/.local/bin ~/.config/systemd/user install -d -m 700 ~/.config/htcondor-watch touch ~/.config/htcondor-watch/{expected_nodes,gpu_instances} In expected_nodes, enter one HTCondor machine name per line. In gpu_instances, enter one Prometheus instance per line. Use the exact names in your inventory, keeping the port on GPU instances. An empty file disables that check. Both files must exist. Blank lines and lines beginning with # are ignored.
For delivery, enable Incoming Webhooks on your Slack app, choose Add New Webhook to Workspace, and authorize the intended channel. Save the URL as the only line of the file below. The URL contains a secret. Slack incoming webhooks
(umask 077 && touch ~/.config/htcondor-watch/slack_webhook) chmod 600 ~/.config/htcondor-watch/slack_webhook nano ~/.config/htcondor-watch/slack_webhook import fcntl import json import math import os import subprocess import sys import time from datetime import datetime, timezone from http.client import HTTPException from pathlib import Path from urllib import error, parse, request CONFIG = Path.home() / ".config/htcondor-watch" DATA = Path.home() / ".local/state/htcondor-watch" FAIL, RECOVER, MAX_GAP, FRESH = 900, 600, 180, 120 # seconds QUERY = ('node_textfile_mtime_seconds{job="node",file=~"(.*/)?gpu_vram[.]prom"} ' 'and on (job, instance) (up{job="node"} == 1)') def inventory(filename): lines = (CONFIG / filename).read_text().splitlines() return {s.strip() for s in lines if s.strip() and not s.lstrip().startswith("#")} def machines(): result = subprocess.run(["condor_status", "-af", "Machine"], check=True, capture_output=True, text=True, timeout=15) return set(result.stdout.split()) def gpu_times(): url = "http://127.0.0.1:9090/api/v1/query?" + parse.urlencode({"query": QUERY}) with request.urlopen(url, timeout=10) as response: reply = json.load(response) if (not isinstance(reply, dict) or reply.get("status") != "success" or reply.get("warnings")): raise ValueError("Prometheus returned an error or warning") data = reply["data"] if data["resultType"] != "vector" or not isinstance(data["result"], list): raise ValueError("expected a Prometheus instant vector") times = {} for row in data["result"]: name = row["metric"]["instance"] if not isinstance(name, str) or not name: raise ValueError("missing instance label") sample = row.get("value") if (name in times or not isinstance(sample, list) or len(sample) != 2 or not isinstance(sample[1], str)): times[name] = None continue try: times[name] = float(sample[1]) except (TypeError, ValueError, OverflowError): times[name] = None return times def read_source(fetch): try: return fetch() except (OSError, HTTPException, ValueError, KeyError, TypeError, subprocess.SubprocessError) as exc: print(f"{fetch.__name__}: {exc}", file=sys.stderr) return None def gpu_status(times, name, now): if times is None: return "unknown", None if name not in times: return "missing", None stamp = times[name] if stamp is None or not math.isfinite(stamp) or not 0 < stamp <= now: print(f"{name}: ambiguous or invalid VRAM file timestamp", file=sys.stderr) return "unknown", None age = now - stamp if age >= FAIL: return "stale", stamp return ("fresh" if age <= FRESH else "aging"), stamp def transition(rec, now, status, stamp=None): previous = rec.get("last") rec["last"] = now if status == "unknown" or previous is None or not 0 <= now - previous <= MAX_GAP: rec.update(bad=None, good=None, first_stamp=None) if status == "unknown": return None if status in ("missing", "stale"): rec.update(good=None, first_stamp=None) if rec.get("bad") is None: rec["bad"] = now if not rec.get("alerted") and (status == "stale" or now - rec["bad"] >= FAIL): rec["alerted"] = True return "alert" return None rec["bad"] = None if not rec.get("alerted") or status == "aging": rec.update(good=None, first_stamp=None) return None if rec.get("good") is None: rec.update(good=now, first_stamp=stamp) elif now - rec["good"] >= RECOVER and (stamp is None or stamp > rec["first_stamp"]): rec.update(alerted=False, good=None, first_stamp=None) return "recover" return None def save(db): tmp = DATA / "state.tmp" tmp.write_text(json.dumps(db, allow_nan=False)) tmp.replace(DATA / "state.json") def notify(lines): secret = (CONFIG / "slack_webhook").read_text().strip() payload = json.dumps({"text": "\n".join(lines)}).encode() req = request.Request(secret, data=payload, headers={"Content-Type": "application/json"}) with request.urlopen(req, timeout=10) as reply: if reply.status != 200 or reply.read().strip() != b"ok": raise ValueError("unexpected Slack acknowledgment") def deliver(db): pending = db["pending"] if not pending or time.time() < db.get("retry_after", 0): return batch = pending[:10] try: notify(batch) except error.HTTPError as exc: if exc.code == 429: delay = exc.headers.get("Retry-After", "") seconds = int(delay) if delay.isascii() and delay.isdigit() else 60 db["retry_after"] = time.time() + seconds save(db) raise SystemExit(f"Slack HTTP {exc.code}; pending events retained") from None except (OSError, HTTPException, ValueError) as exc: raise SystemExit(f"Slack delivery unconfirmed ({type(exc).__name__})") from None del pending[:len(batch)] db.pop("retry_after", None) save(db) def poll(): path = DATA / "state.json" db = json.loads(path.read_text()) if path.exists() else {"targets": {}} records = db["targets"] pending = db.setdefault("pending", []) nodes, gpus = inventory("expected_nodes"), inventory("gpu_instances") observations = {} if nodes: visible = read_source(machines) now = time.time() for name in nodes: status = "unknown" if visible is None else ("present" if name in visible else "missing") observations["node:" + name] = (now, status, None) if gpus: times = read_source(gpu_times) now = time.time() for name in gpus: status, stamp = gpu_status(times, name, now) observations["gpu:" + name] = (now, status, stamp) for retired in records.keys() - observations.keys(): del records[retired] for key, (observed, status, stamp) in observations.items(): action = transition(records.setdefault(key, {}), observed, status, stamp) if action: utc = datetime.fromtimestamp(observed, timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ") pending.append(f"{utc} {action.upper()} {key} ({status})") save(db) deliver(db) if __name__ == "__main__": os.umask(0o077) DATA.mkdir(mode=0o700, parents=True, exist_ok=True) with (DATA / "monitor.lock").open("a") as lock: fcntl.flock(lock, fcntl.LOCK_EX) poll() What happens when Slack is unavailable?
Each poll stores the incident state and any newly detected transitions together in ~/.local/state/htcondor-watch/state.json before it tries the webhook. Each message line carries the UTC detection time. If an alert and a recovery both occur during a delivery outage, the monitor retains both in order. It sends up to ten events per poll and removes that batch only after an HTTP 200 response with the body ok. HTTP 429 defers the next send according to Retry-After; polling continues in the meantime. Slack webhook responses · Slack rate limits
The file lock serializes overlapping invocations. A timeout after Slack accepted a message, or a crash before the local acknowledgment is saved, can still cause a duplicate. Deleting the state file discards open incidents and unsent messages. Removing a target from the inventory retires it on the next poll. Any transitions already waiting for delivery retain their original timestamps.
Run the local exercise
Save this as exercise.py beside watch_pool.py, then run python3 ~/.local/bin/exercise.py. It calls the same transition code with one simulated poll per minute, using an in-memory record and no webhook.
from watch_pool import gpu_status, transition t0 = 1_700_000_000 worker = "worker:9100" record = {} def check(minute, file_minute): now = t0 + minute *...
