Monitoring HTCondor Jobs and GPU Memory with Prometheus and Grafana

Part 5 of 6 in Building and Operating an HTCondor Pool for Research Computing.

One RTX 3060 on our lab dashboard showed 4.18 GiB of free VRAM. On the same worker, the unclaimed HTCondor slot advertised 0 GPUs. A job requesting one GPU could not match that slot at the time of the snapshot. To investigate an idle job, I check its scheduler first, then the resources the worker advertises, and finally the physical card.

What each measurement covers

Our lab configuration scrapes two kinds of endpoints every 15 seconds. A custom exporter serves htcondor_* metrics on port 9888. Node Exporter serves host metrics and our custom gpu_vram_* readings on port 9100. The lab exporter waits 15 seconds after each collection pass, and the GPU writer is scheduled on a 30-second timer. Consecutive scrapes can repeat the same reading.

The screenshots below are excerpts from our dashboard. The smaller collectors at the end of the article are examples for reproducing the queue-count, CPU-allocation, and VRAM queries. The lab dashboard also uses slot and job-detail metrics from its full exporter.

Connect Prometheus and Grafana

Start with a working pool, Prometheus, Grafana, and Node Exporter on a Linux worker. The monitoring account needs permission to run condor_q and condor_status against the pool. For GPU readings, the worker also needs a working NVIDIA driver and the textfile writer. If you do not already export the custom metrics, set up the relevant collector below before testing the panels.

Add these two entries under your existing scrape_configs key in prometheus.yml. Replace the example worker address here and in the queries. In this example Prometheus and the HTCondor exporter run directly on the same host.

scrape_configs: - job_name: htcondor scrape_interval: 15s scrape_timeout: 12s static_configs: - targets: ['127.0.0.1:9888'] - job_name: node scrape_interval: 15s static_configs: - targets: ['192.0.2.10:9100'] 

Validate and reload the configuration. This command assumes a systemd unit named prometheus.service. Prometheus configuration and reload

promtool check config /etc/prometheus/prometheus.yml && sudo systemctl kill --kill-whom=main --signal=HUP prometheus.service 

In Grafana, add a Prometheus data source with a URL reachable from the Grafana server. Use http://127.0.0.1:9090 if both services run directly on the same host. Choose Save & test, then run up{job=~"htcondor|node"} in Explore. Expect one series with value 1 per configured target. An absent target and a listed target returning 0 are different conditions. Check Prometheus’s Targets page for the discovered endpoints and their last scrape errors. To save any query below as a panel, add a visualization to a dashboard, select this data source, paste the expression in Code mode, and save the dashboard. Grafana Prometheus data source · Prometheus target health

Count jobs on the schedulers you intend to monitor

Our original htcondor_jobs_running gauge counts running jobs visible to the service account’s condor_q -json on its default schedd. The default is the caller’s jobs, with exceptions for queue superusers and site configuration. If that query fails, the lab code resets both the running count and htcondor_schedd_up to zero.

For a pool view, the lab exporter separately calls condor_q -global -allusers -json, parses its output as one JSON document, and groups the ads by schedd, owner, and job status. A command or JSON parsing error clears those series. The example exporter below queries an explicit list of schedds. If one query fails it omits the whole queue snapshot, so a partial count cannot appear as a pool total. Both approaches count visible jobs currently in the queue. Short jobs can enter and leave between observations. HTCondor 25.0 condor_q

For a Jobs in queue by status time-series panel, use one exporter for the pool. Two exporters observing the same jobs would double the sum.

sum by (job_status) ( htcondor_allusers_jobs_by_owner_status{job="htcondor"} and on (job, instance) (htcondor_schedd_allusers_up{job="htcondor"} == 1) and on (job, instance) (up{job="htcondor"} == 1) ) 

Show htcondor_schedd_allusers_up{job="htcondor"} and up{job="htcondor"} alongside the count. Read them as queue-query health and scrape health, respectively. A valid empty snapshot has no count series. A valid snapshot with no held jobs has no held series. If a health check fails, the filtered count also disappears. Leave gaps visible in Grafana (Connect null values → Never). A stalled loop in the original exporter can leave both the counts and query health at their last values. The direct queue check below helps diagnose that case.

If you want a single Held jobs Stat to show a real zero, supply a zero for an exporter with no held series, then apply both health checks. With one healthy exporter and an empty queue this returns 0. A failed query or scrape leaves the Stat empty.

sum( ( htcondor_allusers_jobs_by_owner_status{job="htcondor", job_status="held"} or on (job, instance) (0 * up{job="htcondor"}) ) and on (job, instance) (htcondor_schedd_allusers_up{job="htcondor"} == 1) and on (job, instance) (up{job="htcondor"} == 1) ) 

On the monitoring host, run these commands as the exporter account. Add schedd="schedd.example.org" to the count metric’s selector in the panel query, using the same advertised name as in the command.

condor_status -schedd -af Name schedd='schedd.example.org' set -o pipefail condor_q -name "$schedd" -allusers -nobatch -af JobStatus | sort | uniq -c 

The left column is the count and the right column is the status code, including 1 for idle, 2 for running, and 5 for held. Counts can change between the command and the scrape. A successful global query alone does not prove every expected schedd contributed. In Part 4’s A → B workflow, B has no job ad until DAGMan submits it. The DAGMan management job does have an ad, so it can contribute to running while A is idle. Use htcondor dag status <dagman-cluster-id> to inspect that particular workflow.

Compare HTCondor CPU allocation with host activity

For a partitionable parent slot, TotalSlotCpus is its reported CPU capacity and Cpus is the amount left in the parent. For one worker, sum the parent capacities and remaining CPUs separately, then calculate 100 × (capacity − remaining) / capacity. Dynamic children have already drawn from these parents. Counting the children again would inflate the totals. The same calculation for RAM uses TotalSlotMemory and Memory in MiB. The denominator is capacity advertised to HTCondor. HTCondor machine ClassAd attributes

In our exporter, ip comes from the worker’s StartdIpAddr, while instance on that series identifies the exporter on port 9888. On Node Exporter series, instance identifies the worker on port 9100. Confirm the address mapping before comparing panels, especially if you scrape by DNS name. The examples assume one worker per advertised IPv4 address.

CPU allocated by HTCondor · Use a Bar gauge or Time series, with Percent (0–100).

100 * ( htcondor_machine_cpu_allocation_ratio{job="htcondor", ip="192.0.2.10"} and on (job, instance) (htcondor_pool_up{job="htcondor"} == 1) and on (job, instance) (up{job="htcondor"} == 1) ) 

Host CPU time outside idle · Use a Time series with the same unit. Compute a rate per logical CPU before averaging, and keep samples only while the worker’s scrape is healthy.

100 * (1 - avg by (job, instance) ( rate(node_cpu_seconds_total{job="node", instance="192.0.2.10:9100", mode="idle"}[5m]) )) and on (job, instance) (up{job="node"} == 1) 

The first value is an allocation snapshot. The second is a five-minute average of non-idle host time, which includes Linux I/O wait. For example, allocating 12 of the 16 CPUs offered to HTCondor gives 75%. On a host with 32 logical CPUs, 12 continuously busy threads would account for 37.5% if the other CPUs remained idle. For host RAM, Node Exporter’s node_memory_MemAvailable_bytes measures the kernel’s estimate of available memory. Prometheus rate function

Worker F’s allocation bar reads 0%. At the right edge of the 24-hour plot, its host CPU trace is near 37%. The panels use different time windows and cannot identify which process accounts for the activity.

If a machine is absent or its allocation looks unexpected, inspect the source ads:

condor_status -constraint 'PartitionableSlot =?= True' \ -af Machine Name StartdIpAddr TotalSlotCpus Cpus TotalSlotMemory Memory 

A worker with only static slots, or with no supported IPv4 address in its parent ads, has no per-machine series in these collectors. The original lab code substitutes zero for missing numeric attributes. Check the raw values and a positive denominator before interpreting a surprising percentage.

Compare slot availability with physical GPU memory

On the worker, query each physical card. The three numeric columns are total, free, used, in MiB.

nvidia-smi --query-gpu=uuid,name,memory.total,memory.free,memory.used \ --format=csv,noheader,nounits 

Our writer multiplies the values by 1,048,576 and writes byte gauges to gpu_vram.prom. Set Grafana’s unit to bytes (IEC) and the legend to {{machine}} · {{gpu_uuid}}. In the Part 2 sharing setup, two advertised GPU resources can point to the same card. To find the card a PyTorch job used, match the worker and physical UUID from that run. A model name alone can identify more than one card. Use the driver’s reported free value directly. NVIDIA can reserve part of the framebuffer for its own use. NVIDIA framebuffer memory fields

The unclaimed slot offers 16 CPUs and 93.5 GiB of RAM, but has no GPU share left to match. The 4.18 GiB reading covers the card as a whole. These panels show the advertisements and driver reading captured by the dashboard. They do not identify which process used the VRAM.

For an actual idle job, run condor_q -better-analyze 123.0 on the submit host, substituting its job ID. That checks the job’s requests and match requirements. VRAM observed in Grafana can change before a job starts.

To inspect the advertised slot inventory directly, set the constraint to the worker’s actual HTCondor machine name.

condor_status -constraint 'Machine == "worker.example.org"' -af Name State Activity Cpus Memory GPUs 

Show the age of the GPU reading

Stopping the GPU timer leaves Node Exporter able to serve the old file. Prometheus can keep reporting up = 1 and attach a new scrape timestamp to the same byte value. Node Exporter exposes the last modification time of a successfully read textfile as a metric value. Graph the file age in seconds:

time() - node_textfile_mtime_seconds{job="node", file=~"(.*/)?gpu_vram[.]prom"} 

For Free VRAM by GPU, use the following expression. It admits real zero-byte readings and drops a worker’s values when the target is down, the timestamp is missing, or the file is 90 seconds old. The lower bound also rejects future-dated files. Keep host clocks synchronized and use one matching file per scrape target.

gpu_vram_free_bytes{job="node"} and on (job, instance) (up{job="node"} == 1) and on (job, instance) ( ((time() - node_textfile_mtime_seconds{job="node", file=~"(.*/)?gpu_vram[.]prom"}) >= 0) < 90 ) 

Both comparisons omit bool, so they remove samples that fail the test. The and then keeps each card’s byte value only if its worker passes. For a current Stat panel, choose Instant and end the dashboard time range at now. For a history panel, choose Range and leave gaps visible. PromQL operators · Grafana query types

The 90-second display threshold is an example for the 30-second timer below, which sets AccuracySec=1s. The lab guide’s timer omits that setting. systemd’s default accuracy window is one minute, so check observed intervals before adopting the same threshold on an existing worker. Part 6 uses a separate 15-minute alert threshold and an inventory of expected workers. A current file can still omit one GPU on a multi-card worker. systemd timer accuracy

Trace a missing reading

curl -fsS http://localhost:9100/metrics | grep -E '^(gpu_vram_free_bytes|node_textfile_mtime_seconds|node_textfile_scrape_error)' systemctl status gpu-vram-export.timer --no-pager journalctl -u gpu-vram-export.service -n 20 --no-pager 

Example collectors for the panel queries

Use these listings if your installation lacks the custom metrics. Keep a single producer for each endpoint or output file. The first collector runs on the monitoring host, the second on each GPU worker.

Save the following as pool_metrics.py. It needs Python 3 and the HTCondor command-line tools. Each scrape reads each named schedd once and builds counts in memory before serving the result. If a schedd query fails, it omits all queue counts for that scrape and serves htcondor_schedd_allusers_up 0. The parent-slot query has its own health gauge. It queries the schedds and then the collector sequentially, with a three-second timeout per command. For two schedds the command time budget is nine seconds, plus overhead. For more schedds, allow more than 3 × (number of schedds + 1) seconds inside the scrape timeout and increase the scrape interval as needed. Prometheus requires the timeout to fit within the interval.

import collections import ipaddress import math import re import subprocess import sys from wsgiref.simple_server import make_server STATUS = {0: "unexpanded", 1: "idle", 2: "running", 3: "removed", 4: "completed", 5: "held", 6: "transferring_output", 7: "suspended"} def rows(*args): out = subprocess.run(args, capture_output=True, text=True, check=True, timeout=3) return [line.strip().split("\t") for line in out.stdout.splitlines() if line.strip()] def label(value): return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") def queue_samples(): counts = collections.Counter() for schedd in schedulers: for owner, status in rows("condor_q", "-name", schedd, "-allusers", "-nobatch", "-af:t", "Owner", "JobStatus"): counts[schedd, owner, STATUS.get(int(status), "unknown")] += 1 return [f'htcondor_allusers_jobs_by_owner_status{{schedd="{label(s)}",' f'owner="{label(o)}",job_status="{k}"}} {n}' for (s, o, k), n in counts.items()] def cpu_samples(): capacity = collections.defaultdict(lambda: [0, 0]) for addr, total, remaining in rows("condor_status", "-constraint", "PartitionableSlot =?= True", "-af:t", "StartdIpAddr", "TotalSlotCpus", "Cpus"): match = re.match(r"<([0-9.]+):", addr) if match: total, remaining = float(total), float(remaining) if not math.isfinite(total) or not (total > 0 and 0 <= remaining <= total): raise ValueError(f"invalid CPU capacity for {addr}") ip = str(ipaddress.IPv4Address(match[1])) pair = capacity[ip] pair[0] += total pair[1] += remaining return [f'htcondor_machine_cpu_allocation_ratio{{ip="{ip}"}} {(total-left)/total}' for ip, (total, left) in capacity.items()] def metrics(env, start_response): if env["PATH_INFO"] != "/metrics": start_response("404 Not Found", [("Content-Type", "text/plain")]) return [b""] lines = [] queries = [ ("htcondor_allusers_jobs_by_owner_status", "htcondor_schedd_allusers_up", queue_samples), ("htcondor_machine_cpu_allocation_ratio", "htcondor_pool_up", cpu_samples), ] for metric, health, collect in queries: try: samples = collect() except (OSError, subprocess.SubprocessError, ValueError) as exc: print(f"{health}: {exc}", file=sys.stderr, flush=True) ok = 0 else: lines.append(f"# TYPE {metric} gauge") lines.extend(samples) ok = 1 lines.extend([f"# TYPE {health} gauge", f"{health} {ok}"]) start_response("200 OK", [("Content-Type", "text/plain; version=0.0.4; charset=utf-8")]) return [("\n".join(lines) + "\n").encode()] if __name__ == "__main__": schedulers = list(dict.fromkeys(sys.argv[1:])) if not schedulers: sys.exit("usage: python3 pool_metrics.py SCHEDD_NAME [SCHEDD_NAME ...]") with make_server("127.0.0.1", 9888, metrics) as server: server.serve_forever() 

On the monitoring host, run the exporter in the foreground under the account that can query the pool. Replace the example name with a name from condor_status -schedd -af Name and add other expected schedds as separate arguments. The exporter binds to loopback.

python3 pool_metrics.py schedd.example.org 

In a second terminal on that host, run curl -fsS http://127.0.0.1:9888/metrics. Look for both health gauges set to 1. A visible running or idle job produces a count. If the queue is empty, the job health gauge can still read 1. A partitionable slot with a supported address supplies a CPU series. Leave the exporter running while trying the panels.

Save the following as gpu_metrics.py. It checks the driver’s exit status and validates the rows it receives before replacing the file. It cannot detect an omitted card unless you compare its output with an expected UUID inventory. A failed call exits nonzero and leaves the last good file in place, so its age increases until a later success. A temporary file in the output directory keeps the rename atomic. Node Exporter reads only completed *.prom files here. Node Exporter textfile collector

import csv import os import socket import subprocess import sys import tempfile from pathlib import Path def label(value): return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") def write_snapshot(path): fields = "uuid,name,memory.total,memory.free,memory.used" result = subprocess.run(["nvidia-smi", "--query-gpu=" + fields, "--format=csv,noheader,nounits"], check=True, capture_output=True, text=True, timeout=10) machine = label(socket.getfqdn()) seen = set() kinds = ("total", "free", "used") samples = {kind: [] for kind in kinds} for row in csv.reader(result.stdout.splitlines(), skipinitialspace=True): uuid, name, *values = [field.strip() for field in row] if not uuid.startswith("GPU-") or uuid in seen or not name or len(values) != 3: raise ValueError(f"invalid or duplicate GPU row: {row}") seen.add(uuid) labels = f'machine="{machine}",gpu_uuid="{label(uuid)}",gpu_name="{label(name)}"' total, free, used = map(int, values) if total <= 0 or not (0 <= free <= total and 0 <= used <= total): raise ValueError(f"invalid memory reading for {uuid}") for kind, mib in zip(kinds, (total, free, used)): samples[kind].append(f'gpu_vram_{kind}_bytes{{{labels}}} {mib * 1024**2}\n') if not seen: raise ValueError("nvidia-smi returned no GPUs") with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", dir=path.parent, delete=False) as tmp: try: for kind, lines in samples.items(): tmp.write(f"# TYPE gpu_vram_{kind}_bytes gauge\n") tmp.writelines(lines) tmp.flush() os.fchmod(tmp.fileno(), 0o644) os.replace(tmp.name, path) finally: if os.path.exists(tmp.name): os.unlink(tmp.name) if __name__ == "__main__": write_snapshot(Path(sys.Source：https://dev.to/nha6ki/monitoring-htcondor-jobs-and-gpu-memory-with-prometheus-and-grafana-40f6
My favorites

Related posts

Remove GPS and EXIF from a JPEG in the browser, without re-encoding it

Every photo from a phone carries a small hidden file inside it: EXIF metadata. Camera model, lens, timestamps, sometimes

0dev.to JavaScript

Where did my SOL go? Finding the rent locked in empty Solana token accounts

If you've traded memecoins on Solana for a few months, your wallet balance is probably a bit lower than your trades say

0dev.to JavaScript

A Google Place ID finder in 10 lines of JS

The problem A local business owner asked me: "How do I send customers a link that opens the Google review box straight a

0dev.to JavaScript

ArtCraft：用 AI 和 Rust 构建开源创意工具生态

不只替代 Photoshop，而是开放工具的修改权 ArtCraft 的目标不只是开发一款 Photoshop 替代品，而是借助 AI 重建一批用户熟悉的软件，并将它们开源。 在 AI Engineer Code Summit 的演讲中，创

0dev.to Rust

Node.js Sending Domain Health Checks With 2 Metrics for Commerce MX Cutovers

Consider a proposed page during an e-commerce mail-provider cutover: the sending domain's published records and the mail

0dev.to Go

Node.js Receipt Trust: Realtime Channels and Durable Message Queues Explained

Use a realtime channel for ephemeral typing indicators and a message queue for read receipts that require notification d

0dev.to Go