Running PyTorch Jobs on Shared GPUs with HTCondor

Part 2 of 6 in Building and Operating an HTCondor Pool for Research Computing.

Our November 2026 inventory included a GTX 1080 with 8 GiB of memory and a TITAN RTX with 24 GiB. With our sharing configuration, HTCondor can assign two jobs to the same physical GPU. We’ll submit one PyTorch job, inspect which GPU it sees, and check the result of a small matrix multiplication.

Start with a pool that can return the hostname from a remote shell job, as in Part 1. The machine you submit from can have a different Python installation from the worker. The wrapper below records which worker HTCondor selected, and the Python script records which interpreter that worker started.

What one GPU request means in our pool

Our lab’s setup instructions enable GPU sharing on NVIDIA workers with the two settings below. If you followed the optional GPU setup in Part 1, you can go straight to checking what the worker advertises.

use feature : GPUs GPU_DISCOVERY_EXTRA = -properties -divide 2 

With -divide 2, HTCondor lists each detected GPU ID twice and halves its advertised GlobalMemoryMb for matching. For example, a device that discovery reports as 24,576 MiB would advertise 12,288 MiB per share. Two jobs assigned the same card draw from the same VRAM. This setting alone does not enforce a per-job memory limit.

On the access point, run condor_status -gpus -compact to see the advertised model names. On a worker, use the following command to see the physical devices and their full UUIDs. Keep the output with the worker’s hostname so you can look up the driver version for the machine that runs your test.

nvidia-smi --query-gpu=index,uuid,name,driver_version --format=csv 

Choose a PyTorch build for the workers

Our lab’s training tutorial uses PyTorch 2.4.1 with CUDA 12.1 and runs a transferred script with Python on shared storage. This article uses the same shared-interpreter setup for a smaller test that exits if CUDA is unavailable.

The GTX 1080 is a Pascal GPU with compute capability 6.1. The TITAN RTX is a Turing GPU with compute capability 7.5. For Linux x86-64, the official PyTorch 2.8.0 CUDA 12.6 wheel includes support for both architectures. The CUDA 12.8 wheel from the same release dropped Pascal support. The 12.6 wheel is therefore a candidate to test on both cards. Each worker still needs a compatible NVIDIA driver. PyTorch 2.8 architecture support · Official wheel installation commands

Before submitting, prepare a dedicated Python environment that every eligible worker can run at the same absolute path. The example uses /shared/envs/pytorch/bin/python. Substitute your environment’s path in both the install command and the job wrapper. If you already have a suitable PyTorch build in that environment, skip the installation step.

/shared/envs/pytorch/bin/python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu126 

Run a small job that requires CUDA

On the access point, save this as check_gpu.py in your job directory.

import os import sys print("Interpreter:", sys.executable, flush=True) print("CUDA_VISIBLE_DEVICES:", os.environ.get("CUDA_VISIBLE_DEVICES", "<unset>"), flush=True) import torch print("PyTorch:", torch.__version__, flush=True) print("CUDA version used to build PyTorch:", torch.version.cuda, flush=True) if not torch.cuda.is_available(): raise SystemExit("CUDA is unavailable to this PyTorch process") print("Device: cuda:0", torch.cuda.get_device_name(0), flush=True) print("Compute capability:", torch.cuda.get_device_capability(0), flush=True) x = torch.ones((128, 128), device="cuda:0") result = (x @ x)[0, 0].item() print("Matrix product [0, 0]:", result) if result != 128.0: raise SystemExit(f"Expected 128.0, got {result}") 

Save this wrapper as run_gpu.sh. Replace the example path with the absolute path to your environment’s Python on the worker.

#!/bin/sh printf 'Execution host: %s\n' "$(hostname)" exec /shared/envs/pytorch/bin/python check_gpu.py 

Save gpu.submit in the same directory.

universe = vanilla executable = run_gpu.sh should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = check_gpu.py request_cpus = 1 request_memory = 2 GB request_gpus = 1 output = log/gpu.$(ClusterId).$(ProcId).out error = log/gpu.$(ClusterId).$(ProcId).err log = log/gpu.$(ClusterId).log queue 1 

HTCondor transfers the wrapper and check_gpu.py. The wrapper starts the Python interpreter already available to the worker. Here request_memory requests host RAM, and request_gpus = 1 requests one advertised GPU resource.

To test a specific card model, copy its advertised name from condor_status -gpus -compact and add a require_gpus line before queue 1. For example, if the worker advertises the name shown here, use:

require_gpus = (DeviceName == "NVIDIA GeForce GTX 1080") 

A capability test alone would also allow the lab’s TITAN X (Pascal), which has the same compute capability as the GTX 1080. Submit once for the GTX 1080, then replace the name with the TITAN RTX’s advertised name and submit again. Keep the cluster ID for each run and check the model PyTorch prints before recording a pass for that card. For routine jobs, gpus_minimum_capability sets a lower bound if all eligible models are supported by your build. HTCondor GPU submit options

mkdir -p log chmod +x run_gpu.sh condor_submit gpu.submit condor_q -nobatch 

Verify the CUDA calculation

From the submission directory on the access point, replace 123 with the cluster ID for the run you want to check. The submit file queues one process, so its job ID ends in .0. Wait up to three minutes for a completion or removal event.

cluster=123 job="$cluster.0" event_log="log/gpu.$cluster.log" condor_wait -wait 180 "$event_log" "$job" 

If the wait times out or reports an error, use the troubleshooting table below. A zero exit status from condor_wait can also mean the job was removed. Check the job’s own exit status and read its output.

condor_history "$job" -af JobStatus ExitBySignal ExitCode cat "log/gpu.$job.out" 

For a successful run, look for both of the following:

• The history shows 4 false 0. The job completed, exited normally, and returned exit code zero.
• Stdout shows Matrix product [0, 0]: 128.0. Each element sums 128 products of ones. Calling .item() reads the selected element back from the GPU as a Python number.

To identify the card, start with Execution host and CUDA_VISIBLE_DEVICES in stdout. HTCondor sets the latter from the job’s assignment. Inside the process, CUDA numbers visible devices starting at zero, so cuda:0 refers to the first device visible to this job. If HTCondor supplied a full GPU UUID or a unique prefix, match it against the UUIDs from nvidia-smi on that worker. A numeric value alone does not give you a physical UUID. HTCondor job environment · CUDA device enumeration

Record the worker’s driver version alongside the PyTorch version, build CUDA version, model and compute capability for each successful run. The 128.0 result verifies this calculation for one job. Demonstrating simultaneous assignment would require two jobs on the same worker with the same physical GPU UUID and overlapping execution times.

Part 3 runs a compiled MATLAB application on a CPU worker. To investigate GPU contention in Part 5, use the worker name and the full GPU UUID once you’ve matched it to the worker’s inventory. The VRAM time series shows free memory across the card at each sample, including the effects of other processes. With the example’s 30-second collection interval, a short job can start and finish between samples.

References

• HTCondor GPU discovery options
• NVIDIA compute capabilities for current and legacy GPUs · Legacy GPU table
• HTCondor job ClassAd attributes

AI assistance was used in drafting and editing this article.