Build an HTCondor Pool on Ubuntu and Run Your First Remote Job

Part 1 of 6 in Building and Operating an HTCondor Pool for Research Computing.

Suppose you have one training run per participant in a study and several workstations that could run them. With HTCondor, you can submit each run as a job and specify the CPU, memory, and GPU it needs. The workstations advertise what they can offer. HTCondor keeps each job in a queue until it can match it to a suitable machine. The machines that cooperate this way form a pool.

I administer the central manager for our lab’s pool. Our November 2026 inventory lists twelve machines. The workstation that hosts the central manager also accepts submissions and runs jobs. Some of the other workstations only run jobs, while others also accept submissions.

For this walkthrough, we’ll put each role on a separate Ubuntu 24.04 VM. You’ll submit a shell script and a text file from the access point (the submit node). The script will run on the worker and print its hostname and a line from the file. You’ll read both lines back on the access point.

Give each machine a role

Example hostname	HTCondor role	What happens there
`manager.example.org`	Central manager	The collector receives descriptions of available resources. The negotiator matches queued work to those resources.
`submit.example.org`	Access point (submit node)	You log in here. The schedd keeps your job in its queue and arranges the transfer of input and output.
`worker.example.org`	Execution point (execute node)	The startd advertises resources as slots. A slot is a portion of the machine’s resources available for a job. The worker runs your program.

Start with three fresh VMs on a private network. Each needs Bash, curl, sudo, and access to the Ubuntu and HTCondor package repositories. Run uname -m on the access point and worker and check that the results match. The submit file below uses HTCondor’s default architecture requirement, which matches the architecture of the access point even when the executable is a shell script.

Allow TCP port 9618 between pool members. Before installing, check name resolution on each host. Substitute your own hostnames below and confirm that each resolves to its intended private address.

getent hosts manager.example.org submit.example.org worker.example.org

Install HTCondor on the three hosts

Start on the manager, then install the access point, then the worker. Enter the same pool password on all three. On each host, the installer uses that shared secret to generate a daemon token. The daemons use these tokens to authenticate to one another with HTCondor’s IDTOKENS method. Researchers submit jobs from their own local accounts on the access point.

On each host, download the installer into your home directory. Set manager to the same real manager name everywhere. In a Bash shell on each host, first run the one assignment from its row, then run the common commands below. The installer takes the manager’s name even on the worker.

Host you are logged into	Set this variable
Manager	`role=--central-manager`
Access point	`role=--submit`
Worker	`role=--execute`

cd ~ curl -fsSL https://get.htcondor.org -o get-htcondor.sh manager=manager.example.org bash ./get-htcondor.sh --dry-run --channel 25.0 \ "${role:?Set role using the table above}" "$manager"

The dry run shows the proposed installation commands. The 25.0 channel selects the HTCondor 25.0 LTS release series. Once the role and manager name look right, run the following in the same Bash session on that host. The silent prompt keeps the pool password out of your typed command history. The installer reads it from standard input.

sudo -v ( IFS= read -r -s -p 'Pool password: ' pool_password || exit 1 printf '\n' [[ -n "$pool_password" ]] || exit 1 sudo -n /bin/bash ./get-htcondor.sh --no-dry-run \ --channel 25.0 "${role:?Set role using the table above}" "$manager" <<<"$pool_password" )

After the installer succeeds on each machine, check the installed version, the manager it points to, and the local service:

condor_version condor_config_val CONDOR_HOST systemctl is-active condor

Once the manager’s service is active, test its TCP port from both the access point and the worker. Use the manager variable you set on each host. On Ubuntu, nc is available in the netcat-openbsd package (sudo apt install netcat-openbsd if needed).

nc -vz -w 3 "$manager" 9618

If the connection times out or is refused, check the manager’s service, the resolved address, and the firewall rules. A successful connection establishes TCP reachability. To check whether the daemons have also registered with the collector, run the following on the access point after all three services have started. Allow time for their first advertisements.

condor_status -master -af Name condor_status -schedd -af Name condor_status -af Machine Name State Activity Cpus Memory

For this layout, look for all three hosts in the first query, the access point in the second, and at least one slot on the worker in the third. A master entry tells you that the host’s managing daemon advertised itself. The slot entry tells you what that worker is offering. A machine can appear in the first list while offering no slots.

Send a file to the worker and get its contents back

mkdir -p ~/first-condor-job/log cd ~/first-condor-job printf '%s\n' 'sample-001' > marker.txt cat > probe.sh <<'SH' #!/bin/sh set -eu printf 'Execution host: ' hostname printf 'Input received: ' cat marker.txt SH chmod +x probe.sh

In that directory, save this as first.sub. HTCondor will transfer the executable and the specified input file, so the worker can run the test without a shared home directory, NAS mount, or Python environment.

universe = vanilla executable = probe.sh transfer_input_files = marker.txt should_transfer_files = YES when_to_transfer_output = ON_EXIT request_cpus = 1 request_memory = 128 MB output = log/first.$(ClusterId).$(ProcId).out error = log/first.$(ClusterId).$(ProcId).err log = log/first.$(ClusterId).log queue 1

Submit the job and note the cluster number printed by condor_submit. A quick job may already have finished by the time you inspect the queue.

condor_submit first.sub condor_q -nobatch

Set cluster to the number printed at submission. The submit file queues one process, so its job ID ends in .0. Wait up to three minutes for a completion or removal event.

cluster=123 # Replace 123 with your cluster number. job="$cluster.0" event_log="log/first.$cluster.log" condor_wait -wait 180 "$event_log" "$job"

If the wait times out or reports an error, use the troubleshooting steps below. A zero exit status from condor_wait can also mean the job was removed. Check the job’s own exit status and read its standard output on the access point.

condor_history "$job" -af JobStatus ExitBySignal ExitCode cat "log/first.$job.out"

For a successful run, expect 4 false 0 in the history. These values mean the job completed, exited normally, and returned exit code zero. The output file should look like this.

Execution host: worker Input received: sample-001

The first line should match the output of hostname on your worker. The second shows that the script read the transferred file. Reading both lines on the access point confirms that stdout made the return trip.

Troubleshoot the first job

If the wait times out, the job may still be queued or running. On the access point, check its state and recent events.

condor_q "$job" -af JobStatus HoldReason tail -n 20 "$event_log"

JobStatus 1 means Idle, 2 means Running, and 5 means Held. For a running job, repeat the wait with the same ID. If the queue query returns no row, check the history again. If the event log is missing, check the cluster number and make sure you are in ~/first-condor-job.

If you see this	What to check
The worker is missing from the collector	On the worker, check `condor_config_val CONDOR_HOST` and `systemctl status condor`. Verify name resolution and TCP 9618 to the manager. Find the daemon log directory with `condor_config_val LOG` and inspect `MasterLog` and `StartLog` for startup or authentication errors.
The job is still Idle when the wait times out	On the access point, run `condor_q -better-analyze "$job"`. Compare the reported requirements with the worker’s `Arch`, available resources, and start policy. Use `condor_status -af Machine Arch Cpus Memory` to see what it advertises.
The job is Held	Read the hold reason. For a missing file or permission error, check that the submitting account can read the input file and write to the output directory. For a connection error during file transfer, check connectivity between the access point and the worker. The earlier `nc` test checked the path to the manager.
The job ran but the output is incomplete	Read `log/first.$job.err`, the event log, and the exit status. An error from `cat marker.txt` places the failure inside the running job.

How I add a workstation to the lab pool

For workstations joining our existing lab pool, I issue the daemon token centrally. A colleague sends me the workstation’s network address. I create the token on the central manager with condor_token_create and give it to the colleague. They configure the workstation with our manager and trust domain and install the token under /etc/condor/tokens.d/ with restricted permissions.

Our central manager is configured with use ROLE : CentralManager, Submit, Execute. For a workstation that will both accept submissions and run jobs, our setup instructions specify use ROLE : Submit, Execute. I look for its schedd with condor_status -schedd and its execution slots with condor_status. Submitting a job from the colleague’s local account checks that the workstation’s schedd accepts jobs from that user. To see which workstation actually ran the job, check the hostname printed by the probe.

People also use these workstations at their desks. Our setup instructions use START = (Cpus > 0) && (LoadAvg / Cpus < 0.4) to decide when a slot can accept a new job. A workstation can remain visible in the pool while this condition prevents a new job from starting. On a partitionable slot, Cpus counts the CPUs still available. As jobs claim CPUs, the denominator gets smaller, so even an unchanged LoadAvg can bring the ratio up to the cutoff.

Optional GPU sharing on an NVIDIA worker

After the CPU test succeeds, you can configure HTCondor to let two jobs share one NVIDIA GPU. On the idle worker, check that nvidia-smi -L lists the expected card, then apply the settings below.

sudo tee /etc/condor/config.d/90-lab-gpu.config >/dev/null <<'CONF' use feature : GPUs GPU_DISCOVERY_EXTRA = -properties -divide 2 CONF sudo systemctl restart condor

-divide 2 makes condor_gpu_discovery list the same device ID twice in DetectedGPUs and report half its usual GlobalMemoryMb value for matching. Two jobs that each request one GPU can therefore be assigned the same card. They share its VRAM, and either job can use more than the advertised half if enough memory is free. Once the worker advertises again, run condor_status -gpus on the access point to inspect the pool’s GPU resources.

Where to go after the first job

Run PyTorch on an NVIDIA worker (Part 2). Once the worker advertises a GPU, submit a small matrix multiplication and verify the result from the access point.
Run a compiled MATLAB application on a CPU worker (Part 3). Start from the successful CPU test. Part 3 adds the build machine and MATLAB Runtime.
Run jobs in dependency order (Part 4). Use DAGMan to make a second job eligible to run only after the first succeeds. You can try the first example on your CPU pool. Writing HTCondor DAG Files