Running multiple HTCondor jobs becomes more complicated when one task depends on the results of another. A preprocessing job may need to finish before analysis can begin, and a final aggregation job may need to wait for several analyses to succeed.

HTCondor's Directed Acyclic Graph Manager (DAGMan) lets you define these dependencies in a plain-text DAG file. This guide covers job dependencies, parameterized jobs, concurrency limits, PRE/POST scripts, and recovery after failures.

The examples assume an existing HTCondor pool and working submit descriptions. They are illustrative and have not been newly executed or benchmarked for this article.

1. What Is a DAG File?

A directed acyclic graph (DAG) consists of nodes connected by directed edges, with no cycles.

In an HTCondor DAG:

  • A node is a unit of work associated with an HTCondor submit description. A node may submit one or more jobs, depending on its queue statement.
  • An edge represents a dependency between nodes.
  • A DAG file (usually ending in .dag) declares nodes with JOB and dependencies with PARENT and CHILD.

DAGMan submits the jobs whose dependencies have been satisfied and tracks their progress. You can submit a DAG with:

condor_submit_dag my.dag

2. Minimal Example: A → B

Consider two stages: task B must start only after task A has completed successfully.

Directory layout 

project/ ├── A.submit ├── B.submit └── my.dag

DAG file: my.dag 

JOB A A.submit JOB B B.submit PARENT A CHILD B

JOB A A.submit defines a node named A using A.submit. The PARENT line makes A a prerequisite for B. By default, a child becomes eligible to run only when all its parents have succeeded.

For a minimal Linux example, the submit descriptions could be:

A.submit

executable = /bin/echo arguments = "Task A" output = A.out error = A.err log = A.log queue

B.submit

executable = /bin/echo arguments = "Task B" output = B.out error = B.err log = B.log queue

These example executables assume a compatible Linux execution environment. Submit the workflow from the directory containing my.dag:

cd project condor_submit_dag my.dag

DAGMan will submit node B after node A succeeds. Actual scheduling and start times depend on the available HTCondor resources.

3. Defining Nodes with JOB

The basic syntax is:

JOB NodeName SubmitFile

For example:

JOB Preprocess preprocess.submit JOB Train train.submit JOB Eval eval.submit

Node names must be unique within a DAG. Simple names containing letters, digits, and underscores are convenient. The node name does not need to match the submit file name.

A submit file can contain queue 1, queue 10, or another queue specification. Consequently, a DAG node should not always be interpreted as exactly one HTCondor job.

4. Expressing Dependencies with PARENT and CHILD

The basic dependency syntax is:

PARENT ParentNodeName CHILD ChildNodeName

Multiple parents and children can be declared on the same line:

PARENT A CHILD B C PARENT B C CHILD D

This describes a diamond-shaped DAG:

 A / \ B C \ / D

The complete DAG file is:

JOB A A.submit JOB B B.submit JOB C C.submit JOB D D.submit PARENT A CHILD B C PARENT B C CHILD D

After A succeeds, B and C become eligible to run independently. D becomes eligible only after both B and C succeed. Whether B and C actually run simultaneously depends on scheduler capacity and any configured limits.

This pattern is useful for preprocessing followed by parallel analyses and a final aggregation step.

5. Passing Per-Node Parameters with VARS

You can reuse a single submit description across multiple DAG nodes and change its parameters using VARS.

DAG file: runs.dag 

JOB RUN1 run_job.submit JOB RUN2 run_job.submit VARS RUN1 idx="1" alpha="0.1" VARS RUN2 idx="2" alpha="0.5"

Submit description: run_job.submit 

executable = /usr/bin/python3 transfer_executable = False arguments = train.py --idx $(idx) --alpha $(alpha) should_transfer_files = YES transfer_input_files = train.py when_to_transfer_output = ON_EXIT log = logs/run_$(idx).log output = logs/run_$(idx).out error = logs/run_$(idx).err queue

DAGMan supplies the values of idx and alpha when submitting each node. The submit description expands $(idx) and $(alpha) into the corresponding arguments and log-file names.

Before submitting, create the log directory:

mkdir -p logs

This example assumes that train.py exists and accepts --idx and --alpha, and that /usr/bin/python3 is available on the execution machines. transfer_input_files makes the script available in the remote job directory, while transfer_executable = False avoids transferring the system Python interpreter itself. Adjust file-transfer settings to match your cluster's shared-filesystem and software environment.

6. Limiting Submitted Nodes with CATEGORY and MAXJOBS

Suppose three DAG nodes run GPU-intensive workloads, but you want DAGMan to have at most one of those nodes submitted at any given time.

JOB G1 gpu_job1.submit JOB G2 gpu_job2.submit JOB G3 gpu_job3.submit CATEGORY G1 GPUJOB CATEGORY G2 GPUJOB CATEGORY G3 GPUJOB MAXJOBS GPUJOB 1

CATEGORY assigns nodes to a named category. MAXJOBS limits how many nodes in that category may have job clusters submitted to HTCondor simultaneously. A submitted node can be idle or running; it occupies a category slot until its jobs finish.

Important: This is a per-DAG submission throttle, not a cluster-wide GPU allocation policy. If a node's submit description queues multiple jobs, MAXJOBS GPUJOB 1 does not necessarily mean that only one GPU job runs. To request GPU resources from HTCondor, use appropriate GPU resource requests (such as request_gpus) in the submit descriptions and follow your pool's scheduling policy.

The same mechanism can limit other categories, such as preprocessing nodes, without changing their dependencies.

7. Running PRE and POST Scripts

DAGMan can invoke scripts before or after a node's jobs:

JOB T1 train.submit SCRIPT PRE T1 ./pre_check.sh $NODE SCRIPT POST T1 ./post_cleanup.sh $NODE $RETURN

Here, $NODE is a DAGMan script macro for the node name, and $RETURN is available to POST scripts to report the node's job return status. The script files must exist and be executable. They are not included in this illustrative example.

  • A PRE script runs before DAGMan submits the node's jobs. If it fails, the jobs are not submitted, and the node fails by default.
  • A POST script runs after the node's submitted jobs have finished. Its exit status participates in determining whether the node succeeds. In particular, a successful POST script can cause a node to be considered successful even when a job failed, so handle failures deliberately.
  • PRE and POST scripts run on the access point, not necessarily on the machine executing the node's jobs. They are best kept lightweight.

Typical uses include validating input files, checking generated outputs, and cleaning up temporary files.

8. Submitting and Monitoring a DAG

Submit the DAG:

condor_submit_dag my.dag

This submits a DAGMan management job in the scheduler universe. DAGMan then submits the jobs defined by the nodes as their dependencies allow.

Useful monitoring commands include:

# View jobs in the queue condor_q # Show individual jobs rather than the default batch summary condor_q -nobatch # View jobs belonging to a user condor_q <username> # Inspect progress using the DAGMan job ID htcondor dag status <dagman-job-id>

The last command requires an HTCondor release that provides the htcondor dag interface. Replace the placeholders with your actual user name and DAGMan job ID.

To temporarily pause DAGMan from managing and submitting additional work, you can hold its management job and later release it:

condor_hold <dagman-job-id> condor_release <dagman-job-id>

Alternatively, the newer DAG interface provides:

htcondor dag halt <dagman-job-id> htcondor dag resume <dagman-job-id>

These are not identical operations: holding DAGMan stops its management process from making progress while already-submitted jobs continue; halting allows active work and POST scripts to finish, then writes recovery state and exits. Consult the HTCondor manual before using either on a production workflow.

9. Recovering from Failures with a Rescue DAG

If one or more nodes fail, DAGMan may continue independent work before it exits. On failure, it writes a Rescue DAG recording progress so completed nodes do not need to be rerun.

A typical rescue file name is:

my.dag.rescue001

After correcting the underlying problem, resubmit the original DAG:

condor_submit_dag my.dag

By default, DAGMan detects the most recent applicable Rescue DAG and uses it to resume the workflow, skipping nodes already marked complete. Review the failed node's logs and the DAGMan debug output before resubmitting.

10. Reusable Workflow Templates

A sequential workflow: A → B → C 

JOB A A.submit JOB B B.submit JOB C C.submit PARENT A CHILD B PARENT B CHILD C

Preprocessing → parallel analyses → aggregation 

JOB PRE preprocess.submit JOB RUN1 run.submit JOB RUN2 run.submit JOB RUN3 run.submit JOB RUN4 run.submit VARS RUN1 idx="1" VARS RUN2 idx="2" VARS RUN3 idx="3" VARS RUN4 idx="4" JOB MERGE merge.submit PARENT PRE CHILD RUN1 RUN2 RUN3 RUN4 PARENT RUN1 RUN2 RUN3 RUN4 CHILD MERGE

All four RUN nodes depend on successful preprocessing. MERGE depends on all four runs succeeding.

Allow at most one submitted GPU-workload node in this DAG 

JOB G1 gpu_job.submit JOB G2 gpu_job.submit JOB G3 gpu_job.submit VARS G1 idx="1" VARS G2 idx="2" VARS G3 idx="3" CATEGORY G1 GPUJOB CATEGORY G2 GPUJOB CATEGORY G3 GPUJOB MAXJOBS GPUJOB 1

This last example assumes gpu_job.submit uses $(idx) if that variable is needed. As explained above, the throttle limits submitted DAG nodes, not the total number of GPU processes across the pool.

11. Practical Considerations

  • Define job requirements in .submit files. DAG files express dependencies and workflow-level controls; CPU, memory, GPU, and file-transfer requirements belong in the underlying submit descriptions.
  • Check paths and permissions. Referenced submit files, scripts, input data, and output directories must be accessible in the appropriate environments.
  • Distinguish job failure from DAG failure. A failed node may block downstream nodes while independent branches finish.
  • Treat the examples as templates. Replace placeholder programs and script names with real implementations and verify behavior against your installed HTCondor version.

Conclusion

DAGMan turns independent HTCondor submissions into dependency-aware workflows. Start with JOB and PARENT/CHILD, use VARS to parameterize repeated tasks, and apply CATEGORY/MAXJOBS when you need per-DAG submission limits. PRE/POST scripts and Rescue DAGs provide additional control for validation, cleanup, and failure recovery.

References

AI Disclosure: An LLM assisted with translating, organizing, and editing the English text. The examples are illustrative; no new execution tests or benchmark results are claimed.