When submitting a job to an HPC cluster using Slurm, you will often come across options such as --ntasks, --cpus-per-task, --ntasks-per-node, --mem and --mem-per-cpu.

At first glance, these options might look similar. They all describe resources required by a job, but each controls something different.

Understanding these options is important because requesting the wrong resources can lead to inefficient resource allocation, poor application performance or jobs waiting unnecessarily in the queue.

Let's understand the concepts behind these options, starting with the basics.

1. First, understand the difference between a node, CPU and task

Before exploring the Slurm options, we need to understand three basic concepts.

Node: A node is simply a computer within an HPC cluster. It has its own CPU cores, memory and other hardware resources.

CPU core: Think of a CPU core as a worker that can execute instructions. If a compute node has 32 CPU cores, it has 32 physical processing cores available.

Task: A task is an individual unit of work launched by Slurm. In MPI applications, a task commonly represents an MPI process.

For example, if you run an MPI application with 8 processes, you have 8 tasks. Each process works on its own portion of the calculation and communicates with other processes when necessary.

A task is not necessarily the same as a CPU core. One task can use one CPU core or multiple CPU cores.

This distinction is the foundation for understanding Slurm resource requests.

2. Understanding CPU allocation options

--ntasks: How many tasks do you need?

The --ntasks option tells Slurm how many tasks your application needs in total.

For example:

#SBATCH --ntasks=8 

This requests 8 tasks.

If you are running an MPI application, this generally means 8 MPI processes.

Slurm can distribute these tasks across one or multiple compute nodes, depending on the available resources and other job requirements.

For example:

  • Node 1: 4 tasks
  • Node 2: 4 tasks

The total is still 8 tasks.

Remember: --ntasks defines the total number of tasks, not the number of CPU cores.

--cpus-per-task: How many CPU cores does each task need?

Now imagine that one task needs more than one CPU core to perform its work.

This is common with applications using OpenMP or other multithreaded programming models.

The --cpus-per-task option tells Slurm how many CPUs to allocate to each task.

For example:

#SBATCH --ntasks=1 #SBATCH --cpus-per-task=8 

This requests:

  • 1 task
  • 8 CPUs for that task

In an OpenMP application, one process can create multiple threads to perform calculations in parallel.

You can configure the number of OpenMP threads using:

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 

Remember: --cpus-per-task controls the CPU allocation for each task, not the total number of tasks.

--ntasks-per-node: How should tasks be distributed?

What if you want to control how many tasks run on each compute node?

This is where --ntasks-per-node becomes useful.

For example:

#SBATCH --ntasks=16 #SBATCH --ntasks-per-node=8 

You are requesting:

  • 16 tasks in total
  • Maximum 8 tasks per node

A possible distribution is:

  • Node 1: 8 tasks
  • Node 2: 8 tasks

Another example:

#SBATCH --ntasks=12 #SBATCH --ntasks-per-node=4 

Slurm can distribute the tasks across three nodes, with 4 tasks on each.

Remember: --ntasks defines the total number of tasks, while --ntasks-per-node controls the maximum number of tasks placed on each node.

3. Understanding memory allocation options

CPU resources are only one part of an HPC job. Applications also need memory to store data, intermediate calculations and other information while running.

Slurm provides two commonly used options for requesting memory.

--mem: How much memory does each node need?

The --mem option specifies the amount of memory required per node.

For example:

#SBATCH --nodes=2 #SBATCH --mem=32G 

This requests 32 GB of memory per node.

Since the job requests two nodes, the total requested memory across the allocation is 64 GB.

Why per node?

Because each compute node has its own physical RAM. Memory on one node is not automatically available to applications running on another node.

Remember: --mem specifies memory per node, not total memory across the job.

--mem-per-cpu: How much memory does each allocated CPU need?

Instead of requesting a fixed amount of memory per node, you can request memory based on the number of allocated CPUs.

This is what --mem-per-cpu does.

For example:

#SBATCH --ntasks=8 #SBATCH --cpus-per-task=2 #SBATCH --mem-per-cpu=4G 

Let's calculate the request.

Total allocated CPUs:

8 tasks × 2 CPUs = 16 CPUs.

Memory request:

16 CPUs × 4 GB = 64 GB.

If the tasks are distributed evenly across two nodes, each node receives 8 CPUs and 32 GB of requested memory.

This approach can be useful when memory requirements generally increase with the number of CPU resources allocated.

Remember: --mem-per-cpu specifies memory per allocated CPU, whereas --mem specifies memory per node.

An important detail: --mem, --mem-per-cpu and --mem-per-gpu are alternative memory request methods. You should not combine them in the same job.

Also, Slurm has a configurable maximum memory-per-CPU value. If a request exceeds that limit, Slurm may adjust the effective memory-per-CPU and CPU allocation. Therefore, the actual request can depend on cluster configuration.

4. How do these options work together?

Let's imagine an HPC cluster where each compute node has:

  • 32 CPU cores
  • 128 GB RAM

You are running an MPI application that requires 16 processes, with 2 CPU cores per process.

Your job script could look like this:

#!/bin/bash #SBATCH --job-name=hpc-calculation #SBATCH --nodes=2 #SBATCH --ntasks=16 #SBATCH --ntasks-per-node=8 #SBATCH --cpus-per-task=2 #SBATCH --mem-per-cpu=4G #SBATCH --time=02:00:00 srun ./my_mpi_application 

Let's break down the allocation.

Number of nodes: 2

Total tasks: 16

Tasks per node: 8

CPUs per task: 2

Total CPU allocation: 16 × 2 = 32 CPUs

Memory per CPU: 4 GB

Total memory request: 32 × 4 = 128 GB

Each node receives:

  • 8 tasks
  • 16 CPUs
  • 64 GB memory

This is how different Slurm options work together to describe the resources required by an application.

5. What does the Slurm administrator need to configure?

Here is something that users often overlook.

Slurm resource requests do not work independently of the cluster configuration. The administrator determines how resources are tracked, scheduled and enforced.

Let's look at some important configuration parameters in slurm.conf and cgroup.conf.

CPU and memory scheduling

Consider the following configuration:

SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory 

SelectType=select/cons_tres enables consumable resource scheduling.

SelectTypeParameters=CR_Core_Memory tells Slurm to track consumable CPU cores and memory.

Other configurations include CR_Core, CR_CPU and CR_CPU_Memory, depending on how the administrator wants Slurm to track resources.

For memory aware scheduling, a *_Memory configuration is particularly important.

CPU and memory enforcement

Scheduling resources and enforcing resource limits are two different things.

For example, Slurm might reserve memory for a job, but that does not automatically mean the application is prevented from exceeding its allocation.

Administrators can configure:

In slurm.conf:

TaskPlugin=task/cgroup,task/affinity 

In cgroup.conf:

ConstrainCores=yes ConstrainRAMSpace=yes 

These settings enable cgroup based CPU and memory constraints.

In simple terms:

  • ConstrainCores=yes restricts jobs to their allocated CPU cores.
  • ConstrainRAMSpace=yes enables memory enforcement through cgroups.

This helps prevent jobs from interfering with other workloads running on the same compute node.

Default memory allocation

Does a user always need to specify --mem or --mem-per-cpu?

No. Administrators can define default memory allocations.

For example:

DefMemPerCPU=4000 

This defines a default memory allocation of 4000 MiB per usable allocated CPU.

Alternatively:

DefMemPerNode=64000 

This defines a default memory allocation of 64000 MiB per node.

These are alternative configuration settings.

Administrators can also define maximum memory limits using MaxMemPerCPU or MaxMemPerNode.

The important point is that the availability and behaviour of memory requests depend partly on how the administrator has configured Slurm.

6. How can you check the configuration?

If you are an HPC administrator or simply want to understand how your cluster is configured, you can inspect some of these settings.

To check the main Slurm configuration:

scontrol show config 

To check the resources available on a compute node:

scontrol show node compute01 

To inspect cgroup configuration:

cat /etc/slurm/cgroup.conf 

The configuration file location may differ depending on the installation.

7. Final thoughts

Understanding Slurm resource options is not just about memorising command line parameters. It is about understanding how your application uses the underlying HPC infrastructure.

An MPI application might need multiple tasks distributed across several nodes. An OpenMP application might need one task with multiple CPU cores. Some applications need a fixed amount of memory per node, while others have memory requirements that scale with CPU allocation.

There are also two sides to resource management: what the user requests and what the cluster administrator configures to schedule and enforce those requests.

Before submitting your next HPC job, think about three questions:

  1. How many tasks or processes does my application need?
  2. How many CPU cores does each task require?
  3. How much memory does the application need per node or per allocated CPU?

Once you understand these concepts, Slurm resource requests become much easier to understand and use effectively.