This documentation will only cover site specific topics. Please, use the web to find solutions and/or read the SLURM documentation:

SLURM Documentation

About -p partition

Note that we don't set a default partition - so the -p partition is needed. To get a list of partitions you have access to, you can use:

sacctmgr show associations User=$(whoami)

Short SLURM Introduction

There are basically three methods of running a SLURM job; using srun, sbatch and salloc. Running a simple job is easy using srun:

$ srun -p partition hostname

Running it on two nodes:

$ srun -p partition --nodes=2 hostname

Two task on the same node:

$ srun -p partition --ntasks=2 hostname

Note that running jobs with srun blocks the shell and only returns once the job has finished. This is usually not the intended behavior, so let's take a look at using sbatch.

The following simple sequential job sets the job name, number of tasks and memory per task (--mem-per-cpu). The batch job prints all SLURM environment variables describing the allocation. It also prints out our site-specific SCRATCH variable which always points to the local scratch area on the allocated node.

#SBATCH --job-name=slurmtest
#SBATCH --ntasks=1
#SBATCH --mem=1G

echo "-----------------"
env | grep SLURM_ | sort | while read var; do
    echo " * $var"

echo "-----------------"

# Two job "steps"
srun hostname
srun sleep 10

The job must be submitted using the sbatch command like:

sbatch -p partition name-of-script.sh

If you have submit access to multiple partitions, you specify the correct one using the -p<partition>. If you have multiple “bank accounts”, you can specify the correct one using -A <account>. Both options can go in the script as well:

#SBATCH --partition=name-of-partition
#SBATCH --account=name-of-account

Note that SLURM distinguishes between “jobs” and “job steps”. Each srun command inside a job script denotes a “step” and is subject to the various job arguments such as “–ntasks” and “–nodes”. All commands not run through srun will be executed only once on the first allocated node (the node where the batch script is running).

Accounting is done for each step. This can be inspected at run-time using sstat <jobid>. To check up on your jobs, use squeue. Jobs may be canceled using scancel.

Parallel jobs

MPI jobs

If using an MPI implementation with support for SLURM (most implementations has such support), a parallel job is just as simple:

#SBATCH --job-name=barrier
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --mem=1G

# If not using the `mpi-selector` that ships with the Mellanox OFED
# (on machines with Mellanox InfiniBand devices), use `module` to load
# an implementation:
module load openmpi-x86_64

srun ./barrier 10000

Note the missing mpirun. SLURM detects that “barrier” is an MPI application an automatically launches the job according to the “SBATCH” specifications. That’s ice-cold cool right?! Of course, you can still execute it with mpirun to take “personal” control of the launch.

Multithreaded jobs

These are jobs that run in one process with multiple threads.

#SBATCH --job-name=slurmtest
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --threads-per-core=1
#SBATCH --mem=1G

# The program that starts the threads


--threads-per-core=1 tells slurm that it should only use one logical core per physical core. If you want to utilize Hyperthreading you can remove it.

Hybrid jobs

A mix of MPI and threading. This is done by setting --cpus-per-task to define the number of threads per MPI process.

OpenMP jobs and Hyper-Threading

OpenMP starts as many threads as there are logical cores to run on. When you ask for a number of threads with --cpus-per-task you will get that number of physical cores. However, with Hyper-Threading SLURM will give you access to all logical cores (typical two per physical core).

When you start an OpenMP program without telling how many threads you want, it will use as many as available. On systems with Hyper-Threading (most today), --cpus-per-task=4 will only give you 2 cores each with 2 logical cores. If you only want one thread per physical core, you can set the number of threads explicitely like this:

#SBATCH --job-name=slurmtest
#SBATCH --nodes=2
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --threads-per-core=1
#SBATCH --mem=1G

# The program that starts the threads


Hyper-Threading and single threaded jobs

If you want to run single threaded jobs you need to add --cpu_bind=threads. I.e. Run a job on each thread (usually 2 per core). Example:

#SBATCH --job-name=slurmtest
#SBATCH --ntasks=1
#SBATCH --cpu_bind=threads
#SBATCH --mem=1G

If you want to run a single thread on a core, then you can add:

#SBATCH --threads-per-core=1


Local scratch

The path to local node scatch storage is defined in the SCRATCH environment variable.


Hyper-threading is a technology supported by most nodes with Intel CPUs. For some applications it might improve performance.

It is off by default. If you want to use it, add the following option to your job script:

#SBATCH --ntasks-per-core=2

Note: By default slurm will only allocate one task per core, but that task will have 2 CPUs on a Hyper-threaded node. This option will give you two tasks per core.

Interactive sessions

With the update to 21.08.8 (May 2022), the steps to allocate an interactive session was changed to: salloc -p partition-name - eg:

[user@fend01 ~]$ salloc -p astro_short -N4
salloc: Pending job allocation 33291741
salloc: job 33291741 queued and waiting for resources
salloc: job 33291741 has been allocated resources
salloc: Granted job allocation 33291741
salloc: Waiting for resource configuration
salloc: Nodes node[837-840] are ready for job
[user@fend01 ~]$ hostname
[user@fend01 ~]$ srun hostname
[user@fend01 ~]$ 
notice how the allocation is for 4 machines - and how srun is needed to run the command on all 4.

Even though SLURM uses cgroups to control how many cores a user can use, we have no control over memory consumption. Thus, I would often be best to request the node in exclusive mode using –exclusive as a parameter to srun. Note that this is NOT the default, so if you mess up each others job allocations, that’s on you ;)

More info is available at: NREL SLURM Changes and SLURM 20.11 - Release notes

Fair-share scheduling

With SLURM we have introduced fair share scheduling on all nodes. This means that users who are under-serviced will receive a higher priority in the queue. You can check up on your usage with sshare. For now, all users have the same shares, but this can of course be changed to reflect the wishes of the queue owners - just let us know.

In general we have far better methods of prioritizing users and managing the queues. If you have any specific requests or similar, please contact us and we’ll discuss it.