The Batch System (SLURM)¶

• Large/long/parallel jobs must be run through the batch system.
• Kebnekaise is running Slurm.
• Slurm is an Open Source job scheduler, which provides three key functions.
  • Keeps track of available system resources.
  • Enforces local system resource usage and job scheduling policies.
  • Manages a job queue, distributing work across resources according to policies.
• In order to run a batch job, you need to create and submit a SLURM submit file (also called a batch submit file, a batch script, or a job script).

Basic commands¶

Using a job script is often recommended.

• If you ask for the resources on the command line, you will wait for the program to run before you can use the window again (unless you can send it to the background with &).
• If you use a job script you have an easy record of the commands you used, to reuse or edit for later use.

In the following, JOBSCRIPT is the name you have given your job script and JOBID is the job ID for your job, assigned by Slurm. USERNAME is your username.

• Submit job: sbatch JOBSCRIPT
• Get list of your jobs: squeue -u USERNAME or squeue --me
• Give the Slurm commands on the command line: srun commands-for-your-job/program
• Check on a specific job: scontrol show job JOBID
• Delete a specific job: scancel JOBID
• Delete all your own jobs: scancel -u USERNAME
• Request an interactive allocation: salloc -A PROJECT-ID .......
  • Note that you will still be on the login node when the prompt returns and you MUST preface with srun to run on the allocated resources.
  • I.e. srun MYPROGRAM
• Get more detailed info about jobs:
  sacct -l -j JOBID -o jobname,NTasks,nodelist,MaxRSS,MaxVMSize
  • More flags etc. can be found with man sacct
  • The output will be very wide. To view in a friendlier format, use
    sacct -l -j JOBID -o jobname,NTasks,nodelist,MaxRSS,MaxVMSize | less -S
    • this makes it sideways scrollable, using the left/right arrow key
• Web url with graphical info about a job: job-usage JOBID
• More information: man sbatch, man srun, man ....

b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774852

b-an01 [~]$ squeue --me
         JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
      27774852  cpu_zen4 simple.s bbrydsoe  R       0:00      1 b-cn1701

b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774872
b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774873
b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774874
b-an01 [~]$ squeue --me
            JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
         27774873  cpu_zen4 simple.s bbrydsoe  R       0:02      1 b-cn1702
         27774874  cpu_zen4 simple.s bbrydsoe  R       0:02      1 b-cn1702
         27774872  cpu_zen4 simple.s bbrydsoe  CG      0:04      1 b-cn1702

Job scripts and output¶

The official name for batch scripts in Slurm is Job Submission Files, but here we will use both names interchangeably. If you search the internet, you will find several other names used, including Slurm submit file, batch submit file, batch script, job script.

A job submission file can contain any of the commands that you would otherwise issue yourself from the command line. It is, for example, possible to both compile and run a program and also to set any necessary environment values (though remember that Slurm exports the environment variables in your shell per default, so you can also just set them all there before submitting the job).

Outputs and any errors will per default be placed in the directory you are running from, though this can be changed.

The output file from the job run will default be named slurm-JOBID.out. It will contain both output as well as any errors. You can look at the content with vi, nano, emacs, cat, less…

The exception is if your program creates its own output files, or if you name the output file(s) differently within your jobscript.

Job scripts¶

A job submission file can either be very simple, with most of the job attributes specified on the command line, or it may consist of several Slurm directives, comments and executable statements. A Slurm directive provides a way of specifying job attributes in addition to the command line options.

Naming: You can name your script anything, including the suffix. It does not matter. Just name it something that makes sense to you and helps you remember what the script is for. The standard is to name it with a suffix of .sbatch or .sh.

Simple, serial job script

#!/bin/bash
# The name of the account you are running in, mandatory.
#SBATCH -A hpc2nXXXX-YYY
# Request resources - here for a serial job
# tasks per core is 1 as default (can be changed with ``-c``)
#SBATCH -n 1
# Request runtime for the job (HHH:MM:SS) where 168 hours is the maximum. Here asking for 15 min. 
#SBATCH --time=00:15:00 

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1

# Load the module environment suitable for the job - here foss/2022b 
module load foss/2022b

# And finally run the serial jobs 
./my_serial_program

Let us go through the most commonly used arguments:

• -A PROJ-ID: The project that should be accounted. It is a simple conversion from the SUPR project id. You can also find your project account with the command projinfo. The PROJ-ID argument is of the form
  • hpc2nXXXX-YYY (HPC2N local project)
• -N: number of nodes. If this is not given, enough will be allocated to fullfill the requirements of -n and/or -c. A range can be given. If you ask for, say, 1-1, then you will get 1 and only 1 node, no matter what you ask for otherwise. It will also assure that all the processors will be allocated on the same node.
• -n: number of tasks.
• -c: cores per task. Request that a specific number of cores be allocated to each task. This can be useful if the job is multi-threaded and requires more than one core per task for optimal performance. The default is one core per task.

Simple MPI program

#!/bin/bash
# The name of the account you are running in, mandatory.
#SBATCH -A hpc2nXXXX-YYY
# Request resources - here for eight MPI tasks
#SBATCH -n 8
# Request runtime for the job (HHH:MM:SS) where 168 hours is the maximum. Here asking for 15 min. 
#SBATCH --time=00:15:00 

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1

# Load the module environment suitable for the job - here foss/2022b 
module load foss/2022b

# And finally run the job - use srun for MPI jobs, but not for serial jobs 
srun ./my_mpi_program

Exercises¶

In this section, we are just going to try submitting a few jobs, checking their status, cancelling a job, and looking at the output.

Using the different parts of Kebnekaise¶

As mentioned under the introduction, Kebnekaise is a very heterogeneous system, comprised of several different types of CPUs and GPUs. The batch system reflects these several different types of resources.

At the top we have partitions, which are similar to queues. Each partition is made up of a specific set of nodes. At HPC2N we have three classes of partitions, one for CPU-only nodes, one for GPU nodes and one for large memory nodes. Each node type also has a set of features that can be used to select (constrain) which node(s) the job should run on.

While Kebnekaise has multiple partitions, one for each major type of resource, there is only a single partition, batch, that users can submit jobs to. The system then figures out which partition(s) the job should be sent to, based on the requested features (constraints).

Requesting features¶

To make it possible to target nodes in more detail there are a couple of features defined on each group of nodes. To select a feature one can use the -C option to sbatch or salloc. This sets constraints on the job.

There are several reasons why one might want to do that, including for benchmarks, to be able to replicate results (in some cases), because specific modules are only available for certain architectures, etc.

To constrain a job to a certain feature, use

#SBATCH -C Type

#SBATCH -C 'zen3|zen4'

List of constraints:

For GPUs, the above GPU list of constraints can be used either as a specifier to --gpus=type:number or as a constraint together with an unspecified gpu request --gpus=number or gpus-per-node=number.

Examples, constraints¶

#SBATCH -C zen4

#SBATCH -C 'zen4&GPU_AI'

#SBATCH -C 'zen3|zen4'

Examples, requesting GPUs¶

To use GPU resources one has to explicitly ask for one or more GPUs. Requests for GPUs can be done either in total for the job or per node of the job.

#SBATCH --gpus=1

#SBATCH --gpus-per-node=1

#SBATCH --gpus=type:number

Simple GPU Job - V100

#!/bin/bash
#SBATCH -A hpc2nXXXX-YYY
# Expected time for job to complete 
#SBATCH --time=00:10:00
# Number of GPU cards needed. Here asking for 2 V100 cards
#SBATCH --gpus=v100:2

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1
# Load modules needed for your program - here fosscuda/2021b
ml fosscuda/2021b

./my-gpu-program