The Batch System (SLURM)¶

• Large/long/parallel jobs must be run through the batch system.
• Kebnekaise is running Slurm.
• Slurm is an Open Source job scheduler, which provides three key functions.
  • Keeps track of available system resources.
  • Enforces local system resource usage and job scheduling policies.
  • Manages a job queue, distributing work across resources according to policies.
• In order to run a batch job, you need to create and submit a SLURM submit file (also called a batch submit file, a batch script, or a job script).
• Starting an interactive session through Open OnDemand also runs a batch job. It starts on a compute node. We will look at Open OnDemand at the end of this section.

Basic commands¶

Using a job script is often recommended.

• If you ask for the resources on the command line, you will wait for the program to run before you can use the window again (unless you can send it to the background with &).
• If you use a job script you have an easy record of the commands you used, to reuse or edit for later use.

In the following, JOBSCRIPT is the name you have given your job script and JOBID is the job ID for your job, assigned by Slurm. USERNAME is your username.

• Submit job: sbatch JOBSCRIPT
• Get list of your jobs: squeue -u USERNAME or squeue --me
• Give the Slurm commands on the command line: srun commands-for-your-job/program
• Check on a specific job: scontrol show job JOBID
• Delete a specific job: scancel JOBID
• Delete all your own jobs: scancel -u USERNAME
• Request an interactive allocation: salloc -A PROJECT-ID .......
  • Note that you will still be on the login node when the prompt returns and you MUST preface with srun to run on the allocated resources.
  • I.e. srun MYPROGRAM
• Get more detailed info about jobs:
  sacct -l -j JOBID -o jobname,NTasks,nodelist,MaxRSS,MaxVMSize
  • More flags etc. can be found with man sacct
  • The output will be very wide. To view in a friendlier format, use
    sacct -l -j JOBID -o jobname,NTasks,nodelist,MaxRSS,MaxVMSize | less -S
    • this makes it sideways scrollable, using the left/right arrow key
• Web url with graphical info about a job: job-usage JOBID
• More information: man sbatch, man srun, man ....

b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774852

b-an01 [~]$ squeue --me
         JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
      27774852  cpu_zen4 simple.s bbrydsoe  R       0:00      1 b-cn1701

b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774872
b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774873
b-an01 [~]$ sbatch simple.sh 
Submitted batch job 27774874
b-an01 [~]$ squeue --me
            JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
         27774873  cpu_zen4 simple.s bbrydsoe  R       0:02      1 b-cn1702
         27774874  cpu_zen4 simple.s bbrydsoe  R       0:02      1 b-cn1702
         27774872  cpu_zen4 simple.s bbrydsoe  CG      0:04      1 b-cn1702

Job scripts and output¶

The official name for batch scripts in Slurm is Job Submission Files, but here we will use both names interchangeably. If you search the internet, you will find several other names used, including Slurm submit file, batch submit file, batch script, job script.

A job submission file can contain any of the commands that you would otherwise issue yourself from the command line. It is, for example, possible to both compile and run a program and also to set any necessary environment values (though remember that Slurm exports the environment variables in your shell per default, so you can also just set them all there before submitting the job).

Outputs and any errors will per default be placed in the directory you are running from, though this can be changed.

The output file from the job run will default be named slurm-JOBID.out. It will contain both output as well as any errors. You can look at the content with vi, nano, emacs, cat, less…

The exception is if your program creates its own output files, or if you name the output file(s) differently within your jobscript.

Job scripts¶

A job submission file can either be very simple, with most of the job attributes specified on the command line, or it may consist of several Slurm directives, comments and executable statements. A Slurm directive provides a way of specifying job attributes in addition to the command line options.

Naming: You can name your script anything, including the suffix. It does not matter. Just name it something that makes sense to you and helps you remember what the script is for. The standard is to name it with a suffix of .sbatch or .sh.

Simple, serial job script

#!/bin/bash
# The name of the account you are running in, mandatory.
#SBATCH -A hpc2nXXXX-YYY
# Request resources - here for a serial job
# tasks per core is 1 as default (can be changed with ``-c``)
#SBATCH -n 1
# Request runtime for the job (HHH:MM:SS) where 168 hours is the maximum. Here asking for 15 min. 
#SBATCH --time=00:15:00 

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1

# Load the module environment suitable for the job - here foss/2023b 
module load foss/2023b

# And finally run the serial jobs 
./my_serial_program

Let us go through the most commonly used arguments:

• -A PROJ-ID: The project that should be accounted. It is a simple conversion from the SUPR project id. You can also find your project account with the command projinfo. The PROJ-ID argument is of the form
  • hpc2nXXXX-YYY (HPC2N local project)
• -N: number of nodes. If this is not given, enough will be allocated to fullfill the requirements of -n and/or -c. A range can be given. If you ask for, say, 1-1, then you will get 1 and only 1 node, no matter what you ask for otherwise. It will also assure that all the processors will be allocated on the same node.
• -n: number of tasks.
• -c: cores per task. Request that a specific number of cores be allocated to each task. This can be useful if the job is multi-threaded and requires more than one core per task for optimal performance. The default is one core per task.

Simple MPI program

#!/bin/bash
# The name of the account you are running in, mandatory.
#SBATCH -A hpc2nXXXX-YYY
# Request resources - here for eight MPI tasks
#SBATCH -n 8
# Request runtime for the job (HHH:MM:SS) where 168 hours is the maximum. Here asking for 15 min. 
#SBATCH --time=00:15:00 

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1

# Load the module environment suitable for the job - here foss/2023b 
module load foss/2023b

# And finally run the job - use srun for MPI jobs, but not for serial jobs 
srun ./my_mpi_program

Prepare the exercise environment¶

In this section, we are just going to try submitting a few jobs, checking their status, cancelling a job, and looking at the output.

Exercises¶

Using the different parts of Kebnekaise¶

As mentioned under the introduction, Kebnekaise is a very heterogeneous system, comprised of several different types of CPUs and GPUs. The batch system reflects these several different types of resources.

At the top we have partitions, which are similar to queues. Each partition is made up of a specific set of nodes. At HPC2N we have three classes of partitions, one for CPU-only nodes, one for GPU nodes and one for large memory nodes. Each node type also has a set of features that can be used to select (constrain) which node(s) the job should run on.

While Kebnekaise has multiple partitions, one for each major type of resource, there is only a single partition, batch, that users can submit jobs to. The system then figures out which partition(s) the job should be sent to, based on the requested features (constraints).

Requesting features¶

To make it possible to target nodes in more detail there are a couple of features defined on each group of nodes. To select a feature one can use the -C option to sbatch or salloc. This sets constraints on the job.

There are several reasons why one might want to do that, including for benchmarks, to be able to replicate results (in some cases), because specific modules are only available for certain architectures, etc.

To constrain a job to a certain feature, use

#SBATCH -C Type

#SBATCH -C 'zen3|zen4'

List of constraints:

For GPUs, the above GPU list of constraints can be used either as a specifier to --gpus=type:number or as a constraint together with an unspecified gpu request --gpus=number or gpus-per-node=number.

...
#SBATCH -n 8
#SBATCH -c 2
... 

Examples, constraints¶

#SBATCH -C zen4

#SBATCH -C 'zen4&GPU_AI'

#SBATCH -C 'zen3|zen4'

Examples, requesting GPUs¶

To use GPU resources one has to explicitly ask for one or more GPUs. Requests for GPUs can be done either in total for the job or per node of the job.

#SBATCH --gpus=1

#SBATCH --gpus-per-node=1

#SBATCH --gpus=number
#SBATCH -C type

Simple GPU Job - V100

#!/bin/bash
#SBATCH -A hpc2nXXXX-YYY
# Expected time for job to complete 
#SBATCH --time=00:10:00
# Number of GPU cards needed. Here asking for 2 V100 cards
#SBATCH --gpus=2
#SBATCH -C v100 

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1
# Load modules needed for your program - here fosscuda/2020b
ml fosscuda/2020b

./my-gpu-program

Open OnDemand desktop¶

Open OnDemand is a web service that allows HPC users to schedule jobs, run notebooks and work interactively on a remote cluster from any device that supports a modern browser.

Kebnekaise desktop¶

This is the first submenu point, under “Interactive Apps” -> “Desktops”.

This is used to start a desktop on one or more of the compute nodes after you have been allocated resources. This means you will be able to work as if on that node. That means that anything you run from the desktop immediately runs on the allocated resources, without you having to start (another) job.

Very useful if you want to work interactively with one of the installed pieces of software or your own code.

In addition to starting programs from the terminal, there are various applications available directly from the menu, like Libreoffice and Firefox.

When you choose this, there are some options:

• Desktop Environment: Here you can choose either “mate” (resembles Gnome 2/classic) or “xfce” (lightweight and fast). Personal preferrence.
• Compute Project: Dropdown menu where you can choose (one of) your compute projects to launch with.
• Number of hours: How long you want the job available for. Here you can choose 1-12 hours, but beware that it is a bad idea to pick longer than you need. Not only will it take longer to start, but it will also use up your allocation even if you are not actively doing anything on the desktop. Pick as long as you need to do your job.
• Number of cores: How many cores you want access to. You can choose 1-28 and they each have 4GB memory. This is only a valid field to choose if you pick “any” or “Large memory” for the “Node type” selection.
• Node type: Here you can choose “any”, “any GPU”, or “Large memory”. If you pick “any GPU” you will not pick anything for “Number of cores”.

Jupyter, MATLAB, RStudio, VSCode¶

Aside from starting a Kebnekaise desktop and running programs from there, you can also start some specific applications, namely

• Jupyter notebook
• MATLAB
• RStudio
• VSCode

They are started in much the same way as the Kebnekaise desktop, with the exception that you can generally pick a “Runtime environment” and/or a “Working Directory” to start in. The latter is picked by clicking and choosing in the filebrowser that opens.

There are more information here: Open OnDemand desktop in HPC2N’s documentation.

Runtime environment and multicore jobs¶

This is used if you have created your own environment you want to run in, for instance by adding extra and/or own installed Python modules or R packages.

• In order to use Jupyter with extra modules, you can follow this documentation I have made here: Jupyter with extra modules.
• Using R with your own runtime environment I have described here: R with own runtime environment.
• Running MATLAB as a multicore job is described here: MATLAB with multicores