Details

Published: 01 February 2016

Would like to inform about feature of Slurm that could be useful for our users.

The option is called --dependency which help u to configure your sbatch job to start in "dependency" mod, look for example below:

#!/bin/bash
#SBATCH --dependency=<type>

The available dependency types for job chains are:

• after:<jobID> job starts when job with <jobID> begun execution
• afterany:<jobID> job starts when job with <jobID> terminates
• afterok:<jobID> job starts when job with <jobID> terminates successfully
• afternotok:<jobID> job starts when job with <jobID> terminates with failure
• singleton jobs starts when any previously job with the same job name and user terminates

It could be very comfortable and helpful for running multiple jobs that depend on each other.

More interesting Slurm features everyone can find on the Slurm manual page: http://slurm.schedmd.com/documentation.html

Details

Published: 23 July 2015

If you are running checkpoint-able jobs in Slurm array mod, you would probably like to know where is the checkpoint file of each job.

The thing is when you running jobs in array mod, in the default squeue command you will see the id of the array job and not the unique id of the job which was check-pointed, as well as default stdout and stderr files will be saved with the array id job while all the checkpoints are saved in the folder with the original unique id of the job and not with the array job id.

Below you can see header example of the checkpoint-able job in array mod that will save each slurm job output file both with the array id and with the unique id of the job:

#!/bin/bash

#SBATCH --job-name=ckptJobArray
#SBATCH --partition=ckpt
#SBATCH --array=1-399
#SBATCH --output=out_%A_%a_%j.out
#SBATCH --error=error_%A_%a_%j.err
#SBATCH --checkpoint=360
#SBATCH --checkpoint-dir=/data/ckpt

# The following lines needed for checkponting.
##restarterOpts="StickToNodes"
restarterOpts=""
. /data/scripts/include_for_checkpointing


## To make things simple: ${i} == $SLURM_ARRAY_TASK_ID
i=${SLURM_ARRAY_TASK_ID}

## from here you can write your job commands

Pay attention to the --output and --error lines:
%A is the master array job id, %a is the number of the array job and %j is the unique id number of a job. You will find a checkpoint for your job from the unique id number of the job inside the --checkpoint-dir location.

So basically all you need to do to find which of the created checkpoints refers to your job, is just to compare the output Slurm file name with the checkpoint names(unique job id).

Below some examples:

View squeue with output of unique id number + job array id number + array number:

squeue -o "%.10A %.13i %.9P %.8j %.8u %.8T %.10M %.9l %.6D %R"

Detect which of the array jobs was preempted and restarted at-least one time:

If you are interested to investigate which of the N number of array jobs was preempted and restarted at least one time, you need to read the Slurm stdout/stderr job files. You can use a command that will read all the slurm array output files and search for the "PREEMPTION" word, below is the example of the command you can use to detect which jobs was preempted:

grep "DUE TO PREEMPTION" /YOUR/WORKDIR/*

This command will search which jobs was cancelled due to preemption.

Details

Published: 18 October 2015

Details

Published: 23 July 2015

sbatch command should be used to run jobs by default

srun and salloc commands should be used only if the user want to run interactive job and see the output on the screen.

TIP:  You do not have to create a special batch script file in order to run the job with the sbatch command, you can use the --wrap option to run the job directly from the command line. Example:

sbatch -N1 -j testjob -o job_%j.out -e job_%j.err --wrap 'hostname'

So basically you should first give an allocation of resources for the job(number of nodes, jobname, stdout and stderr files) and only after allocation you should add the --wrap option(not before!) and after the --wrap you can execute your job command.

More information can be on Slurm page: http://hivehpc.haifa.ac.il/index.php/slurm

Details

Published: 23 July 2015

If anyone would like to connect to the University network through Iphone/Android phone, here is a quick manual on how to set it up:

Iphone:

- Go to app store > search for app named: Junos Pulse > install the app(it is free!) > start the application > go to VPN tab > configuration > and enter required information:

Name(optional): Haifa univ network
URL: sslt.haifa.ac.il
Username: your vpn juniper username
authentification type: should be set to password
Realm - leave it clear
Role - leave it clear

Android:

- Go to play market > search for app named: Junos Pulse > install the app(free!) > start the app > go to VPN tab > go to connections > setup a new connection > enter requited information:

Name(optional): Haifa univ network
URL: sslt.haifa.ac.il
Username: your vpn juniper username
authentification type: should be set to password
Realm - leave it clear
Role - leave it clear

After connecting to VPN, you can connect to Hive using any of possible SSH clients that are free to use from play market or app store