Euler - Leonhard Cluster Tips & Tricks

Tutorials you should read first, especially this one on submitting jobs

Access the clusters through ssh: ssh nethz@euler.ethz.ch

Modules

Application/Module List: Euler, Leonhard

• Search for modules on the wiki to see commands to load them
  • wiki can be outdated, use module avail gcc instead, this also takes into account loaded modules
• Be sure to load the modules in the correct order (new or legacy > compiler > libs > mpi)
• load module new to allow loading of newer versions (also affects modules listed by module avail)
• Depending on your compiler or MPI module only certain modules are available (see application list)
  • Known to work: euler: module load new gcc/6.3.0 cmake/3.11.4 boost/1.62.0_py3 open_mpi/3.0.0
    leonhard: module load StdEnv nano ninja gcc/8.2.0 cmake/3.14.4 boost/1.63.0 cuda/10.1.243 openmpi/4.0.1
    note: OpenMP is part of the compiler
• Boost
  • euler: use the py3 versions to ensure boost is compatible with newer compiler versions leonhard: use boost/1.63.0 not 1.69.0 if boost graph cannot be found in cmake

Getting Your Code Onto A Cluster

Use git not scp! Use scp to download your generated data. You can also add ssh keys on a cluster.

Useful Scripts

Below is a script quick_commands.sh to quickly be able to login to the clusters and up/download data. It is recommended to place this in your .bashrc or .bash_profile so you can always use the commands.

Commands:

• euler, leo, slab, slab2, slab3 to start an ssh session
• scp2cluster <euler | leo | slab1> <source> <destination> to upload source files to the destination relative to the home directory on the cluster
• scp2here to download

Ensure that you set your nethz

nethz=""
alias euler="ssh ${nethz}@euler.ethz.ch"
alias leo="ssh ${nethz}@login.leonhard.ethz.ch"
alias slab="ssh ${nethz}@slab1.ethz.ch"
alias slab2="ssh ${nethz}@slab2.ethz.ch"
alias slab3="ssh ${nethz}@slab3.ethz.ch"

function scp2cluster(){
  if [[ -z "$1" || -z "$2" || -z "$3" ]]; then
    echo -e "Usage : ${BOLD_FMT}scp2cluster <euler | leo | slab1> <source> <destination>${RESET_FMT}"
    echo    "Output: scp <source> ${nethz}@<cluster>:/cluster/home/${nethz}/<destination>"
    return 1
  fi

  get_cluster_login $1
  scp -r "${2}" "${nethz}@${cluster_login}.ethz.ch:/cluster/home/${nethz}/${3}"
}

function scp2here(){
  if [ -z ${nethz} ]; then
    echo -e "Please ${BOLD_FMT}set your nethz${RESET_FMT} in env.config or in your .bashrc"
    return 1
  elif [[ -z "$1" || -z "$2" ]]; then
    echo -e "Usage : ${BOLD_FMT}scp2here <euler | leo | slab1> <source> [destination]${RESET_FMT}"
    echo    "Output: scp ${nethz}@<cluster>:/cluster/home/${nethz}/<source> <destination | .>"
    echo    "        Empty destination downloads to current directory"
    return 1
  fi

  destination="${3}"
  if [[ -z "$3" ]]; then
    destination=.
  fi

  get_cluster_login $1
  scp -r "${nethz}@${cluster_login}.ethz.ch:/cluster/home/${nethz}/${2}" "${destination}"
}

function get_cluster_login() {
    if [[ $1 == *"leo"* ]]; then
    cluster_login="login.leonhard"
  else
    cluster_login=$1
  fi
}

If you get errors, please check your line endings are LF (Linux/Mac) first.
See also: https://scicomp.ethz.ch/wiki/Accessing_the_clusters#SSH_keys

Here is another script load_modules which can be used to quickly load your modules. Run this in each ssh session.

#!/usr/bin/env bash
# This script loads the according Environment Modules for either euler or leonhard cluster

if [[ "$0" == "$BASH_SOURCE" ]] ; then
  echo "Please run script with 'source' instead of ./"
  echo "    (modules must be loaded to current shell)"
  exit
fi

module purge  #unload all modules first
if [[ "$1" == *"euler"* ]]; then
  module load new gcc/6.3.0 cmake/3.11.4 boost/1.62.0_py3 open_mpi/3.0.0
elif [[ "$1" == *"leo"* ]]; then
  module load StdEnv nano ninja gcc/8.2.0 cmake/3.14.4 boost/1.63.0 cuda/10.1.243 openmpi/4.0.1
else
    echo "Usage: source load_modules.sh <cluster>"
    echo "       cluster = euler or leo"
    return 1
fi

module list

Misc

• To compile you may prefer to use ninja-build over make as it is faster (not available on euler)

  cmake -GNinja ..
  ninja              #instead of make -j5
  #Leonhard: module load ninja
  

• To see good MPI process vs omp threads combinations see this
• See LSF docs from IBM for help on commands/flags not found in the wiki