Skip to main content

Automatic testing using a batch queue system on HPC clusters

Acknowledgment

We gratefully thank NERSC for providing access to the Perlmutter supercomputer, a HPE Cray EX system equipped with the Cray Compiler Environment (CCE).

Goal

A step-by-step guide for setting up and running automated tests using a batch queue system on HPC clusters.

Prerequisites​

Ensure you have:

  • Access to a HPC cluster with a batch queue system.
  • Codee installed and accessible from the cluster.
  • Access to the Cray Compiler Environment in the cluster.

Automated testing using the batch queue system​

Thornado​

We will run a codee screening report over Thornado to show an example on how to analyze code automatically. Ensure you have cloned Thornado in your login node.

git clone https://github.com/endeve/thornado

The following SBATCH script can be used as reference; create launch.sh, to launch the job, and thornado.sh, to generate the compile_commands.json file for thornado.

launch.sh
#!/bin/bash

#SBATCH --job-name=codee_thornado_analysis
#SBATCH --time=0:15:00
#SBATCH -C cpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=64

module load codee

bear -- ./thornado.sh
codee screening
thornado.sh
# Necessary modules
module load PrgEnv-cray
module load cray-hdf5-parallel/1.12.2.9


# The absolute path to the folder this script is in
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

# All the instruction store here were extracted from the GitHub Action definition file
# `thornado/.github/workflows/ubuntu-latest_gnu.yml` but addapted to work with cray compiler
export THORNADO_MACHINE=perlmutter_cce
export HDF5_INC=/usr/include/hdf5/openmpi
export HDF5_LIB=/usr/lib/x86_64-linux-gnu/hdf5/openmpi
export LAPACK_LIB=/usr/lib
export BLAS_LIB=/usr/lib
export THORNADO_DIR="${SCRIPT_DIR}/thornado"

target=${1:-ApplicationDriver}
make -C "${THORNADO_DIR}/SandBox/TwoMoment_OrderV/Executables/" "${target}"

Remember to add execution permissions to the script:

chmod u+x thornado.sh

Results​

This is the output of the screening report, showing a ranking of all the checkers found in Thornado:

SCREENING REPORT

---Number of files---
Total | C C++ Fortran
----- | - --- -------
99    | 0 0   99

RANKING OF QUALITY CHECKERS

Checker Category                              Priority AutoFixes #    Title
------- ------------------------------------- -------- --------- ---- ----------------------------------------------------------------------------------------------
PWR079  correctness, portability, security    P27 (L1)           9    Avoid undefined behavior due to uninitialized variables
PWR072  correctness, security                 P27 (L1) 1         4    Split the variable initialization from the declaration to prevent the implicit 'save' behavior
PWR008  correctness, modern, security         P18 (L1)           20   Declare the intent for each procedure parameter
PWR068  correctness, modern, security         P9 (L2)            18   Encapsulate procedures within modules to avoid the risks of calling implicit interfaces
PWR070  correctness, modern, security, memory P6 (L2)            331  Declare array dummy arguments as assumed-shape arrays
PWR003  modern, security, other               P6 (L2)            92   Explicitly declare pure functions
PWR069  correctness, modern, security         P6 (L2)  4         4    Use the keyword only to explicitly state what to import from a module
PWR012  modern, memory                        P2 (L3)            2    Pass only required fields from derived type as parameters
PWR001  correctness, modern, security         P1 (L3)            522  Declare global variables as function parameters
------- ------------------------------------- -------- --------- ---- ----------------------------------------------------------------------------------------------
Total                                                  5         1002

RANKING OF OPTIMIZATION CHECKERS

Checker Category Priority AutoFixes #    Title
------- -------- -------- --------- ---- -----------------------------------------------------------------------------
PWR043  memory   P18 (L1)           2    Consider loop interchange by replacing the scalar reduction value
PWR021  control  P18 (L1)           1    Consider loop fission with scalar to vector promotion to enable vectorization
PWR053  vector   P12 (L1) 180       180  Consider applying vectorization to forall loop
PWR060  control  P12 (L1)           64   Consider loop fission to separate gather memory access pattern
PWR054  vector   P12 (L1) 31        31   Consider applying vectorization to scalar reduction loop
PWR022  control  P4 (L3)            20   Move invariant conditional out of the loop to facilitate vectorization
PWR034  memory   P4 (L3)            9    Avoid strided array access to improve performance
PWR029  control  P3 (L3)            15   Remove integer increment preventing performance optimization
PWR040  memory   P3 (L3)            5    Consider loop tiling to improve the locality of reference
PWR036  memory   P2 (L3)            406  Avoid indirect array access to improve performance
PWR035  memory   P2 (L3)            243  Avoid non-consecutive array access to improve performance
PWR016  memory   P2 (L3)            181  Use separate arrays instead of an Array-of-Structs
PWR049  control  P2 (L3)            21   Move iterator-dependent condition outside of the loop
RMK010  memory   P0 (L4)            89   Strided memory accesses in the loop body may prevent vectorization
RMK012  control  P0 (L4)            15   Conditional execution in the loop body may prevent vectorization
RMK014  memory   P0 (L4)            8    Unpredictable memory accesses in the loop body may prevent vectorization
------- -------- -------- --------- ---- -----------------------------------------------------------------------------
Total                     211       1290
Last updated on