Automatic testing through GitHub
Walk you through on how to use GitHub actions to run Codee related jobs.
This guide was made using an Azure HBv4 machine. You can follow along configuring your own Azure machine as your self-hosted runner.
Prerequisites
Ensure you have:
- An Azure machine configured as self-hosted GitHub runner ready to start listening to jobs. You can see more details on how to do this following the official Github guide to do so.
- Codee installed and accessible from your self-hosted GitHub runner.
- A GitHub repository with the code that you want to be automatically tested.
Automated testing in GitHub
Once you have all the prerequisites in place, you can configure GitHub Actions to execute jobs
when specific events occur on the repository. In this guide, we will configure our workflow
to automatically trigger GitHub actions when a push is made to the main branch and whenever
changes are made to a Pull Request targeting to the main branch. This jobs can also be launched
manually from GitHub if necessary from the Actions tab.
To do so you need to add configuration files in .yml format to the .github/workflows
directory. A repository may have as many workflows as you decide. In this guide we will
show you how to configure an action to analyze a code with Codee and other one to run
benchmarks using Codee as shown in the previous guides.
In this guide we will use the MATMUL example that can be found in codee-demos.
OpenBLAS
We will run a codee screening report over openBLAS
to show an example on how to analyze code automatically. In this case, we will be adding the
OpenBLAS code as a submodule of out git repository to manage our own GitHub workflows.
Before running the Codee analysis, first we need to compile the code and generate the
compile_commands.json file.
Remember that you need to have Codee installed and accessible on your GitHub runner, as well as all the dependencies necessary to compile your code.
You can avoid to compile the code each time if you have your compile_commands.json file cached.
name: openBLAS analysis
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
test-echo:
runs-on: Genoa
steps:
- name: Checkout of the repository
run: |
rm -rf openBLAS
git reset --hard
git clean -xfd
- name: Checkout of the repository
uses: actions/checkout@v4
with:
submodules: true
fetch-depth: 0
- name: Screening report
run: |
cd openBLAS
bear -- make
codee screening -p /home/codee/projects/OpenBLAS/compile_commands.json
With this simple configuration file, each time that a push is made to the main branch or
to a branch that has an open Pull Request targetting the main branch.
Results
These are the results obtained by the codee screening:
SCREENING REPORT
----Number of files-----
Total | C C++ Fortran
----- | ---- --- -------
5193 | 3169 0 2024
RANKING OF QUALITY CHECKERS
Checker Category Priority AutoFixes # Title
------- ------------------------------------- -------- --------- ----- ----------------------------------------------------------------------------------------
PWR079 correctness, portability, security P27 (L1) 11 Avoid undefined behavior due to uninitialized variables
PWR063 correctness, modern, security P12 (L1) 715 Avoid using legacy Fortran constructs
PWR068 correctness, modern, security P9 (L2) 13224 Encapsulate procedures within modules to avoid the risks of calling implicit interfaces
PWR008 correctness, modern, security P9 (L2) 9 2338 Declare the intent for each procedure argument
PWR007 correctness, modern, security P9 (L2) 2190 2193 Disable the implicit declaration of variables and procedures
PWR069 correctness, modern, security P9 (L2) 16 16 Use the keyword only to explicitly state what to import from a module
PWR003 modern, security, other P6 (L2) 78 Explicitly declare pure functions
PWR018 security, control P6 (L2) 6 Call to recursive function within a loop inhibits vectorization
PWR071 modern, portability, security P3 (L3) 7135 Prefer real(kind=kind_value) for declaring consistent floating types
PWR002 correctness, security P3 (L3) 3402 Declare scalar variables in the smallest possible scope
PWR037 correctness, security P3 (L3) 11 Potential precision loss in call to mathematical function
PWR073 correctness, modern, security P3 (L3) 3 Transform common block into a module for better data encapsulation
PWR070 correctness, modern, security, memory P2 (L3) 2143 Declare array dummy arguments as assumed-shape arrays
PWR028 security, control P2 (L3) 717 Remove pointer increment preventing performance optimization
PWR030 security, control P2 (L3) 2 Remove pointer assignment preventing performance optimization for perfectly nested loops
PWR001 correctness, modern, security P1 (L3) 228 Pass global variables as function arguments
------- ------------------------------------- -------- --------- ----- ----------------------------------------------------------------------------------------
Total 2215 32222
RANKING OF OPTIMIZATION CHECKERS
Checker Category Priority AutoFixes # Title
------- -------- -------- --------- ----- ---------------------------------------------------------------------------------------------
PWR039 memory P27 (L1) 30 30 Consider loop interchange to improve the locality of reference and enable vectorization
RMK015 other P18 (L1) 3172 Tune compiler optimization flags to increase the speed of the code
PWR032 scalar P18 (L1) 31 Avoid calls to mathematical functions with higher precision than required
PWR020 control P18 (L1) 10 Consider loop fission to enable vectorization
PWR021 control P18 (L1) 6 Consider loop fission with scalar to vector promotion to enable vectorization
PWR053 vector P12 (L1) 1701 1703 Consider applying vectorization to forall loop
PWR054 vector P12 (L1) 449 451 Consider applying vectorization to scalar reduction loop
PWR060 control P12 (L1) 8 Consider loop fission to separate gather memory access pattern
PWR044 scalar P9 (L2) 69 Avoid unnecessary floating-point data conversions involving constants
PWR045 scalar P9 (L2) 9 Replace division with a multiplication with a reciprocal
PWR046 scalar P9 (L2) 1 Replace two divisions with a division and a multiplication
PWR017 control P8 (L2) 4 Using countable while loops instead of for loops may inhibit vectorization
PWR023 memory P6 (L2) 4 Add 'restrict' for pointer function arguments to hint the compiler that vectorization is safe
PWR022 control P4 (L3) 1782 Move invariant conditional out of the loop to facilitate vectorization
PWR034 memory P4 (L3) 391 Avoid strided array access to improve performance
PWR010 memory P4 (L3) 188 Avoid column-major array access in C/C++
PWR024 multi P4 (L3) 12 Loop can be rewritten in OpenMP canonical form
PWR048 scalar P3 (L3) 803 Replace multiplication/addition combo with an explicit call to fused multiply-add
PWR029 control P3 (L3) 304 Remove integer increment preventing performance optimization
PWR040 memory P3 (L3) 15 Consider loop tiling to improve the locality of reference
PWR035 memory P2 (L3) 1365 Avoid non-consecutive array access to improve performance
PWR036 memory P2 (L3) 658 Avoid indirect array access to improve performance
PWR049 control P2 (L3) 306 Move iterator-dependent condition outside of the loop
PWR016 memory P2 (L3) 55 Use separate arrays instead of an Array-of-Structs
RMK010 memory P0 (L4) 557 Strided memory accesses in the loop body may prevent vectorization
RMK014 memory P0 (L4) 115 Unpredictable memory accesses in the loop body may prevent vectorization
------- -------- -------- --------- ----- ---------------------------------------------------------------------------------------------
Total 2180 12049