Refactor test.sh into test-methods shared by run actions in CI build.yml

[gapisback]

Problem: Consistency between developer run tests v/s those run in CI.

Build-and-test tests run by developers are manual, and can sometimes go out-of-sync with tests run in CI jobs, implemented in build.yml. While adding new changes to this repo, one has to frequently go through multiple manual make executions and verify the outputs.

Some of these commands are duplicted in the run methods in CI's build.yml

This will eventually create friction when developers "think" their tests are complete, but CI jobs fail due to some other modes of test-execution.

Solution: Refactor test.sh into code that can be shared by build.yml rules.

I would like to start refactoring nascent test-code chunks in test.sh so that they can be directly re-used by the actions in build.yml run under CI.

The idea is that each test-case in test.sh is refactored into a small function, which will be called under build.yml by invoking test.sh with that named test-case-function.

An example will be a function named build-and-run-unit-tests() in test.sh and this will be invoked through CI as something like:

 30     - name: build-and-run-unit-tests
 31       run: BUILD_MODE=${{ matrix.build_type }} make clean && BUILD_MODE=${{ matrix.build_type }} CC=gcc     LD=g++ ./test.sh build-and-run-unit-tests

• Refactor test.sh into a collection of small "test-do-this" methods.
• Developers can simply run $ ./test.sh which will run through all these test-methods on their dev-box
• CI build.yml will be a series of run methods invoking each such test-method.
• To verify individual combinations, developers can run, say, $ ./test.sh build-and-run-c-samples

This way, developers can be sure that everything that needs to be built and tested in CI is also tested on their own dev boxes before submitting a PR for CI jobs.

Reference:

This exact same approach was implemented in the Certifier Framework oss project, after months of struggling with this dev-vs-CI inconsistency issue.

See test.sh and their build.yml for a reference implementation of how this turned out.

For example, a test-call in build.yml will look like:

 38     - name: test-core-certifier-programs
 39       run: |
 40         ./CI/scripts/test.sh test-core-certifier-programs

Where the matching test-code in test.sh is a function named exactly like the name: above:

112 # #############################################################################
113 function test-core-certifier-programs() {
114     echo "******************************************************************"
115     echo "* Check that core certifier programs still compile and clear tests"
116     echo "* (Also builds shared libraries for use by Python bindings.)"
117     echo "******************************************************************"
118     echo " "
119     pushd "${CERT_ROOT}"/src > /dev/null 2>&1
120
121     # ---------------------------------------------------------------------
122     # Check that core certifier programs still compile and clear tests
123     # ---------------------------------------------------------------------
124     set -x
125     make -f certifier.mak clean
126     make -f certifier_tests.mak clean
[...]

This approach is basically stitching bunches of simple Bash scripting together into something that can be used reliably.

This technique has been used successfully in that other project.

Overview of features:

• For all kinds of programs to run and test: C++ executables, Bash script, Python pytests.
• A reliable cleanup() method has been implemented which robustly checks for non-zero $rc from command failures
• Arg --list spits out the list-of-tests that can be run. --helpis also supported.

I would like to replicate that approach in this project.

If there are better ways to do this kind of dev-test-CI integration, I'm happy to hear about it.

[gapisback]

Converted this idea into issue #25 where dev refactoring will be done.