Performance monitoring over time: Implement some airspeed velocity performance tests

[dfsp-spirit]

To implement performance monitoring over time, we should Implement some airspeed velocity performance tests for Syncopy.

See airspeed velocity (asv) docs for more info. Basically we would like to be able to compare performance between different commits (releases, branches).

This is now WIP, see branch 494_asv

[dfsp-spirit]

After some research, we know that

• airspeed velocity exists and is suitable to identify regressions in commits. It works by saving (in a JSON file) the output of performance tests, which are stored as Python code, very similar to unit tests. The files are tagged with machine, number of cores, commit, etc. The idea is to save the JSON files over time to be able to compare run times. Visualization is supported. It works by creating a new virtual env, installing the software version of a given commit into it, and timing the tests.
• however, the tool seems not well maintained and not very flexible, see asv issue 1219
• integration of asv into our github CI pipeline would be very nice, but we are not sure how/where to upload the report JSON files in that case. Same problem as with coverage, where the problem is solved by uploading to 3rd-party websites like coveralls. See asv issue 796. UPDATE: There seems to exist an option to upload to gh-pages, the the asv gh-pages command.
• an example for asv usage on github is here, but they use setup.py

[dfsp-spirit]

I have opened an issue with asv, I cannot get it to work with poetry.

[dfsp-spirit]

I have it working in the 494_asv branch with setup.py for now. Type asv run -v HEAD^!to benchmark the latest commit (in your current branch).

[dfsp-spirit]

I noticed that the scipy team has a nice wrapper around asv in their runtests.py script.