Drop compilation tests?

[ianfixes]

@per1234 I saw the announcement about Github Actions for Arduino. Does this mean I should plan to drop such support from this project?

I get the sense that arduino_ci is a bit more advanced at the moment, given that I can specify different dependencies for each compiled example. But I assume the Arduino action is going to be developed further as well.

Should I plan to drop support for some or all of arduino_ci in favor of the official action?

[per1234]

Does this mean I should plan to drop such support from this project?

Well, first I should state that I am very biased because the arduino/compile-sketches and arduino/report-size-deltas are my pet projects. Of course I'm going to think they are the best option for sketch compilation tests. However, I also very much welcome competition in this space. I've never held back in my meager attempts to be of assistance with this project, as well as my other primary competitors: adafruit/ci-arduino and ArminJo/arduino-test-compile.

My opinion is that arduino_ci should focus on doing one thing very well. To me, that one thing is unit testing.

Should I plan to drop support for some or all of arduino_ci in favor of the official action?

I don't think anything about the sketch compilation action release should make you consider dropping the unit testing aspects of arduino_ci.

I am not aware of any plans Arduino has to reproduce what arduino_ci offers in terms of unit tests. You can see the approach Arduino is taking to unit testing here:

ArduinoCore-API

tests:
https://github.com/arduino/ArduinoCore-API/tree/master/test
workflow:
https://github.com/arduino/ArduinoCore-API/blob/master/.github/workflows/unit-tests.yml

ArduinoIoTCloud library:

tests:
https://github.com/arduino-libraries/ArduinoIoTCloud/tree/master/extras/test
workflow:
https://github.com/arduino-libraries/ArduinoIoTCloud/blob/master/.github/workflows/unit-tests.yml

ArduinoBLE library (WIP)

tests:
https://github.com/Polldo/ArduinoBLE/tree/ci-workflow/extras/test
workflow:
https://github.com/Polldo/ArduinoBLE/blob/ci-workflow/.github/workflows/unit-tests.yml

I did create the arduino/cpp-test-action action you see used in those workflows, but this is an extremely minimal action that is essentially just running CMake+Valgrind+LCOV commands from the shell. It's just a way to maintain all those commands in a single location, rather than duplicating them in every workflow, as we were doing initially. I think that's all the arduino/cpp-test-action action will ever be. All the real magic is in the Catch2 and FakeIt frameworks, which Arduino has no involvement with, and the unit test code itself, which the firmware developers wrote.

The unit testing efforts have focused on the low hanging fruit for host-side testing. The code it covers does not require extensive mocking of hardware.

The sketch compilation actions are actually nothing new. Arduino has been using the sketch compilation action for over 1.5 years now, and the report-size-deltas action for a year. They have been developed and used in public repositories from the start. And this is the continuation of work I started four years ago with my per1234/arduino-ci-script project. The only thing that changed yesterday is we finally got around to doing the 1.0.0 release of the actions and putting them in GitHub Marketplace.

I can specify different dependencies for each compiled example. But I assume the Arduino action is going to be developed further as well.

I definitely have a long "to do" list for the projects, but nothing very dramatic as far as the functionality and interface goes. I've been a bit hampered by the lack of a release in how much work I could do to the actions, so this will open things up for me a bit. However, I don't have any plans to add provisions for defining dependencies per sketch to the action. This can already be done via the GitHub Actions workflow system. You can see an example of something along those lines here:
https://github.com/arduino-libraries/ArduinoIoTCloud/blob/master/.github/workflows/compile-examples.yml#L64-L176
and here:
https://github.com/arduino/arduino-examples/blob/main/.github/workflows/compile-examples.yml
That approach seems to be working out just fine for the action's users.

There has been some talk on and off about adding some form of support to Arduino CLI for defining library dependencies via the sketch metadata, and in fact Arduino Web Editor already does allow associating specific libraries with a sketch. Since arduino/compile-sketches uses Arduino CLI under the hood, it will inherit any advancements made to Arduino CLI.

[ianfixes]

This is all good info. For clarity, I would only consider dropping support for the compilation of examples, and even then only if you set up a local version of the test runner so you could try out the CI on your local machine first.

I'm very glad to see CI/CD and testing start to pop up in an official capacity at Arduino -- my goal was to have such tooling, not write it myself. To that end, I'm wondering if/how the Arduino team would ever want to adopt the testing methodology I'm using (if not the ruby implementation itself) into a project owned by that organization. Which members of the Arduino team are part of that decision making process? I'd like to get a sense of what level of maturity I'd need to shoot for in order for that to be considered viable.