Instrumentation: code coverage

[galdolber]

I'm evaluating different solutions to implement code coverage for code I'm running with sci.

Is there any way to use the interpreter to achieve this kind of instrumentation?
Some way to hook into the evaluation process.

Sponsor

[borkdude]

@galdolber Thank you so much for sponsoring, I really appreciate that.

Currently there aren't any hooks that would help with this. Perhaps this could be done with macros. I'm not deeply familiar with how other solutions do this in Clojure. What are your alternatives so far?

[galdolber]

So far this is what I found:
https://github.com/cloverage/cloverage will rewrite the code and wrap with the instrumentation code
https://github.com/technomancy/radagast more simplistic implementation, looks like it wraps vars and uses metadata to tracks access to them

Didn't try to run or migrate any of them to sci yet.
Cloverage's approach is good, but given that sci is an interpreter I have the feeling something much better can be done.
I've been looking at sci's code, If I were to try to add these kind of hooks, any clues where I should start?

[borkdude]

SCI vars are similar to Clojure vars: they can be instrumented like instrumentation in clojure.spec.test.alpha works by replacing the root binding with a function that "wraps" using (sci/)alter-var-root. This would be the easiest implementation that requires no changes to SCI.

When adding things to SCI for instrumentation, probably the analyzer.cljc file is the place to look and add instrumentation stuff to the nodes that are generated and executed at runtime.

[borkdude]

I was able to hack cloverage + bb locally to make something work:

It requires a branch of bb called tools-analyzer and the branch cloverage-bb that I pushed here:

https://github.com/borkdude/cloverage/tree/cloverage-bb

Don't know if your use case was babashka or more SCI in general

[borkdude]

A follow up:

In the latest babashka / SCI (master) loop*, fn* are now special forms and loop, fn, defn, defmacro, and and and or are implemented as macros, which is a step in the right direction for this use case.

The rcf library has a kind of tools.analyzer-like thing going on in their code which is compatible with the latest babashka. It could be an interesting way to enable code coverage tools as well.

[borkdude]

With a few hacks I was able to make tools.analyzer work in bb:

babashka/babashka#1441

Perhaps that will also help this use case.