API for adding other visitors

[bridgesign]

This issue is kind of a mix of question and feature request. I am not sure if it makes sense given:

1. Its current stage
2. The scope of the project (Not sure if its limited only to C)

Scenario:

The project relies on llvmlite which was developed by numba. The numba project has put in a lot of effort in making efficient jit around numpy API in addition to cuda support. What I ideally want to do is use the numba parser to create LLVM IR for functions dealing with maths and call it through a c function defined in PythoC. Calling numba jit function via cfunc is not possible in numba and I want to avoid the python wrap and unwrap between calls.

Main Issue:

Forgetting about the issue of where to find the relevant code in numba, for the PythoC code base, what will be the easiest way to add another visitor so that it is correctly added in the unified registry and handle function calls?

• Is there already an API in PythoC to do this?
• If not, then what is a likely direction to approach this?
• Is it the case that it will require a huge refactor on the PythoC codebase?

In general, I think it will be good that PythoC exposes API to add separate visitors to allow adding or easy monkey patching for extending with other libraries.

PS: I am not a compilers person so I might have missed some obvious things.

[1flei]

I am also considering something like calling triton kernel from pythoc code so it can support GPU easily. This may not be done by ast visitor though.
What I am still planning is trying to get the maybe ptx from triton (in your case could be llvm ir from numba), and somehow wrap it in pythoc so it can be called as c-like function. It is still in plan and not very clear how to do that exactly yet.

[bridgesign]

Thanks for the reply. The reason I thought why visitor might be a good idea is because:

• of this work PyDSL.
• Currently the implementation of compile uses the annotations from the raw function definition to add it in the registry (this is what I understand from the code but might be wrong)
• It also becomes possible to call pythoc functions from external functions (example calling pythoc function from a numba function). Otherwise, the programmer needs to fight an uphill battle to keep certain type of functions as an outer layer always.
• It is easier to understand where there's a need to create a dispatch and how it needs to be created.

[1flei]

Currently the ast visitor of pythoc does not only map type hints etc. in the registry and generate to llvm ir, but also do a lot compile time computation for meta-programming and code generation. So I do not think it is easy to cleanly replace the ast visitor. It could be possible to extend the current system via duck type interface (e.g. handle_call) to extend the functionality of pythoc.