JCF: on James Paul Turner's recommendation, generate pybind11-stubgen…

[jcfreeman2]

… output files when that tool is installed

Description

Basically, if pybind11-stubgen is available and a package calls daq_add_python_bindings, pybind11-stubgen will be called to generate *.pyi files to sit alongside the other files in the relevant install directory (e.g., $DBT_AREA_ROOT/install/conffwk/lib64/python/conffwk).

I'm adding this change since Pawel and James wish to use pybind11-stubgen to install a kind of type checking in Python.

To test, create a work area, and clone daq-cmake on this feature branch in $DBT_AREA_ROOT/sourcecode. Also clone one (or more) packages with Python bindings. See what happens when you run dbt-build, both with and without pybind11-stubgen locally installed.

Type of change

• Documentation (non-breaking change that adds or improves the documentation)
• [X ] New feature or enhancement (non-breaking change which adds functionality)
• Optimization (non-breaking change that improves code/performance)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (whatever its nature)

Testing checklist

These are inapplicable to daq-cmake.

• Unit tests pass (e.g. dbt-build --unittest)
• Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
• Full set of integration tests pass (dunedaq_integtest_bundle.sh)
• Python tests pass if applicable (e.g. python -m pytest)
• Pre-commit hooks run successfully if applicable (e.g. pre-commit run --all-files)

Comments here on the testing

Further checks

• Code is commented where needed, particularly in hard-to-understand areas
• Code style is correct (dbt-build --lint, and/or see https://dune-daq-sw.readthedocs.io/en/latest/packages/styleguide/)
• If applicable, new tests have been added or an issue has been opened to tackle that in the future.
  (Indicate issue here: # (issue))

[PawelPlesniak]

Thank you @jcfreeman2, I can confirm that having this branch with a local copy of conffwk has generated the relevant stubs.

git status
On branch develop
Your branch is up to date with 'origin/develop'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        python/conffwk/__init__.pyi
        python/conffwk/_daq_conffwk_py.pyi
        python/conffwk/dal.pyi
        python/conffwk/dalproperty.pyi
        python/conffwk/proxy.pyi
        python/conffwk/schema.pyi

Following this change I have a few questions

• Can we include the generated stubs in nightly releases? As a pre-requisite, we will need pybind11-stubgen as a dependency, where would be a suitable place to have this? daqpyutils may seem like a relevant option.
• If we can have this as part of the virtual environment, can we isolate the stubs into a separate subdirectory? I am concerned that the average developer will not know what these are and may be unsure on how to handle these.

[jcfreeman2]

Thanks for taking a look @PawelPlesniak :

• As far as including this in nightly releases, we can bring this up at the next Software I&T just so people are aware of the plan
• When I was testing in a work area, the *.pyi files were only appearing in the ./install directory of the work area, and I definitely didn't intend for *.pyi files to be created in the source tree of the package. I'm trying to recreate what you found, which is the *.pyi files appearing inside the conffwk repo itself, but having a bit of trouble - if you have the history of what you did, could you send it to me, here or on Slack?

[PawelPlesniak]

Sorry for the delay with the response. To implement this, I ran

pip install pybind11-stubgen
dbt-build
dbt-workarea-env

However, when running, I ended up not getting stubs when using dbt-build. Instead, I had to run the following from the release root

cd sourcecode
pybind11-stubgen -o conffwk/python/ conffwk

Note the above is from your suggested changes, but ran manually

[jcfreeman2]

I've now installed pybind11-stubgen in the externals on /cvmfs and there's now a test build available, EXTFD_DEV_260721_A9, which (1) will load it in when you set up the environment and (2) has daq-cmake installed from this feature branch. You can, e.g., build conffwk to see what happens with this setup to test this feature branch.

[PawelPlesniak]

I can confirm that when executing the following commands, with <RELEASE> as either EXTFD_DEV_270621_A9 or NFD_DEV_270623_A9, the following commands

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n <RELEASE>
cd <RELEASE>
. env.sh
cd sourcecode
git clone git@github.com:DUNE-DAQ/daq-cmake.git
cd daq-cmake
git switch johnfreeman/support_pybind11_stubgen
cd ../..
dbt-build
dbt-workarea-env
cd sourcecode
git clone git@github.com:DUNE-DAQ/conffwk.git
cd ..
dbt-build

work as normal, with the last dbt-build generating log files containing

-- Installing: /nfs/home/pplesnia/nightlyDev/EXTFD_DEV_260721_A9/install/conffwk/lib64/python/conffwk/__init__.pyi
-- Installing: /nfs/home/pplesnia/nightlyDev/EXTFD_DEV_260721_A9/install/conffwk/lib64/python/conffwk/_daq_conffwk_py.pyi
-- Installing: /nfs/home/pplesnia/nightlyDev/EXTFD_DEV_260721_A9/install/conffwk/lib64/python/conffwk/dal.pyi
-- Installing: /nfs/home/pplesnia/nightlyDev/EXTFD_DEV_260721_A9/install/conffwk/lib64/python/conffwk/dalproperty.pyi
-- Installing: /nfs/home/pplesnia/nightlyDev/EXTFD_DEV_260721_A9/install/conffwk/lib64/python/conffwk/proxy.pyi
-- Installing: /nfs/home/pplesnia/nightlyDev/EXTFD_DEV_260721_A9/install/conffwk/lib64/python/conffwk/schema.pyi

Looking inside the path sourcecode/conffwk/python/conffwk/, there are also no *.pyi files, with these instead being included in the install path, keeping the stubs separate and clean.

In both of these releases, the version of pybind11-stubgen is the latest available on its project page - v2.5.5. Thank you John!

[PawelPlesniak]

I have a request for a change here, to include the following pattern in the new install line:

PATTERN "py.typed" 

There are two ways that signatures can be checked:

• The source code is written in python (which is the case for repositories that contain both C++ and python), and both the function arguments and return types are specified following the PEP 561 standards. The use of py.typed signifies this compliance, and will get installed for packages that both have this file and are ready for the signature checking.
• The source code is written in C++, and python code is allowed to use it through pybind11. This PR allows for the stub files (*.pyi) to be generated, which allows for the function signatures to be inspected.

As it currently stands, the inclusion of *.pyi only will only allow for the stubs of the pybind11 generated code to be checked, not of the native python code that has signatures specified. An example of where we will need this is in this conffwk PR, which requires checking of both the pybind11 stubs as well as the native python code.

[PawelPlesniak]

I have one thing to discuss, which I have left in the comment, after which I will be happy for this PR to go in

[PawelPlesniak]

LGTM, thank you!