mypy - FSM

[miruuna]

Description

Fixes issue #846

Added type hints tosrc/drunc/fsm/. All Protocols are found in fsm/_protocols.py.

The relevant changes in the user workflow have been documented here (link URL)

Type of change

• New feature / enhancement
• Optimization
• Bug fix
• Breaking change
• Documentation

List of required branches from other repositories

None

Change log

Made fsm mypy compliant.

Suggested manual testing checklist

mypy src/drunc/fsm --follow-imports=silent - silent is needed as the other imports are not mypy compliant yet
pytest src/drunc/fsm

Prior to marking this as "Ready for Review"

Tests ran on: local device

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

• Unit tests (pytest --marker) passed
  • With relevant marker
  • Without marker
• Integration tests passed
  • Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
  • Full daqsystemtest_integtest_bundle.sh
• Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
• Drunc integration tests pass (./scripts/drunc_integtest_bundle.sh)

Final checklist prior to marking this as "Ready for Review"

• Code is clearly commented.
• New unit tests have been added, or is documented in # ISSUE NUMBER
• A suitable reviewer has been chosen from this list.

Reviewer checklist

• This branch has been rebased with develop prior to testing.
• Suggested manual tests show changes.
• CI workflows fails documented (if present)
• Integration tests passed (on either np0x or IC HEP clusters)
  • Use the following guidelines to determine which of the integration tests you need to run
    • You do not need to run any integration tests if
      • Code changes are not associated with src/
      • PR changes only affect docstrings
      • In this case, be sure to validate any suggested manual testing.
    • Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
      • PR changes only affect a few log entries
      • PR changes are small, and do not have a large impact on the workflow (use carefully)
    • Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
  • What to do if the integration tests fail?
    • Only concern yourself if failures related to drunc are in the log files
    • If non-drunc failure appears:
      • Validate failure in fresh working area
      • Contact Pawel if unsure
• If you have ran the full integration test bundle, leave a comment on the PR stating
  • Which host the integration tests have ran on
  • [Optional] A copy of the test summary
• Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

• Changes only affect the Run Control, are in a single repository, and do not affect the end user.
  • Changes are documented in docstrings and code comments
  • Wiki has been updated if architectural or endpoint changes
• Otherwise
  • Workflow changes demonstrated in the Change Log (if necessary)
  • Wiki has been updated (if necessary)
  • #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

[miruuna]

Used pybind11-stubgen to automatically generate the stubs for conffwk. To generate (or regenerate) the stubs int he /typings/ directory, run:
pybind11-stubgen conffwk --output-dir=typings
This would only have to be done if conffwk changes.

However, after running that we get:

pybind11_stubgen - [  ERROR] In conffwk._daq_conffwk_py._Configuration.destroy_obj : Invalid expression 'dunedaq::conffwk::ConfigObject'
pybind11_stubgen - [  ERROR] In conffwk._daq_conffwk_py._Configuration.get_obj : Invalid expression 'dunedaq::conffwk::ConfigObject'
pybind11_stubgen - [  ERROR] In conffwk._daq_conffwk_py._Configuration.get_objs : Invalid expression 'dunedaq::conffwk::ConfigObject'
pybind11_stubgen - [WARNING] Raw C++ types/values were found in signatures extracted from docstrings.

That probably means that the pybind11 wrapper for conffwk might have missed some bits or didn't link the return types correctly.

In the case of the conffwk's dal submodule used in /fsm/, a couple of the stubbed methods returned Any so they need to be added manually. That is done in the clean_stubs.py script, which again, should be run only if changes to conffwk were added, or the stubs need changing.

The stubs for daqconf were added manually.

[jamesturner246]

I've looked into this, and the issue with the faulty bindings type generation lies in the binding source in the conffwk repository. Luckily it's an easy fix, but will require synchronisation with the maintainers as we need to set up some CI to automate type binding generation.

@PawelPlesniak, @miruuna, let's discuss this on Thursday.

[PawelPlesniak]

Conversation regarding the required changes in conffwk and the idea of CI workflows regarding stub generation have been started on Slack between @jcfreeman2, @jamesturner246, and mysefl

[PawelPlesniak]

Thank you @miruuna and @jamesturner246
I took a look and there are a few residual errors thrown by mypy, these are

mypy .
utils.py:89: error: "Message" has no attribute "value"  [attr-defined]
utils.py:91: error: "Message" has no attribute "value"  [attr-defined]
utils.py:93: error: "Message" has no attribute "value"  [attr-defined]
utils.py:95: error: "Message" has no attribute "value"  [attr-defined]
core.py:129: error: No overload variant of "update" of "MutableMapping" matches argument type "object"  [call-overload]
core.py:129: note: Possible overload variants:
core.py:129: note:     def update(self, SupportsKeysAndGetItem[str, object], /) -> None
core.py:129: note:     def update(self, SupportsKeysAndGetItem[str, object], /, **kwargs: object) -> None
core.py:129: note:     def update(self, Iterable[tuple[str, object]], /) -> None
core.py:129: note:     def update(self, Iterable[tuple[str, object]], /, **kwargs: object) -> None
core.py:129: note:     def update(self, **kwargs: object) -> None
actions/file_run_registry.py:4: error: Skipping analyzing "daqconf.consolidate": module is installed, but missing library stubs or py.typed marker  [import-untyped]
actions/db_run_registry.py:6: error: Skipping analyzing "daqconf.consolidate": module is installed, but missing library stubs or py.typed marker  [import-untyped]
actions/db_run_registry.py:6: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
actions/db_run_registry.py:7: error: Skipping analyzing "daqconf.jsonify": module is installed, but missing library stubs or py.typed marker  [import-untyped]
actions/db_run_registry.py:8: error: Skipping analyzing "daqconf.validate": module is installed, but missing library stubs or py.typed marker  [import-untyped]
actions/thread_pinning.py:4: error: Skipping analyzing "conffwk": module is installed, but missing library stubs or py.typed marker  [import-untyped]
actions/thread_pinning.py:4: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
configuration.py:5: error: Skipping analyzing "conffwk": module is installed, but missing library stubs or py.typed marker  [import-untyped]
configuration.py:19: error: Type of variable becomes "Any" due to an unfollowed import  [no-any-unimported]
configuration.py:21: error: Argument 4 to "_fill_pre_post_transition_sequence_oks" becomes "list[Any] | None" due to an unfollowed import  [no-any-unimported]
configuration.py:93: error: "object" has no attribute "states"  [attr-defined]
configuration.py:94: error: "object" has no attribute "initial_state"  [attr-defined]
configuration.py:98: error: "object" has no attribute "actions"  [attr-defined]
configuration.py:103: error: "object" has no attribute "transitions"  [attr-defined]
configuration.py:114: error: "object" has no attribute "pre_transitions"  [attr-defined]
configuration.py:119: error: "object" has no attribute "post_transitions"  [attr-defined]
configuration.py:136: error: "object" has no attribute "command_sequences"  [attr-defined]
Found 20 errors in 6 files (checked 21 source files)

I can take a look at the daqconf work if wanted, other than that I think this is close to complete.
Note - this used a local copy of John's PR for generating stubs for conffwk.

[jamesturner246]

Thanks @PawelPlesniak. I hate to say it, but in this case the effort of merging versus quickly redoing the same changes in a fresh branch may be better in the latter. Sorry for the noise @miruuna

I'm away on leave from tomorrow until the 27th. FYI

[PawelPlesniak]

The current failure of the pytest running is associated with the fact that we have a separate PR for the compile-time instances of the OKS-defined classes, running these tests with DUNE-DAQ/conffwk#34 and DUNE-DAQ/daqconf#631 addresses the issues, and allows for mypy . ran from src/drunc/fsm to pass again from develop.

[PawelPlesniak]

Integ tests passed on np04-srv-028 using NFD_DEV_260720_A9

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Tue Jul 21 05:06:55 PM CEST 2026
Log file is: /tmp/pytest-of-pplesnia/dunedaq_integtest_bundle_20260721162413.log

⮕ Running daqsystemtest/3ru_1df_multirun_test.py ⬅
======================== 6 passed ✅ in 249.22s (0:04:09) =========================
⮕ Running daqsystemtest/3ru_3df_multirun_test.py ⬅
======================== 6 passed ✅ in 252.21s (0:04:12) =========================
⮕ Running daqsystemtest/disabled_tpg_test.py ⬅
========================= 3 passed ✅ in 60.64s (0:01:00) =========================
⮕ Running daqsystemtest/example_system_test.py ⬅
======================== 12 passed ✅ in 235.63s (0:03:55) ========================
⮕ Running daqsystemtest/fake_data_producer_test.py ⬅
======================== 6 passed ✅ in 240.04s (0:04:00) =========================
⮕ Running daqsystemtest/long_window_readout_test.py ⬅
============================== 1 skipped 🟡 in 1.65s ==============================
⮕ Running daqsystemtest/minimal_system_quick_test.py ⬅
============================== 4 passed ✅ in 53.78s ==============================
⮕ Running daqsystemtest/readout_type_scan_test.py ⬅
======================== 33 passed ✅ in 613.82s (0:10:13) ========================
⮕ Running daqsystemtest/sample_ehn1_multihost_test.py ⬅
======================== 4 skipped 🟡 in 109.15s (0:01:49) ========================
⮕ Running daqsystemtest/small_footprint_quick_test.py ⬅
============================== 3 passed ✅ in 55.41s ==============================
⮕ Running daqsystemtest/tpg_state_collection_test.py ⬅
======================== 5 passed ✅ in 111.93s (0:01:51) =========================
⮕ Running daqsystemtest/tpreplay_test.py ⬅
======================== 6 passed ✅ in 142.13s (0:02:22) =========================
⮕ Running daqsystemtest/tpstream_writing_test.py ⬅
======================== 4 passed ✅ in 108.68s (0:01:48) =========================
⮕ Running daqsystemtest/trigger_bitwords_test.py ⬅
======================== 18 passed ✅ in 309.81s (0:05:09) ========================

mypy check is also clear again

[miruuna]

Hi, apologies for the time taken to get back to this - I am a bit lost now. Are we waiting for the two PRs in conffwk and daqconf to go in?

[PawelPlesniak]

Hi Miruna, yes sorry, especially about how long this has been waiting. I am in the process of finalizing those two PRs for review. If you could look at my commits to this to ensure that they are OK, we can get this merged in ASAP.

[PawelPlesniak]

The two PRs have been marked as ready for review. Once merged, we will be able to merge this PR too.

[PawelPlesniak]

For a status update on this, the following PRs have gone in

• Adding stubs, and their inclusion in the venv conffwk#34
• Writing all function signatures in Python daqconf#631

The stubs generated from signature checking using conffwk, however, are not installed in the .venv, which is being addressed in this issue

Addressing this remaining issue should make the last mypy errors be resolved, as soon as that issue is addressed, this PR will be retested, and will be merged.

Remaining mypy errors for reference

mypy src/drunc/fsm/
src/drunc/fsm/configuration.py:5: error: Skipping analyzing "conffwk.dal": module is installed, but missing library stubs or py.typed marker  [import-untyped]
src/drunc/fsm/configuration.py:5: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
src/drunc/fsm/configuration.py:19: error: Type of variable becomes "Any" due to an unfollowed import  [no-any-unimported]
src/drunc/fsm/configuration.py:21: error: Argument 4 to "_fill_pre_post_transition_sequence_oks" becomes "list[Any] | None" due to an unfollowed import  [no-any-unimported]
src/drunc/fsm/configuration.py:82: error: Target type of cast becomes "Any" due to an unfollowed import  [no-any-unimported]
`z``

[PawelPlesniak]

The changes required to address DUNE-DAQ/daq-cmake#174 are delayed by a few days once the developer returns from AL. As a reminder, these changes refer to the distribution of stub files through CVMFS. As we will not have mypy checks as a CI workflows sooner than at the end of the current sprint, there will not be any failures raised by release generation and testing scripts by merging this PR. As such, unless there are any comments otherwise, towards the end of the work day I will re-run the checks and merge this PR into develop.

[miruuna]

For a status update on this, the following PRs have gone in

* [Adding stubs, and their inclusion in the venv conffwk#34](https://github.com/DUNE-DAQ/conffwk/pull/34)

* [Writing all function signatures in Python daqconf#631](https://github.com/DUNE-DAQ/daqconf/pull/631)

The stubs generated from signature checking using conffwk, however, are not installed in the .venv, which is being addressed in this issue

Addressing this remaining issue should make the last mypy errors be resolved, as soon as that issue is addressed, this PR will be retested, and will be merged.

Remaining mypy errors for reference

mypy src/drunc/fsm/
src/drunc/fsm/configuration.py:5: error: Skipping analyzing "conffwk.dal": module is installed, but missing library stubs or py.typed marker  [import-untyped]
src/drunc/fsm/configuration.py:5: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
src/drunc/fsm/configuration.py:19: error: Type of variable becomes "Any" due to an unfollowed import  [no-any-unimported]
src/drunc/fsm/configuration.py:21: error: Argument 4 to "_fill_pre_post_transition_sequence_oks" becomes "list[Any] | None" due to an unfollowed import  [no-any-unimported]
src/drunc/fsm/configuration.py:82: error: Target type of cast becomes "Any" due to an unfollowed import  [no-any-unimported]
`z``

Thanks @PawelPlesniak for the updates. I'll be working on getting these errors fixed today.

[PawelPlesniak]

Thanks Miruna, I have previously had these errors addressed when the conffwk stubs were installed, so I am not sure there is anything left to fix. For reference (if I am remembering these correctly), running the following commands from the work area root should address these errors when running mypy src/drunc/fsm

cd sourcecode
pybind11-stubgen -o conffwk/python/conffwk conffwk
dbt-build
dbt-workarea-env

I will report here with the results of my checks later today

[PawelPlesniak]

My apologies for not having looked at this yesterday.

As there have been no commits to this branch since the last set of integration tests, I have opted to only run the minimum system quick test to validate that there is no breaking functionality, which succeeded.

To validate these changes for the mypy checking of the fsm source code, I have run

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260728_A9
cd NFD_DEV_260728_A9
. env.sh
cd sourcecode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b PawelPlesniak/Mypy-DO_NOT_MERGE_ME
cd ..
dbt-build 
dbt-workarea-env
cd pythoncode
git clone git@github.com:DUNE-DAQ/drunc.git -b miruuna/846-feature-mypy-fsm
cd drunc
pip install -e .[dev]
pip install mypy
mypy src/drunc/fsm

which succeeded as

Success: no issues found in 21 source files

Note - for the mypy checks, the PawelPlesniak/Mypy-DO_NOT_MERGE_ME branch of conffwk is the one that has both the signature checking enabled through the inclusion of py.typed and the stubs of the pybindsrc code installed with (from the sourcecode root)

pybind11-stubgen -o conffwk/python conffwk

This can and should be used for the remainder of the mypy development effort.

[PawelPlesniak]

Thank you Miruna, and sorry about the gross delay from when you did this to when it got reviewed.