[Bug]: A change in the last day to `drunc` seems to have broken the integration tests

[jcfreeman2]

Description

In a nutshell, if you set up a work area based on last night's nightly, NFD_DEV_260717_A9, and proceed to run
dunedaq_integtest_bundle.sh -r daqsystemtest --include minimal_system_quick_test
you'll see this error, which is enough to result in the test failing despite other outputs (raw data file, etc.) looking fine:
[2026/07/17 15:23:39 UTC] ERROR sh.py:1725 drunc.ssh_process_lifetime_manager_shell Warning: untrusted X11 forwarding setup failed: xauth key data not generated

Steps to reproduce

Reproduction described above. Once you've recreated the problem, you can also see that using drunc from the prior nightly works fine. Assuming you're in the base of your work area with the environment set up:

cd pythoncode
git clone https://github.com/DUNE-DAQ/drunc
cd drunc
git checkout fee38fdfa02  # The code from the `NFD_DEV_260716_A9` nightly
cd ../..
dbt-build
dunedaq_integtest_bundle.sh -r daqsystemtest --include minimal_system_quick_test

...should work.

Version and environment

NFD_DEV_260717_A9

Impact

Large

Anything else?

No response

[PawelPlesniak]

I do not think this issue is caused by drunc, but rather SSH configurations on various hosts. To recreate this problem, I ran the MSQT on np04-srv-029 using the same release, and no issues have been raised

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n last_fddaq
cd NFD_DEV_260717_A9
. env.sh
dbt-build
dbt-workarea-env
dunedaq_integtest_bundle.sh -k minimal_system_quick_test.py

I am following this up with tests with running as np04daq, which has a different config, as well as daq.fnal.gov, which is the original location where this issue was reported from.

[PawelPlesniak]

Rerunning the MSQT as myself on np04-srv-029 using

dunedaq_integtest_bundle.sh -k minimal_system_quick_test.py --verbosity 5

also raised no errors.

[PawelPlesniak]

Re-running the above as np04daq on np04-srv-029 also did not show any issues.

[PawelPlesniak]

Rerunning the same on daq.fnal.gov using my FNAL services account has not managed to duplicate this issue. Logfile location

/tmp/pytest-of-plesniak/dunedaq_integtest_bundle_20260717105810.log

[PawelPlesniak]

The tests were demonstrated to fail when running in a different mode, as

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n last_fddaq
pytest <PATH_TO_MSQT>

When running with the methods I tried, the tests passed.
This does not address the X11 log message, which is another issue.

[jcfreeman2]

Another data point - if I log in to np04-srv-024 as np04daq, I also see a failure if I do this:

 8583  07/18/26 16:27:43 . /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh 
 8584  07/18/26 16:27:48 setup_dbt latest
 8585  07/18/26 16:28:00 dbt-setup-release -n NFD_DEV_260717_A9
 8586  07/18/26 16:28:16 source ~np04daq/bin/web_proxy.sh -u
 8587  07/18/26 16:28:28 pytest -v -s $DAQSYSTEMTEST_SHARE/integtest/minimal_system_quick_test.py

where the messages on the screen are:

[2026/07/18 14:28:51 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'root-controller' (session: 'minimal', user: 'np04daq') was terminated unexpectedly through the remote pid. Reported exit code: 127.
[2026/07/18 14:28:51 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-controller' (session: 'minimal', user: 'np04daq') was terminated unexpectedly with unusual SSH client exit code of 127. Reported exit code: 127.

...etc., and in the logfiles (e.g., /tmp/pytest-of-np04daq/pytest-3/run0/log_np04daq_minimal_ru-controller.txt) you see things like:

bash: line 1: drunc-controller: command not found

[PawelPlesniak]

When I ran as np04daq, I ran with dbt-create which I then executed with dunedaq_integtest_bundle.sh , which passed. Summarising the differences between the testing approaches:

• dbt-create works with dunedaq_integtest_bundle but not with pytest -v -s $DAQSYSTEMTEST_SHARE/integtest/minimal_system_quick_test.py
• dbt-setup-release does not work with pytest -v -s $DAQSYSTEMTEST_SHARE/integtest/minimal_system_quick_test.py

On Monday, I will also try dbt-setup-release on np04daq, executing with dunedaq_integtest_bundle .

[PawelPlesniak]

Failed approaches

dbt-setup-release + pytest

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-setup-release -n last_fddaq
pytest -s --integtest-verbosity 6 $DAQSYSTEMTEST_SHARE/integtest/minimal_system_quick_test.py

CLI logs

[2026/07/20 09:30:22 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'root-controller' from session 'minimal' with UUID 1a0d4f2b-cfc1-4b15-a115-c1440bedb571 on host np04-srv-028
[2026/07/20 09:30:23 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'df-controller' from session 'minimal' with UUID e59e7a62-4d41-48af-969a-43eb4f1ac1b7 on host np04-srv-028
[2026/07/20 09:30:23 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'dfo-01' from session 'minimal' with UUID e77a9336-6afb-461b-b168-700d03eb4fd9 on host np04-srv-028
[2026/07/20 09:30:23 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'root-controller' (session: 'minimal', user: 'pplesnia') was terminated unexpectedly through the remote pid. Reported exit code: 1.
[2026/07/20 09:30:24 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-controller' (session: 'minimal', user: 'pplesnia') was terminated unexpectedly through the remote pid. Reported exit code: 1.
[2026/07/20 09:30:24 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'df-01' from session 'minimal' with UUID d5c37fdd-9081-4178-a86e-966e423292c0 on host np04-srv-028
[2026/07/20 09:30:24 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'dfo-01' (session: 'minimal', user: 'pplesnia') was terminated unexpectedly through the remote pid. Reported exit code: 127.
[2026/07/20 09:30:24 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'trg-controller' from session 'minimal' with UUID 7e3ea5ba-ca2f-4c8b-93ac-20b2863848fd on host np04-srv-028
[2026/07/20 09:30:24 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-01' (session: 'minimal', user: 'pplesnia') was terminated unexpectedly through the remote pid. Reported exit code: 127.
[2026/07/20 09:30:25 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'mlt' from session 'minimal' with UUID 2387e7c2-68a1-45f9-93b5-eb7ec4e58cc2 on host np04-srv-028
[2026/07/20 09:30:25 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'mlt' (session: 'minimal', user: 'pplesnia') was terminated unexpectedly with unusual SSH client exit code of 127. Reported exit code: 127.
[2026/07/20 09:30:25 UTC] INFO       ssh_process_manager.py:322               drunc.process_manager.SSH_SHELL_process_manager    Process 'trg-controller' (session: 'minimal', user: 'pplesnia') was terminated unexpectedly through the remote pid. Reported exit code: 1.
[2026/07/20 09:30:25 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'ru-controller' from session 'minimal' with UUID 897a0375-9d3d-4b65-ad60-efdb364424d9 on host np04-srv-028
[2026/07/20 09:30:26 UTC] INFO       ssh_process_manager.py:387               drunc.process_manager.SSH_SHELL_process_manager    Booted 'ru-det-conn-0' from session 'minimal' with UUID 583930b5-58a3-4bbd-b540-fac84300c840 on host np04-srv-028

After a timeout from finding the application

.----------
🚨 Problem(s) found in logfile /tmp/pytest-of-pplesnia/pytest-274/run0/log_pplesnia_minimal_root-controller.txt:
    raise PackageNotFoundError(name)

importlib.metadata.PackageNotFoundError: drunc

----------
🚨 Problem(s) found in logfile /tmp/pytest-of-pplesnia/pytest-274/run0/log_pplesnia_minimal_df-controller.txt:
    raise PackageNotFoundError(name)

importlib.metadata.PackageNotFoundError: drunc

----------
🚨 Problem(s) found in logfile /tmp/pytest-of-pplesnia/pytest-274/run0/log_pplesnia_minimal_trg-controller.txt:
    raise PackageNotFoundError(name)

importlib.metadata.PackageNotFoundError: drunc

----------
🚨 Problem(s) found in logfile /tmp/pytest-of-pplesnia/pytest-274/run0/log_pplesnia_minimal_ru-controller.txt:
    raise PackageNotFoundError(name)

importlib.metadata.PackageNotFoundError: drunc

F
🚨 An incorrect number of raw data files was found, expected 1, found 0 🚨
F
🚨 Unable to find the data for key "minimal" when looking up metric "minimal/df-01/df-01-trb/dfmodules.TRBInfo/generated_trigger_records" in collated opmon data. 🚨
🚨 Unable to find the data for key "minimal" when looking up metric "minimal/df-01/df-01-trb/dfmodules.TRBInfo/generated_trigger_records" in collated opmon data. 🚨

Logfile entries

Root controller

Traceback (most recent call last):
  File "/nfs/home/pplesnia/.local/bin/drunc-controller", line 33, in <module>
    sys.exit(load_entry_point('drunc', 'console_scripts', 'drunc-controller')())
  File "/nfs/home/pplesnia/.local/bin/drunc-controller", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/usr/lib64/python3.9/importlib/metadata.py", line 542, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib64/python3.9/importlib/metadata.py", line 196, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: drunc

Other logs

Connectivity service correctly does not display any registered applications

dbt-setup-release + drunc-unified-shell

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-setup-release -n last_fddaq
drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-1x1-config pawel

This fails for the same reason on boot - here are the logs of the connectivity service

[2026-07-20 11:40:18 +0200] [4105651] [INFO] Starting gunicorn 23.0.0
[2026-07-20 11:40:18 +0200] [4105651] [INFO] Listening at: http://0.0.0.0:30005/ (4105651)
[2026-07-20 11:40:18 +0200] [4105651] [INFO] Using worker: gthread
[2026-07-20 11:40:18 +0200] [4105709] [INFO] Booting worker with pid: 4105709
[2026-07-20 11:40:18 +0200] [4105709] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 608, in spawn_worker
    worker.init_process()
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/workers/gthread.py", line 94, in init_process
    super().init_process()
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 135, in init_process
    self.load_wsgi()
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 147, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 57, in load
    return self.load_wsgiapp()
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 47, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/nfs/home/pplesnia/.local/lib/python3.9/site-packages/gunicorn/util.py", line 370, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'connectivityserver'
[2026-07-20 11:40:18 +0200] [4105709] [INFO] Worker exiting (pid: 4105709)
[2026-07-20 11:40:18 +0200] [4105651] [ERROR] Worker (pid:4105709) exited with code 3
[2026-07-20 11:40:18 +0200] [4105651] [ERROR] Shutting down: Master
[2026-07-20 11:40:18 +0200] [4105651] [ERROR] Reason: Worker failed to boot.

dbt-setup-release + dunedaq_integtest_bundle.sh

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-setup-release -n last_fddaq
dunedaq_integtest_bundle.sh -k minimal --verbosity 6

Gave the same behaviour as using pytest directly

Successful approaches

dbt-create + pytest

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n last_fddaq
cd NFD_DEV_260720_A9/
. env.sh
dbt-build
dbt-workarea-env
pytest -s --integtest-verbosity 6 $DAQSYSTEMTEST_SHARE/integtest/minimal_system_quick_test.py

dbt-create + dunedaq-integtest-bundle

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n last_fddaq
cd NFD_DEV_260720_A9/
. env.sh
dbt-build
dbt-workarea-env
dunedaq_integtest_bundle.sh -k minimal --verbosity 6

Current summary

There is clearly a difference with the way that modules are recognised when setting an area up using dbt-setup-release and dbt-create. I will look at the difference in logs between running in debug mode using dbt-setup-release and dbt-create