Failures of integration tests with disabled connectivity services

[PawelPlesniak]

Description

Fixes issue #909

Initial PR for investigating the cause of failures of tests that use the "--no-integtest-connsvc" option in integration tests.

Type of change

• New feature / enhancement
• Optimization
• Bug fix
• Breaking change
• Documentation

List of required branches from other repositories

N/A

Change log

Only logs for now

Suggested manual testing checklist

dunedaq_integtest_bundle.sh -k minimal --verb 5 --stop --pytest-options "--no-integtest-connsvc"

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: WHAT HOSTNAME from release RELEASE_NAME

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

• Unit tests (pytest --marker) passed
  • With relevant marker
  • Without marker
• Integration tests passed
  • Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
  • Full daqsystemtest_integtest_bundle.sh
• Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
• Drunc integration tests pass (./scripts/drunc_integtest_bundle.sh)

Final checklist prior to marking this as "Ready for Review"

• Code is clearly commented.
• New unit tests have been added, or is documented in # ISSUE NUMBER
• A suitable reviewer has been chosen from this list.

Reviewer checklist

• This branch has been rebased with develop prior to testing.
• Suggested manual tests show changes.
• CI workflows fails documented (if present)
• Integration tests passed (on either np0x or IC HEP clusters)
  • Use the following guidelines to determine which of the integration tests you need to run
    • You do not need to run any integration tests if
      • Code changes are not associated with src/
      • PR changes only affect docstrings
      • In this case, be sure to validate any suggested manual testing.
    • Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
      • PR changes only affect a few log entries
      • PR changes are small, and do not have a large impact on the workflow (use carefully)
    • Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
  • What to do if the integration tests fail?
    • Only concern yourself if failures related to drunc are in the log files
    • If non-drunc failure appears:
      • Validate failure in fresh working area
      • Contact Pawel if unsure
• If you have ran the full integration test bundle, leave a comment on the PR stating
  • Which host the integration tests have ran on
  • [Optional] A copy of the test summary
• Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

• Changes only affect the Run Control, are in a single repository, and do not affect the end user.
  • Changes are documented in docstrings and code comments
  • Wiki has been updated if architectural or endpoint changes
• Otherwise
  • Workflow changes demonstrated in the Change Log (if necessary)
  • Wiki has been updated (if necessary)
  • #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

[PawelPlesniak]

When running the option --pytest-options "--no-integtest-connsvc" with integration tests, I first noticed the following

[2026/07/08 15:48:11 UTC] WARNING    process_manager_driver.py:602            drunc.process_manager_driver                       Connectivity service client (csc) is not available. Using static OKS configuration only.

To investigate the cause of this further, I added some CRITICAL logs to see what is being manipulated with the connectivity service client. This results in

++++++++++ DRUNC Session BEGIN ++++++++++
[2026/07/08 15:48:10 UTC] INFO       shell.py:186                             drunc.unified_shell                                Setting up to use the process manager with configuration ssh-standalone and configuration id "minimal" from 
oksconflibs:/tmp/pytest-of-pplesnia/pytest-192/config0/integtest-session-resolved.data.xml
[2026/07/08 15:48:10 UTC] INFO       shell.py:208                             drunc.unified_shell                                Starting process manager
[2026/07/08 15:48:10 UTC] INFO       process_manager.py:111                   drunc.process_manager                              process_manager communicating through address 10.73.136.70:41779
[2026/07/08 15:48:10 UTC] CRITICAL   shell.py:277                             drunc.unified_shell                                Investigating the session DAL's connectivity service: None
[2026/07/08 15:48:10 UTC] INFO       shell.py:537                             drunc.unified_shell                                unified_shell ready with process_manager and controller commands

Which indicates that the connectivity service is not present in the session DAL. I am unclear as to whether this is the intent behind --pytest-options "--no-integtest-connsvc", but this block in the integrationtest repository suggests whether the connectivity service should be disabled or not. As far as I am aware, and following this issue, it seems to me that as of now the connectivity service is a requirement of the session, but this should be discussed further, and seems like a motivation to develop towards addressing #397 .

I leave this open for discussion with @bieryAtFnal as he is the original poster of issue #909