Continuing Aurash's work on MSQT CI workflow

[PawelPlesniak]

Description

Fixes issue # ISSUE NUMBER

WHAT DOES THIS PR CHANGE - ONE LINE.

DON'T FORGET TO LABEL THE ISSUE WITH THE APPROPRIATE TOPIC

DOCUMENT THE CHANGE BELOW OR DELETE IT

The relevant changes in the user workflow have been documented here (link URL)

Type of change

• New feature / enhancement
• Optimization
• Bug fix
• Breaking change
• Documentation

List of required branches from other repositories

WHAT PRs NEED TO BE INCLUDED TO MAKE THE CHANGE.

Change log

WHAT HAS CHANGED.

Suggested manual testing checklist

LIST COMMANDS TO DEMONSTRATE CHANGE

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: WHAT HOSTNAME from release RELEASE_NAME

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

• Unit tests (pytest --marker) passed
  • With relevant marker
  • Without marker
• Integration tests passed
  • Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
  • Full daqsystemtest_integtest_bundle.sh
• Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
• Drunc integration tests pass (./scripts/drunc_integtest_bundle.sh)

Final checklist prior to marking this as "Ready for Review"

• Code is clearly commented.
• New unit tests have been added, or is documented in # ISSUE NUMBER
• A suitable reviewer has been chosen from this list.

Reviewer checklist

• This branch has been rebased with develop prior to testing.
• Suggested manual tests show changes.
• CI workflows fails documented (if present)
• Integration tests passed (on either np0x or IC HEP clusters)
  • Use the following guidelines to determine which of the integration tests you need to run
    • You do not need to run any integration tests if
      • Code changes are not associated with src/
      • PR changes only affect docstrings
      • In this case, be sure to validate any suggested manual testing.
    • Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
      • PR changes only affect a few log entries
      • PR changes are small, and do not have a large impact on the workflow (use carefully)
    • Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
  • What to do if the integration tests fail?
    • Only concern yourself if failures related to drunc are in the log files
    • If non-drunc failure appears:
      • Validate failure in fresh working area
      • Contact Pawel if unsure
• If you have ran the full integration test bundle, leave a comment on the PR stating
  • Which host the integration tests have ran on
  • [Optional] A copy of the test summary
• Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

• Changes only affect the Run Control, are in a single repository, and do not affect the end user.
  • Changes are documented in docstrings and code comments
  • Wiki has been updated if architectural or endpoint changes
• Otherwise
  • Workflow changes demonstrated in the Change Log (if necessary)
  • Wiki has been updated (if necessary)
  • #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

[PawelPlesniak]

This showed that the real issue is that despite a required/recommended split, there are still issues

============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /cvmfs/dunedaq-development.opensciencegrid.org/nightly/NFD_DEV_260724_A9/spack-0.22.0
configfile: pytest.ini
plugins: cov-7.1.0, integrationtest-4.0.0, anyio-4.12.1
collected 3 items

../../../../../cvmfs/dunedaq-development.opensciencegrid.org/nightly/NFD_DEV_260724_A9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/daqsystemtest-NFD_DEV_260724_A9-ep74w3g75xpzvdhc57enucx3i4alnpto/share/integtest/small_footprint_quick_test.py 

DEBUG: CPU count is 4, required number is 4, recommended number is 8
DEBUG: Free memory is 14.23 GB, required amount is 4, recommended amount is 6
DEBUG: Free disk space on "." is 78.30209350585938 GB, required amount is 1

*** Note: This computer (ee3cac73b8a4) does not have the recommended amount of resources to run this test.
 * CPU count is 4, recommended CPU count is 8.

Generating app=0
New nic adding nic with id nic-0
Generating stream_no=0
Readout map file /__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/ReadoutMap.data.xml
Adding /__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml to include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
Creating OKS database file /__w/drunc/drunc/nightly/NFD_DEV_260724_A9/pytest-of-root/pytest-0/config0/readout-segment.data.xml
E
*** Note: This computer (ee3cac73b8a4) does not have the recommended amount of resources to run this test.
 * CPU count is 4, recommended CPU count is 8.


!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 2.10s ===============================

\u0001F7E1 WARNING: Unable to copy the pytest directory for this failed test (). \u0001F7E1

[PawelPlesniak]

Continuing debugging the workflow, the issue has been found. When running the workflow using small_footprint_quick_test.py, the test requires 4 CPUs and recommends 8. The runner logs that it has 4 CPUs available, but fails to run as

DEBUG: CPU count is 4, required number is 4, recommended number is 8
DEBUG: Free memory is 14.24 GB, required amount is 4, recommended amount is 6
DEBUG: Free disk space on "." is 79.17105102539062 GB, required amount is 1

*** Note: This computer (91617a354e76) does not have the recommended amount of resources to run this test.
 * CPU count is 4, recommended CPU count is 8.

Generating app=0
New nic adding nic with id nic-0
Generating stream_no=0
Readout map file /__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/ReadoutMap.data.xml
Adding /__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml to include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
/__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/integrationtest-objects.data.xml already in include list
Creating OKS database file /__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/readout-segment.data.xml
E
*** Note: This computer (91617a354e76) does not have the recommended amount of resources to run this test.
 * CPU count is 4, recommended CPU count is 8.

This log was taken directly from this CI workflow run . @bieryAtFnal do you know what could be causing this? The command that is run to run the test is

dunedaq_integtest_bundle.sh --stop-on-failure --pytest-options "--skip-resource-checks" -k small_footprint_quick_test.py --verbosity 5 --tmpdir . 2>&1

The test is stopped due to the resource allocation issue, and when the --skip-resource-checks option is specified, the number of resources is still validated. It seems that the validation is also against the recommendation, instead of the minimum requirement.

[PawelPlesniak]

After several other attempted runs, the core error message is still

 Note: This computer (58fc5bbc58a6) does not have the recommended amount of resources to run this test.

At the top of the tests, there is debug log

DEBUG: CPU count is 4, required number is 4, recommended number is 8
DEBUG: Free memory is 14.24 GB, required amount is 4, recommended amount is 6
DEBUG: Free disk space on "." is 79.17105102539062 GB, required amount is 1

Is there a way to check available resources against the required numbers instead of the requested numbers?

[bieryAtFnal]

Hi @PawelPlesniak , I believe that the problem is something other than the recommended number of CPUs. The "note" that you are seeing in the console output does not indicate an error. It is just a note.

I just ran dunedaq_integtest_bundle.sh --stop-on-failure --pytest-options "--skip-resource-checks" -k small_footprint_quick_test.py --verbosity 5 --tmpdir . 2>&1 on daq.fnal.gov after locally modifying the small_footprint_quick_test to recommend 8000 ! CPUs, and it ran fine.

In my successful run, there were messages related to the generation of the OKS configuration after

• Creating OKS database file /__w/drunc/drunc/nightly/NFD_DEV_260727_A9/pytest-of-root/pytest-0/config0/readout-segment.data.xml

The absence of more config-generation messages in your test seems the more likely problem. Have you confirmed that the github runner has write access to the local area on disk? (--tmpdir .)

[PawelPlesniak]

Thank you for investigating this further Kurt, I will take a look at these suggestions later today.

[bieryAtFnal]

As confirmation that the recommended number of CPUs is not causing a problem, you could locally modify your copy of small_footprint_quick_test to recommend <=4 CPUs...

[bieryAtFnal]

And, we can work together to locally modify your software area to better understand if it is the configuration generation that is experiencing the problem...