In integration tests, store drunc console output in a log file and check for warnings and errors by bieryAtFnal · Pull Request #162 · DUNE-DAQ/integrationtest

bieryAtFnal · 2026-07-22T14:39:39Z

Description

Currently, in our integration test infrastructure, the drunc console output...

is filtered according to the user-specified verbosity level
has all messages that include the words "error" or "warning" displayed on the console, independent of the requested verbosity level
is returned to the validation functions in our integtests so that custom checks can be run against it

However, there is no checking of this console output when determining whether the integtest passed or failed. The main goal of these changes is to add this checking.

My sense is that it would be great if we could use the same mechanism to allow some messages in the drunc console output to be ignored and others to be required when determining success or failure of the integtest, just like we do with messages in DAQ application log files. This led me to consider simply storing the drunc console output in a log file and doing the log-file-checking on it along with checking the existing log files. I can imagine that there might be other benefits of having the drunc console output in a log file (e.g. later reference), so I went ahead with that idea.

The changes in this PR...

store the full drunc console in a log file (in the same location on disk as all of the other integtest log files)
expand the use of centralized ignored strings a little bit
- we need to ignore lines in the drunc console output that have the word "error" in them but do not actually indicate an error
- I can imagine that centralized ignored strings will be useful in the future if we want/need to ignore a phrase in all integtests
fixed some bugs in the handling of excluded and required strings in which only the first filename pattern that matched the current log file name was used instead of all of the matching patterns

Here are sample instructions for testing these changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260722_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/drunc_output_checking
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -k min --verb 4

echo ""
echo -e "\U1F535 \U2705 Note that the integtest ran successfully and some messages were ignored in the drunc logfile. \U2705 \U1F535"
echo ""
echo ""
sleep 3

ls -alF /tmp/pytest-of-${USER}/pytest-current/runcurrent/log*drunc*.txt

echo ""
echo -e "\U1F535 \U2705 Note that a drunc logfile was created in the current Pytest directory. \U2705 \U1F535"
echo ""
echo ""

Type of change

New feature or enhancement (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)

Testing checklist

Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

Code is commented where needed, particularly in hard-to-understand areas

…nsole output to a log file.

…cking to log_file_checks.py

…ed pattern that matched a given logfile name was being checked instead of all patterns that matched the logfile name.

…onsole.

…e X11 forwarding setup failed for all app.

…t_checking

PawelPlesniak

Thank you for this PR @bieryAtFnal, I can confirm that the suggested implementation executed correctly using the suggested script. I have left some suggestions and questions on the PR, some for improvements, others for me to understand the structure more clearly. I hope this is useful.

PawelPlesniak · 2026-07-24T11:48:49Z

                severity=match_logline_prefix.group(1)
                if severity in ("WARNING", "ERROR", "FATAL"):
                    bad_line=True
+            #if not bad_line:


Can this be removed?

PawelPlesniak · 2026-07-24T11:49:33Z

        )

+    # 21-Jul-2026, KAB: phrases that we always want to exclude
+    excluded_substring_map.setdefault("drunc", []).extend(["Substate.*In error.*Endpoint"])


What is the movtivation for excluding the phrase "Endpoint" or "Substate" here?

PawelPlesniak · 2026-07-24T11:50:21Z


+    # 21-Jul-2026, KAB: phrases that we always want to exclude
+    excluded_substring_map.setdefault("drunc", []).extend(["Substate.*In error.*Endpoint"])
+    excluded_substring_map.setdefault(r".*", []).extend(["X11 forwarding setup failed"])


This issue should have been addressed by DUNE-DAQ/drunc#982, I would suggest removing it so we can keep track of whether this issue re-appears in the future

PawelPlesniak · 2026-07-24T11:54:08Z

    proc_returncode = rc_process.returncode

+    # store the full dunerc console output in a log file for reference and checking
+    with open(f"{run_dir}/log_{getpass.getuser()}_drunc_console_output.txt", "w", encoding="utf-8") as ff:


This is exactly what is needed, but I suggest an extension here. Investigating the logfile, mani ANSI escape characters associated with the use of the rich output with color coding is including additional verbosity as

�[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO �[0m �[2;37mshell.py:158 �[0m �[2;37mdrunc.unified_shell �[0m User pplesnia �[32mstarting the unified_shell�[0m �[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO �[0m �[2;37mshell.py:214 �[0m �[2;37mdrunc.unified_shell �[0m �[32mSetting up the SSH_SHELL process manager�[0m with configuration �[32mssh-standalone�[0m �[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO �[0m �[2;37mshell.py:231 �[0m �[2;37mdrunc.unified_shell �[0m Starting process manager �[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO �[0m �[2;37mprocess_manager.py:111 �[0m �[2;37mdrunc.process_manager �[0m process_manager communicating through address �[1;92m10.73.136.70�[0m�[1;32m:�[0m�[1;36m38319�[0m �[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO �[0m �[2;37mshell.py:291 �[0m �[2;37mdrunc.unified_shell �[0m Setting up the controller interface

There was a similar case of this when @emmuhamm overhauled the python logging framework, and the stripping of this was implemented in daqsystemtest in this PR. Would it be possible to implement this?

…drunc log file from integrationtest_drunc.py

…le_checks.py now that the underlying problem has been fixed.

PawelPlesniak

Thank you @bieryAtFnal, I have re-ran these tests and can confirm that the ANSI characters have been stripped out. The only remaining ones are in the various tables, but I think these should not be addressed until the duplicate nature of the table has been addressed, as you have listed in DUNE-DAQ/drunc#921.

I am interested in the selection of the exclusion line entries that I mentioned above, but this should not stop this PR from being merged.

bieryAtFnal · 2026-07-27T15:20:17Z

Thanks, @PawelPlesniak , for checking that the special ANSI characters in the drunc log file are (mostly) gone. And I agree that talking about the remaining special characters later sounds good.

To respond to your other questions/suggestions:

I have removed the "bad_line" line of code (that had been commented out)
I have removed the "X11" logfile message exclusion, as you suggested - good catch
the goal of the "Substate.*In error.*Endpoint" excluded logfile message for drunc logfiles is to filter out lines like the following:

┃ Name              ┃ Info ┃ State   ┃ Substate ┃ In error ┃ Included ┃ Endpoint                  ┃

That line has the word "error" in it, but it does not indicate a true error, of course.
I tried using "| In error |" to exclude that line, but got bogged down by the special characters in that line. So, "Substate.*In error.*Endpoint" seemed easy...

Kurt Biery added 6 commits July 15, 2026 10:20

Added functionality to integrationtest_drunc.py to store the drunc co…

73113a4

…nsole output to a log file.

Added a phrase that we always want to exclude from drunc log file che…

4f55e76

…cking to log_file_checks.py

Fixed a bug in log_file_checks.py - only the first excluded or requir…

7e8cab7

…ed pattern that matched a given logfile name was being checked instead of all patterns that matched the logfile name.

added critical messages to ones that we'll print out from the drunc c…

4cc6708

…onsole.

Updated the standard excluded phrases in log_file_checks.py to includ…

f0e3ea4

…e X11 forwarding setup failed for all app.

Merge remote-tracking branch 'origin/develop' into kbiery/drunc_outpu…

59a351f

…t_checking

bieryAtFnal requested review from PawelPlesniak and emmuhamm July 22, 2026 15:00

PawelPlesniak reviewed Jul 24, 2026

View reviewed changes

Kurt Biery added 2 commits July 26, 2026 15:12

Filtered out ANSI escape characters in the text that we write to the …

a7716ec

…drunc log file from integrationtest_drunc.py

Dropped the 'X11 forwarding' message from the excluded list in log_fi…

e2e2450

…le_checks.py now that the underlying problem has been fixed.

PawelPlesniak self-requested a review July 27, 2026 09:59

PawelPlesniak approved these changes Jul 27, 2026

View reviewed changes

bieryAtFnal merged commit 0f519b4 into develop Jul 27, 2026

bieryAtFnal deleted the kbiery/drunc_output_checking branch July 27, 2026 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

In integration tests, store drunc console output in a log file and check for warnings and errors - #162

In integration tests, store drunc console output in a log file and check for warnings and errors#162
bieryAtFnal merged 8 commits into
developfrom
kbiery/drunc_output_checking

bieryAtFnal commented Jul 22, 2026

Uh oh!

PawelPlesniak left a comment

Uh oh!

PawelPlesniak Jul 24, 2026

Uh oh!

PawelPlesniak Jul 24, 2026

Uh oh!

PawelPlesniak Jul 24, 2026

Uh oh!

PawelPlesniak Jul 24, 2026 •

edited

Loading

Uh oh!

PawelPlesniak left a comment

Uh oh!

bieryAtFnal commented Jul 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

bieryAtFnal commented Jul 22, 2026

Description

Type of change

Testing checklist

Further checks

Uh oh!

PawelPlesniak left a comment

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak left a comment

Choose a reason for hiding this comment

Uh oh!

bieryAtFnal commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PawelPlesniak Jul 24, 2026 •

edited

Loading

bieryAtFnal commented Jul 27, 2026 •

edited

Loading