Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion integtest/process_manager_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@
"Worker with pid \\d+ was terminated due to signal",
"Connection '.*' not found on the application registry",
],
"SSH_SHELL_process_manager": [
"was terminated unexpectedly through the remote pid by a SIGKILL",
],
Comment on lines +35 to +37

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flagging this up for the review of DUNE-DAQ/integrationtest#158

"connectivity-service": [
"errorlog: -",
],
Expand Down Expand Up @@ -345,7 +348,9 @@ def test_restart_mlt_logs(run_dunerc) -> None:

require_pattern_match(
restart_text,
re.compile(r"Process 'mlt'.*?process exited\s+with exit code 0", re.DOTALL),
re.compile(
r"Process 'mlt' \(.*?\) was terminated by the process manager through the remote pid\. Reported exit code: 0\.", re.DOTALL
),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My only concern is this line, I'm not sure we've consistently got things so that the remote pid is always retrieved successfully for a clean exit. Although this is strictly better as it's more up to date so would support merging this

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Aurash, thats useful information to know!

My two cents:

  • Was unaware that we might not have a consistent thing for the remote pid
  • For this integtest I think its fine
    • integtest is not a blocker for most PRs to merge
    • this (transiently?) failing would be good to keep track of the issue until the remote pid issue gets properly fixed

error_message="Did not find the mlt exit-code log line after graceful termination.",
)

Expand Down
12 changes: 11 additions & 1 deletion scripts/drunc_integtest_bundle.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Options:
-k <pipe-delimited string to select which tests will be run ('egrep -i' match to test name)>
-n <number of times to run each individual test, default=1>
-N <number of times to run the full set of selected tests, default=1>
--verbosity <integer verbosity level to pass through to pytest>
--stop-on-failure : causes the script to stop when one of the integtests reports a failure
--concise-output : suppresses run control and DAQApp messages in order to focus on test results
--tmpdir : specifies a root directory to use for test output, e.g. a directory instead of '/tmp'
Expand Down Expand Up @@ -49,14 +50,15 @@ CaptureOutput() {
tee -a $1
}

GETOPT_TEMP=`getopt -o hs:f:l:k:n:N: --long help,stop-on-failure,concise-output,tmpdir: -- "$@"`
GETOPT_TEMP=`getopt -o hs:f:l:k:n:N: --long help,verbosity:,stop-on-failure,concise-output,tmpdir: -- "$@"`
eval set -- "$GETOPT_TEMP"

let first_test_index=0
let individual_test_requested_iterations=1
let full_set_requested_interations=1
let stop_on_failure=0
requested_test_names=
verbosity_level=
PYTEST_COMMAND="pytest -c /dev/null -s --tb=short" # our core pytest command, with DAQ printout included and short pytest traceback

while true; do
Expand Down Expand Up @@ -85,6 +87,10 @@ while true; do
let full_set_requested_interations=$2
shift 2
;;
--verbosity)
verbosity_level=$2
shift 2
;;
--stop-on-failure)
let stop_on_failure=1
PYTEST_COMMAND="${PYTEST_COMMAND} -x" # add the -x option to our pytest command to have it exit on first error
Expand All @@ -106,6 +112,10 @@ while true; do
esac
done

if [[ "${verbosity_level}" != "" ]]; then
PYTEST_COMMAND="$PYTEST_COMMAND --integtest-verbosity ${verbosity_level}"
fi

# check if the numad daemon is running
numad_grep_output=`ps -ef | grep numad | grep -v grep`
if [[ "${numad_grep_output}" != "" ]]; then
Expand Down
Loading