Handling exceptions when applications do not report errors but time out in stateful transitions by PawelPlesniak · Pull Request #868 · DUNE-DAQ/drunc

PawelPlesniak · 2026-03-31T12:45:31Z

Description

Fixes #803
Fixes #869
Fixes #911

If a segment does not reach the target state, it is marked as in error, and the timeout is logged in the relevant server.
Also defines a set of configurations constructed to fail, and defines a set of unit tests to demonstrate this behaviour.
Error recovery with the supervisor will address what happens if an application completes this outside of the designated window. This is defined in #840

Type of change

List of required branches from other repositories

Requires DUNE-DAQ/druncschema#87

Change log

Addressed issues

Now when an application does not complete its transition in the allocated timeout, the session is put into an error state.
Defines a new click command log that allows the user to send a message to the relevant controller or process manager to put into their log file with a selectable severity level.

Other new features

When running in batch mode, the user can start the run with --no-stop-error-batch-mode. Should an error state be encountered, the session is not terminated, and commands can continue to be executed. This is primarily intended for use with integration tests, such that we can inspect the ps and status tables.

Integration tests for failure modes

Defines a set of intentionally failing configurations in config/tests/failure-testing.data.xml, which contain configurations with a set of fake-daq-apps that fail at configurable points in the session lifecycle. These configurations are

ft-death-on-boot-nest-app - this kills a nested application (2 segments deep) on boot.
ft-death-on-boot-top-app - this kills the top application on boot.
ft-death-post-boot-nest-app - this kills a nested application (2 segments deep) after boot, before applications are marked as ready.
ft-death-post-boot-top-app - this kills the top application after boot, before applications are marked as ready.
ft-fsm-cmd-timeout-nest-app - this times out an FSM transition on a nested application.
ft-fsm-cmd-timeout-top-app - this times out an FSM transition on the top application.
ft-fsm-cmd-death-nest-app - this kills a nested fake daq app during a FSM transition.
ft-fsm-cmd-death-top-app - this kills the top fake daq app during a FSM transition.

The differences between the on-boot and post-boot are useful to see, as the post-boot configs allow the applications that die to register themselves on the connectivity service before dying, hence there is different behaviour.

The differences between the nest-app and top-app allow the user to visualize the effect of a nested hierarchy on the control tree, and how the releveant children applications register themselves to the segment parents. The user shoudl expect to see the majority of the apps in the status table on boot when using a nest config, and only the top two apps when using the top config.

Note that these configurations use the FSMconfiguration_noAction state machine, and will not behave fulliy in the standard way. An example includes not being able to reach the running state as there is no run number allocated, which will be reported as a generic gRPC error.

These tests have been integrated into the unit test framework, which can be executed as (from the local drunc root)

source scripts/setup_drunc_config_path.sh
pytest src/drunc/integtest/<integ_test_file_name>

It is recommended for the tester to use --integtest-verbosity 5 to see what is happening during the process, and if more details are wanted from the pytest side, the user can run the tests as pytest -s to show what would be printed.

Suggested manual testing checklist

Run each of the integration tests as defined above.

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: np04-srv-029 from release NFD_DEV_260709_A9.

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

Unit tests (pytest --marker) passed
- With relevant marker
- Without marker
Integration tests passed
- Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
- Full daqsystemtest_integtest_bundle.sh
Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows

Final checklist prior to marking this as "Ready for Review"

Code is clearly commented.
New unit tests have been added, or is documented in # ISSUE NUMBER
A suitable reviewer has been chosen from this list.

Reviewer checklist

This branch has been rebased with develop prior to testing.
Suggested manual tests show changes.
CI workflows fails documented (if present)
Integration tests passed
- Only concern yourself if failures related to drunc are in the log files
- If non-drunc failure appears:
  - Validate failure in fresh working area
  - Contact Pawel if unsure

Once the features are validated and both the unit and integration tests pass, the PRs is ready to be merged.

Prior to merging

Choose one of the following an complete all substeps

Changes only affect the Run Control, are in a single repository, and do not affect the end user.
- Changes are documented in docstrings and code comments
- Wiki has been updated if architectural or endpoint changes
Otherwise
- Workflow changes demonstrated in the Change Log (if necessary)
- Wiki has been updated (if necessary)
- #daq-sw-librarians Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

PawelPlesniak · 2026-03-31T13:30:45Z

In the case where a second application also fails to complete a transition in time, the same error gets thrown. This is likely caused by the nested structure, and the fact that there are multiple layers to this configuration. A robust solution to this problem will take longer to achieve, but I will continue working on it.

…to agreement with SWIT as to where to host these configs)

…M timeouts and death on FSM, REQUIRES TESTING

PawelPlesniak · 2026-06-05T13:58:39Z

Log files do not contain wierd characters from redirecting rich ASCII output to a file with colors
Top app config failure not working?

…to be propagated down to the actual failing app level

PawelPlesniak · 2026-06-08T17:01:12Z

Stream handler has no logging time zone, file name, line number, etc?
Duplicate logs of apps failing
Remove old logs used for debugging
Add druncschema dependency to this PR log
Integrate the failure mode testing into the unit tests

PawelPlesniak · 2026-07-16T13:16:21Z

Integration tests passed on np04-srv-029

======================== 18 passed in 293.70s (0:04:53) ========================

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Thu Jul 16 03:15:52 PM CEST 2026
Log file is: /tmp/pytest-of-pplesnia/dunedaq_integtest_bundle_20260716143520.log

⮕ Running daqsystemtest/3ru_1df_multirun_test.py ⬅
======================== 6 passed ✅ in 244.30s (0:04:04) =========================
⮕ Running daqsystemtest/3ru_3df_multirun_test.py ⬅
======================== 6 passed ✅ in 243.98s (0:04:03) =========================
⮕ Running daqsystemtest/disabled_tpg_test.py ⬅
============================== 3 passed ✅ in 59.25s ==============================
⮕ Running daqsystemtest/example_system_test.py ⬅
======================== 12 passed ✅ in 234.24s (0:03:54) ========================
⮕ Running daqsystemtest/fake_data_producer_test.py ⬅
======================== 6 passed ✅ in 235.73s (0:03:55) =========================
⮕ Running daqsystemtest/long_window_readout_test.py ⬅
============================== 1 skipped 🟡 in 1.52s ==============================
⮕ Running daqsystemtest/minimal_system_quick_test.py ⬅
============================== 4 passed ✅ in 53.46s ==============================
⮕ Running daqsystemtest/readout_type_scan_test.py ⬅
======================== 33 passed ✅ in 576.37s (0:09:36) ========================
⮕ Running daqsystemtest/sample_ehn1_multihost_test.py ⬅
======================== 4 skipped 🟡 in 84.05s (0:01:24) =========================
⮕ Running daqsystemtest/small_footprint_quick_test.py ⬅
============================== 3 passed ✅ in 51.67s ==============================
⮕ Running daqsystemtest/tpg_state_collection_test.py ⬅
======================== 5 passed ✅ in 107.56s (0:01:47) =========================
⮕ Running daqsystemtest/tpreplay_test.py ⬅
======================== 6 passed ✅ in 118.26s (0:01:58) =========================
⮕ Running daqsystemtest/tpstream_writing_test.py ⬅
======================== 4 passed ✅ in 110.07s (0:01:50) =========================
⮕ Running daqsystemtest/trigger_bitwords_test.py ⬅
======================== 18 passed ✅ in 293.70s (0:04:53) ========================

…ansition

PawelPlesniak · 2026-07-17T14:42:06Z

Duplicate code is present in develop between send_msg and log_on_server, resolving this to log_on_server, see #888 for more details

PawelPlesniak · 2026-07-17T14:58:40Z

@emmuhamm should be good to go

…ansition

emmuhamm · 2026-07-23T10:34:34Z

Note that for the review process, this PR and the dependency is up to date with develop as of today's (23 july) nightly.

A few changes that were implemented are porting over some of the new functionality that now exists in develop (particularly about how all lines should have stripped_ansi now) and minor changes to get some of the other tests working. Including increasing the widths of the table. as well as changing the column names.

I'll proceed to give comments, but this may come in bits and pieces

emmuhamm

Review 1/?

Ran out of bandwidth for now but this should be most of my comments for the testing side of things.

Pretty minor comments for this PR, some could be defferred for a bigger issue. Importantly, all the tests run just fine

emmuhamm · 2026-07-21T10:34:03Z

+pytestmark = pytest.mark.skipif(
+    drunc_missing,
+    reason="drunc is not present in DUNEDAQ_DB_PATH, skipping drunc integration tests",
+)


This error message should also say how to fix this. (this applied to all the files)

This is a good point to raise. However, the way in which the configuration file is packaged for retrieval by the integration tests defined within python-only DD repos is still under discussion. Please leave this unresolved so this can be addressed ASAP.

emmuhamm · 2026-07-21T10:37:06Z

+drunc_missing = not any(
+    "drunc" == segment for path in db_path_env.split(":") for segment in path.split("/")
+)


For posterity: why is this necessary?

This is the way to check if drunc is included as part of the DUNEDAQ_DB_PATH env var, so that the path to the configuration file can actually be matched (and the session can be run). This need to be explicitly stated as equality, as initial implementations matched this for druncschema, which did caused failure of the integration tests (the session wouldn't run as it wouldn't find the config, and would just print a stack of errors)

This makes sense, thanks. Can this be added in the comment / error message then?

Mainly that this is one of the checks that the tests actually perform, and when I saw it fail I wasn't sure initially why we had this check in the first place just from reading the code. Will be useful in the future when other people develop drunc and encounter this failing.

Also, I wonder if the comments / error message should also say how this is fixed? eg
export DUNEDAQ_DB_PATH=$PWD:$DUNEDAQ_DB_PATH. Mainly cuz this appears to be a 'user configuration issue' when installing drunc rather than a genuine 'uh oh my development caused drunc to break' thing

emmuhamm · 2026-07-23T10:50:09Z

+    except FileNotFoundError:
+        print(f"Error: {file} not found.")
+        return False


Do we want to return false here or throw? Feels like in a testing environment if we cant find the file it should throw

This should be left as a boolean return, mainly for cleaner output. The way that the test is currently structured is on a query base, where we check whether the line exists, and if it does not, then we return. Both can achieve the same functionaltiy, but I think this is cleaner. Should this fail in the future, when running with pytest -s (-s allows seeing the print statements in the tests), the error will be a one line solution, rather than a stacktrace.

emmuhamm · 2026-07-23T10:54:02Z

 _PS_COLUMNS = ["session", "friendly_name", "user", "host", "uuid", "alive", "exit_code"]
 _STATUS_COLUMNS = [
    "name",
    "info",
    "state",
    "substate",
    "in_error",
    "included",
    "endpoint",
 ]
 _EXEC_REPORT_COLUMNS = ["name", "command_execution", "fsm_transition"]


After the big developments of this tests, im starrting to think we should improve this part. Specifically, the columns themselves are called in the tests 'eg check if [substate] exists'

We should definitely strive to improve this, but this is beyond the scope of this PR.

If you agree, can convert this to an issue?

Sorry the question here is unclear to me, what would you propose changing with these variables?

emmuhamm · 2026-07-23T10:56:53Z

+#! Replace this with a generic one
+def get_rows_by_friendly_name_from_ps_table(
    ps_table: list[dict[str, str]], friendly_name: str
 ) -> list[dict[str, str]]:
    """Return all rows whose `friendly_name` matches exactly after stripping."""
    return [row for row in ps_table if row["friendly_name"].strip() == friendly_name]


+def get_rows_by_name_from_status_table(
+    status_table: list[dict[str, str]], name: str
+) -> list[dict[str, str]]:
+    """Return all rows whose `Name` matches exactly after stripping."""
+    return [row for row in status_table if row["name"].strip() == name]
+


The two functions are nearly identical. Suggest

def get_rows_by_name_from_table(table: list[dict[str,str]], name:str, row_name : str) -> list: return [row for row in table if row[row_name].strip() == name]

Would this be easy to implement? If not, happy to fold this into the issue i mentioned above about the columns

The issue is that the ps table lists the process name as friendly_name, and the status table lists the process name as name. I think we could do this, but this should be a separate PR, including changing the actual ps table process name to just name and not friendly_name

PawelPlesniak · 2026-07-23T15:55:26Z

Thanks @emmuhamm I will look tomorrow
Note to self - requested from @MRiganSUSX , list the application that did not complete in time, and tell the user to look at the logs of that specific one

Co-authored-by: Emir Muhammad <49058518+emmuhamm@users.noreply.github.com>

PawelPlesniak

Sorry, for some reason parts of these comments were just comment replies, others were a second review.

PawelPlesniak · 2026-07-24T12:27:05Z

 _PS_COLUMNS = ["session", "friendly_name", "user", "host", "uuid", "alive", "exit_code"]
 _STATUS_COLUMNS = [
    "name",
    "info",
    "state",
    "substate",
    "in_error",
    "included",
    "endpoint",
 ]
 _EXEC_REPORT_COLUMNS = ["name", "command_execution", "fsm_transition"]


Sorry the question here is unclear to me, what would you propose changing with these variables?

PawelPlesniak · 2026-07-24T12:28:43Z

+#! Replace this with a generic one
+def get_rows_by_friendly_name_from_ps_table(
    ps_table: list[dict[str, str]], friendly_name: str
 ) -> list[dict[str, str]]:
    """Return all rows whose `friendly_name` matches exactly after stripping."""
    return [row for row in ps_table if row["friendly_name"].strip() == friendly_name]


+def get_rows_by_name_from_status_table(
+    status_table: list[dict[str, str]], name: str
+) -> list[dict[str, str]]:
+    """Return all rows whose `Name` matches exactly after stripping."""
+    return [row for row in status_table if row["name"].strip() == name]
+


The issue is that the ps table lists the process name as friendly_name, and the status table lists the process name as name. I think we could do this, but this should be a separate PR, including changing the actual ps table process name to just name and not friendly_name

PawelPlesniak · 2026-07-24T12:29:19Z

+    except FileNotFoundError:
+        print(f"Error: {file} not found.")
+        return False


This should be left as a boolean return, mainly for cleaner output. The way that the test is currently structured is on a query base, where we check whether the line exists, and if it does not, then we return. Both can achieve the same functionaltiy, but I think this is cleaner. Should this fail in the future, when running with pytest -s (-s allows seeing the print statements in the tests), the error will be a one line solution, rather than a stacktrace.

…lication times out

…ithub.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition

emmuhamm

Review 2/?
Hello hello here's another round of reviews. Gone through most of the log code and mostly appears fine. Only got a few comments on that bit

emmuhamm · 2026-07-24T15:40:56Z

+    log = get_logger("unified_shell.log_on_server")
+    log.debug("Logging message to server(s)...")
+
+    if target_server in ["", "process_manager"]:


log --target-server df-controller "hello3"
I would expect this to send a hello3 to the 'df-controller' server. But this doesnt work.

Maybe a smarter system that can distribute it?

(Maybe beyond the scope of this PR but useful to think about)

emmuhamm · 2026-07-24T15:51:20Z

+    def log_on_server(
+        self,
+        text: str,
+        severity: str = "INFO",
+        target: str = "",
+        execute_along_path: bool = False,
+        execute_on_all_subsequent_children_in_path: bool = True,
+    ) -> LogOnServerResponse:


emmuhamm · 2026-07-24T15:51:37Z

            flag=ResponseFlag.EXECUTED_SUCCESSFULLY,
        )
+
+    def log_on_server(


emmuhamm · 2026-07-24T15:52:04Z

    execute_on_all_subsequent_children_in_path: bool,
    extended: bool,
 ) -> None:
+    obj.log.warning(f"Getting status for target '{target}'...")


why is this a warning?

emmuhamm

Review 3/?

Just a quick set of review again to keep the ball rolling. Not sure if GitHub tracks which files I've viewed and which I haven't that you can see but I've gone through a significant chunk of the code now :)

emmuhamm · 2026-07-29T08:27:46Z

 # Original author: KAB, 10-Oct-2023

-integtest_list=( "process_manager_test.py" "controller_test.py" )
+integtest_list=( "process_manager_test.py" "controller_test.py" "test_failure_mode_death_on_boot_nest_app.py" "test_failure_mode_death_on_boot_top_app.py" "test_failure_mode_death_post_boot_nest_app.py" "test_failure_mode_death_post_boot_top_app.py" "test_failure_mode_fsm_cmd_timeout_nest_app.py" "test_failure_mode_fsm_cmd_death_nest_app.py")


Minor comment but we should standardise how we name these tests.

eg. the pm and controller is *_test.py, whereas these failure modes are test_*.py. I don't mind which of these choices we go for either way, but I think we should definitely standardise them

emmuhamm · 2026-07-29T08:30:23Z

+        # The following line is outdated, but in the future when error states and their
+        # recovery are better defined, we can provide better options to the user.


Can you say a few words on why this is outdated?

emmuhamm

Review 3/?

Currently at 30/43 viewed now :)

In the description somewhere, please mention that send_msg is now superseded by the new log message

emmuhamm · 2026-07-29T10:12:04Z


+def count_processes_in_status_response(response: StatusResponse) -> int:
+    """
+    Count the number of processes in the status table, including all children.


Minor comment but should add 'Recursively' somewhere here in the desc

emmuhamm · 2026-07-29T10:37:42Z

+    put_in_error_state: bool = False
+
+    # If the session applications are not found on the connectivity serivce, then the
+    # session is not booted correctly. This is a critical error, the user should be


Should an error be sent over ERS as well?

emmuhamm · 2026-07-29T10:43:10Z

+
+    # Local connectivity serivces are not reported in the status table, but they should
+    # be. Increment the status_process_count by 1 if using the LCS.
+    # TODO: Remove this once the LCS is reported in the status table (issue 745).


Make sure you link this in the relevant issue as well so it doesn't get lost

emmuhamm · 2026-07-29T10:43:18Z

+
+    if ps_process_count != status_process_count:
+        # time.sleep(1)
+        log.debug(  # TODO - once issue 793 is resolved, this should be a log.error


Make sure you link this in the relevant issue as well so it doesn't get lost

emmuhamm · 2026-07-29T10:43:24Z

+            "failure."
+        )
+        # TODO: Uncomment this once the cause of inconsistent status table printing is
+        # understood (issue 793)


Make sure you link this in the relevant issue as well so it doesn't get lost

emmuhamm · 2026-07-29T10:44:13Z

-
    dead_process_count = expected_booted_processes - alive_process_count
+    if dead_process_count > 0:
+        time.sleep(1)


why the sleep?

emmuhamm · 2026-07-29T10:46:07Z

+    # successful and we can return, otherwise, we will log the error and place the
+    # session in an error state if required.
+    in_error_state = obj.get_driver("controller").status().status.in_error
+    if not in_error_state and not put_in_error_state:


why the need for two 'in error state' flags? From a cursory look they can be combined?

eg in the line above something like

put_in_error_state = (put_in_error_state or obj.get_driver("controller").status().status.in_error)

My issues:

the logic gets a bit messy below this line, not entirely sure how the two states are different and disentangled. Might be best to try and rewrite the logic below to prevent the need for two in the first place

If the logic has to remain as such with the two different error flags, the names of the flags should be changed. Just from reading the name its not entirely clear why we have an 'in error state' and a 'put in error state'. A hint on the origin might come in handy for later debugging. Eg 'why do we need to have it be put in error state? shouldn't it be already in an error state anyway?'

emmuhamm · 2026-07-29T10:52:34Z


+    # Keep track of whether the session uses a local connectivity service
+    ctx.obj.session_uses_local_connectivity_service = (
+        session_dal.connectivity_service.host == "localhost"


Can you quickly verify that if this logic holds? Remember that in the PMaaS work we've done theres quite a lot of remapping localhost to whatever the actual hostname is.

It might be better to check for another way instead of checking via the connectivity_service.host?

emmuhamm · 2026-07-29T10:53:58Z

    # Determine the path to the test configurations
    cwd = Path(os.path.abspath(__file__))
-    test_configs = cwd.parent / ".." / "config" / "tests"
+    test_configs = cwd.parent / ".." / "config" / "drunc"


For posterity, why the change?

emmuhamm

Review 4/4

I think that should cover all the files so far.

Thanks for all of this very nice work, last time I checked it performed all the functionality that was desired so this is great!

The comments are mostly in the view of maintainability and clarity I think, rather than any fundamental functionality changes. Let me know if you have any questions and we can chat :)

Also let me know if this is ready for another round of review. Note that currently there are conflicts between this and develop.

emmuhamm · 2026-07-29T12:38:54Z

+<info name="" type="" num-of-items="6" oks-format="data" oks-version="862f2957270" created-by="gjc" created-on="thinkpad" creation-time="20231207T105549" last-modified-by="eflumerf" last-modified-on="ironvirt9.IRONDOMAIN.local" last-modification-time="20241022T175827"/>
+
+<include>
+ <!-- <file path="schema/confmodel/dunedaq.schema.xml"/> -->


emmuhamm · 2026-07-29T12:40:37Z

+<!-- Configs for apps timing out on FSM command execution -->
+<obj class="Variable" id="drunc-ft-fsm-cmd-timeout">
+ <attr name="name" type="string" val="DRUNC_FT_FSM_CMD_TIMEOUT"/>
+ <attr name="value" type="string" val="100"/>


If I understand right, this line is what sets how long the timeout is for the tests?

If so, I was wondering if it was prudent to make this number smaller? If not its just waiting for 100 seconds to pass for no reason? Or am I missing a reason on why it has to be this long?

emmuhamm · 2026-07-30T08:48:45Z

@@ -0,0 +1,205 @@
+"""
+Run a session with a nested segment application dying at the start of boot.


Minor comment: it might be worth mentioning which app is the one dying, eg.

Suggested change

Run a session with a nested segment application dying at the start of boot.

Run a session with the nested segment application (ft-nested-segment-2-application) dying at the start of the boot

L72 says dead_app_name = "ft-nested-segment-2-application", so good to tell the people which app youre killing. Especially important if we have to debug why this test fails in the future (like what we did some time ago :D )

Can you do this for all the tests?

Also maybe moving the dead_app_name up to the top of the file might help in maintenance later down the line as well.

emmuhamm · 2026-07-30T08:51:13Z

+# Connectivity Service)
+# conf_dict.connsvc_port = 12345
+
+# Collate tthe drunc config arguments into a dict to pass to the fixture


Suggested change

# Collate tthe drunc config arguments into a dict to pass to the fixture

# Collate the drunc config arguments into a dict to pass to the fixture

emmuhamm · 2026-07-30T08:51:44Z

+status
+""".split()
+
+dead_app_name = "ft-nested-segment-2-application"


See the above comment I made about moving this up to the top of the file

emmuhamm · 2026-07-30T09:18:16Z

+    def log(self, msg: str) -> None:
+        # simulate sending a message; tests don't assert on this, so store it
+        self._last_sent_msg = msg
+
+    def log_on_server(self, msg: str) -> None:


Can you also specify in the description of this PR what log and log_on_server do, and how they differ?

emmuhamm · 2026-07-30T09:21:39Z

 import requests
 from flask import Flask, Response, request
 from flask_restful import Api, Resource
+from werkzeug.serving import make_server


Ooh new package! Primarily a curiosity question, but what does werkzeug do? Does it come natively installed or do we have to include this in the dependency?

emmuhamm · 2026-07-30T09:22:29Z

+# Set up logging
 get_root_logger("info")
-log = get_logger("fake_daqapp_rest", rich_handler=True)
+log = get_logger("fake_daqapp_rest", stream_handlers=True)


Why the change from rich to stream? Im guessing its for easier log parsing but for posterity

emmuhamm · 2026-07-30T09:23:18Z

 get_root_logger("info")
-log = get_logger("fake_daqapp_rest", rich_handler=True)
+log = get_logger("fake_daqapp_rest", stream_handlers=True)
+log.setLevel("INFO")


Whats the purpose of this line? I thought that get_logger by default sets it up as INFO? Or alternatively, cant you set the log level in get_logger directly?

emmuhamm · 2026-07-30T10:05:01Z

        self.state = "INITIAL"
        self.executing_command = False
        self.log = get_logger("fake_daqapp_rest.AppState")
+        self.log.setLevel("INFO")


Same comment here, why setLevel?

Generating an environment for which the issue can be recreated

a7ba095

PawelPlesniak changed the title ~~Generating an environment for which the issue can be recreated~~ Handling exceptions when applications do not report errors but time out in stateful transitions Mar 31, 2026

WIP

c50fb9f

PawelPlesniak changed the base branch from prep-release/fddaq-v5.6.0 to develop June 4, 2026 16:05

PawelPlesniak added 15 commits June 4, 2026 18:08

Merging

db1574e

Resovling duplicate config

456a461

Adding drunc to the DUNEDAQ_DB_PATH (should get replaced but subject …

785e98b

…to agreement with SWIT as to where to host these configs)

Splitting out the failure mode testing into a separate file

7da158d

Making the configs actually pass oks_dump with the DB path

f6f8c62

Remmoving duplicate confs

8859c4b

Constructed tests for death on boot and post boot, as well as long FS…

234f185

…M timeouts and death on FSM, REQUIRES TESTING

Adding death cases

1bf07c6

Clearer conf param names

34d45b9

Configs developed, time to test

c4ead7f

Remove dupe var definitions

cf4ddb8

Fixing the broken session -> OKS var links

2a9348a

Included the configs, damned be gitignore

c8fa56e

Vars rechecked

1323cce

Moving point of timeout

abde856

PawelPlesniak added 6 commits June 5, 2026 16:59

WIP

468bef9

Root controller is now marked as in error on timeout, but this needs …

92a5314

…to be propagated down to the actual failing app level

Printing the status after marking an error state

1aa6d05

Adding log RPC

2917d12

Logs on remote server functional

899c510

WIP

6554774

PawelPlesniak added 2 commits June 9, 2026 14:36

WIP continuned

b169b22

Logfiles now have a proper format

57a8cfa

Updating the driver script for integration tests

67f39c4

PawelPlesniak marked this pull request as ready for review July 16, 2026 13:19

PawelPlesniak requested a review from emmuhamm July 16, 2026 13:19

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

2e2dcab

…ansition

PawelPlesniak added 2 commits July 17, 2026 16:57

Merge complete

bcf5fe7

Ruff linting

c9a9cc4

emmuhamm mentioned this pull request Jul 21, 2026

ps -u in unified shell broken #979

Closed

emmuhamm added 5 commits July 21, 2026 12:34

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

ddd7860

…ansition

Patch changes from integtest development

2dc0984

Clean up minor issues from other tests

99149a6

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

ee14f62

…ansition

Fix failures

15d1735

emmuhamm reviewed Jul 23, 2026

View reviewed changes

PawelPlesniak added this to the fddaq-v5.7.0 milestone Jul 23, 2026

Update src/drunc/integtest/test_failure_mode_death_on_boot_nest_app.py

113b82b

Co-authored-by: Emir Muhammad <49058518+emmuhamm@users.noreply.github.com>

PawelPlesniak commented Jul 24, 2026

View reviewed changes

PawelPlesniak added 3 commits July 24, 2026 17:13

Adding requested feature to state what command to execute when an app…

215f0c8

…lication times out

extending tests

c6717ab

Merge branch 'PawelPlesniak/IncompleteStatefulCommandTransition' of g…

291e6cc

…ithub.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition

emmuhamm reviewed Jul 24, 2026

View reviewed changes

emmuhamm reviewed Jul 29, 2026

View reviewed changes

emmuhamm mentioned this pull request Jul 29, 2026

[Feature]: Failure mode testing #869

Open

emmuhamm reviewed Jul 30, 2026

View reviewed changes

emmuhamm mentioned this pull request Jul 31, 2026

Pawel plesniak/incomplete stateful command transition DUNE-DAQ/druncschema#87

Open

9 tasks

		# The following line is outdated, but in the future when error states and their
		# recovery are better defined, we can provide better options to the user.

		@@ -0,0 +1,205 @@
		"""
		Run a session with a nested segment application dying at the start of boot.

	Run a session with a nested segment application dying at the start of boot.
	Run a session with the nested segment application (ft-nested-segment-2-application) dying at the start of the boot

	# Collate tthe drunc config arguments into a dict to pass to the fixture
	# Collate the drunc config arguments into a dict to pass to the fixture

Uh oh!

Conversation

PawelPlesniak commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

List of required branches from other repositories

Change log

Addressed issues

Other new features

Integration tests for failure modes

Suggested manual testing checklist

Developer checklist

Prior to marking this as "Ready for Review"

Final checklist prior to marking this as "Ready for Review"

Reviewer checklist

Prior to merging

Notification message for a Slack channel

For an single merge that changes the user workflow

For co-ordinated merge

Uh oh!

PawelPlesniak commented Mar 31, 2026

Uh oh!

PawelPlesniak commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PawelPlesniak commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PawelPlesniak commented Jul 16, 2026

Uh oh!

PawelPlesniak commented Jul 17, 2026

Uh oh!

PawelPlesniak commented Jul 17, 2026

Uh oh!

emmuhamm commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emmuhamm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

PawelPlesniak commented Jul 23, 2026

Uh oh!

PawelPlesniak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmuhamm left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak commented Mar 31, 2026 •

edited

Loading

PawelPlesniak commented Jun 5, 2026 •

edited

Loading

PawelPlesniak commented Jun 8, 2026 •

edited

Loading

emmuhamm commented Jul 23, 2026 •

edited

Loading

emmuhamm left a comment •

edited

Loading