ERS for drunc

[PawelPlesniak]

Description

drunc needs to feed information from each run control process (process manager, controller, session manager, etc) about what errors are being generated at each stage of execution. The current architecture of ERS is as follows:

• An application can report an error as a ERS message (defined here), which is published to our kafka service
• We have a microservice (defined here) that pulls the ERS messages to the PostGreSQL database.

Currently:

• drunc does not publish anything to this service
• This service provides a way to observe the system state without having to look through the logs. As such, we want to publish this information using a parallel or integrated mechanism.

The proposed solution is as follows

• Currently, logging in python applications is handled using the native python logging package. This is not a part of the DUNE-DAQ logging package.
• The messages are handled in this logging with the use of handlers. Currently, we already make use of stdout, stderr, and rich based handlers.
• To publish ERS messages, a handler will be defined that can be added to the current logging infrastructure. This will be defined in the erskafka repo.

Notes:

• Currently we have a per-repository setup for handling logging in python applications. The idea is to standardize this, as has been defined in my [daqpytools](
  Having https://github.com/PawelPlesniak/daqpytools) repo. Integrating into the DUNE-DAQ organization will be extrememly helpful here, as this will be deployable for the other python applications that will benefit from ERS, such as configuration generation and microservices.

Testing Suggestions

There will be unit tests written that can publish to ERS. with pytest, there are optional flags that can be set that run specific tests. For manual testing, one will be able to run a session, kill a process, and see that the process will appear as as unexpected death thread on ERS.

Correlated Issues and/or PRs

N/A

Useful Coordination

Co-ordination will happen with @jcfreeman2 to determine when an appropriate time to integrate daqpytools is, based on that the PR will be prepared. This will rely on work that exists in erskafka.

Once the PR is ready, we can extend it by discussing with a select group as to what notifications we want to see in the ERS feed. Preliminary notifications include

• failed FSM transitions
• processes dying unexpectedly
• entering error state

Impact on Developers or Users

Users will be able to see notifications on the ERS feed when running the ssh-CERN-kafka process manager.

Changes in System Behavior

Additional KafkaPublishers will be generated. This will be slightly more resource intensive.

Needed Changes in Testing

N/A.

[PawelPlesniak]

Add a check for the unified shell for whether the env var DUNEDAQ_ELISA_LOGBOOK_APPARATUS is set, if not then log it both to the unfiied shell and to ERS

[PawelPlesniak]

In progress

[emmuhamm]

New logger tools implemented in the daqpytools repository

TODO:

• Understand the repository
• Import into drunc
• Replace the old drunc logger lines with the new daqpytools logger
• test to see intended behaviour in drunc
• check drunc repo to see what errors we want to send to kafka (make sure not too verbose)