Skip to content

fix(rpc): additional logs#1175

Open
wyzula-jan wants to merge 12 commits into
mainfrom
fix/rpc-timeout-logging
Open

fix(rpc): additional logs#1175
wyzula-jan wants to merge 12 commits into
mainfrom
fix/rpc-timeout-logging

Conversation

@wyzula-jan
Copy link
Copy Markdown
Contributor

@wyzula-jan wyzula-jan commented May 22, 2026

Description

CHANGE TO SHUTDOWN FROM CLIENTtest_available_widgets reveled that with additional logs there is a delay in shutdown and the previous logic was not robust enough. The shutdown was reviewed and adjusted.

Adds RPC timeout diagnostics for BEC Widgets so client, dispatcher, and RPC server logs can be correlated by request_id.

Also:

  • Increases default GUI RPC timeout from 5s to 60s
  • Disables noisy bec_lib.scan_items logs in BEC Widgets
  • Keeps flOMNI GUI calls on the shared client default timeout instead of local custom timeouts

What Is Logged

Client side:

  • RPC request sent
  • RPC response received
  • RPC response timeout

Dispatcher side:

  • RPC request received before Qt callback emit
  • Warning if request already passed the client deadline at dispatcher receipt

RPC server side:

  • RPC request entered on_rpc_update
  • Warning if it entered after the client deadline
  • RPC execution finished
  • Warning if execution finished after the client deadline
  • RPC response published
  • RPC result serialization retry delayed by QTimer.singleShot

Important Log Fields

  • request_id: correlate logs for one RPC call
  • method: RPC method/action
  • receiver / gui_id: target and handling GUI server ids
  • target_gui_id: target widget/object id
  • object_name: client-side proxy name
  • timeout: client RPC timeout
  • dispatch_latency_s: client send to dispatcher receipt
  • stale_on_dispatch: dispatcher received after client deadline
  • receive_latency_s: client send to on_rpc_update
  • stale_on_receive: RPC handler started after client deadline
  • execution_duration_s: RPC method execution time
  • response_after_client_deadline: execution finished after client deadline
  • accepted: response success state
  • args / kwargs: bounded RPC method arguments → commit left in case we would like to reenable

Additional Comments

Also exclude launcher check for non-parented widgets for BECMainWindow which was triggering lot of false positive logs since MainWindow do not need to be parented to launch window.

This distinguishes Redis/dispatcher delay, Qt/RPC handler delay, slow/stuck widget execution, and responses published too late for the client.

Copilot AI review requested due to automatic review settings May 22, 2026 12:49
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds richer, structured logging around GUI RPC request sending, server-side receipt/execution, and response publishing to improve observability and debugging of RPC timing and failures.

Changes:

  • Add client-side RPC logs and attach timing/identifying metadata (sent_at, deadline, timeout, etc.) to outgoing RPC messages.
  • Add server-side logs for RPC receipt latency, execution duration, and “stale/late” request/response detection.
  • Add unit tests validating the new logging behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
bec_widgets/utils/rpc_server.py Adds receipt/execution/response logs, timing helpers, and “stale/late” warnings on the server side.
bec_widgets/cli/rpc/rpc_base.py Adds request/response logging and enriches outgoing RPC metadata with timing/identity fields.
tests/unit_tests/test_rpc_server.py Adds tests asserting server logs for publish status and late-deadline handling.
tests/unit_tests/test_rpc_base.py Adds a test asserting client-side timeout logging + metadata population on timeout.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread bec_widgets/utils/rpc_server.py
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch 2 times, most recently from 00fa1a9 to bfea216 Compare May 22, 2026 13:00
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 22, 2026

Benchmark comparison

Threshold: 20% (lower is better).
Result: 0 regression(s), 0 improvement(s) beyond threshold.

No benchmark regression exceeded the configured threshold.

No benchmark improvement exceeded the configured threshold.

All benchmark results
Benchmark Baseline Current Change Status
BEC IPython client with companion app 5.91908 s 6.05135 s +2.23% ok
BEC IPython client without companion app 2.21378 s 2.17367 s -1.81% ok
Import bec_widgets 0.0143291 s 0.0140835 s -1.71% ok
tests/unit_tests/benchmarks/test_dock_area_benchmark.py::test_add_waveform_to_dock_area 0.151597 s 0.156541 s +3.26% ok

@codecov
Copy link
Copy Markdown

codecov Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Comment thread bec_widgets/cli/rpc/rpc_base.py Outdated
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch from 2440df4 to 5b54ab1 Compare May 22, 2026 15:42
@wyzula-jan wyzula-jan requested a review from a team May 22, 2026 16:05
@wyzula-jan wyzula-jan enabled auto-merge (rebase) May 22, 2026 16:05
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch from 5b54ab1 to f9995c8 Compare May 26, 2026 08:03
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch from f9995c8 to a37eee0 Compare May 26, 2026 08:10
wakonig
wakonig previously approved these changes May 26, 2026
@wyzula-jan wyzula-jan marked this pull request as draft May 27, 2026 08:04
auto-merge was automatically disabled May 27, 2026 08:04

Pull request was converted to draft

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Comment thread tests/unit_tests/test_client_utils.py
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch 2 times, most recently from ff729c2 to 0e73447 Compare May 27, 2026 12:17
@wyzula-jan wyzula-jan marked this pull request as ready for review May 27, 2026 12:56
@wyzula-jan wyzula-jan requested a review from wakonig May 27, 2026 12:56
Comment thread bec_widgets/applications/companion_app.py
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch from 0e73447 to 6670e47 Compare May 28, 2026 09:11
@wyzula-jan wyzula-jan requested a review from wakonig May 28, 2026 09:13
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch from 6670e47 to a37eee0 Compare May 28, 2026 11:34
@wyzula-jan wyzula-jan force-pushed the fix/rpc-timeout-logging branch from a976abb to 674048d Compare May 28, 2026 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants