Skip to content

Add stack health monitor contract prototype - #36

Merged
makers-pet merged 2 commits into
makerspet:mainfrom
xbattlax:health-monitor-contract-xbattlax
Jul 25, 2026
Merged

Add stack health monitor contract prototype#36
makers-pet merged 2 commits into
makerspet:mainfrom
xbattlax:health-monitor-contract-xbattlax

Conversation

@xbattlax

@xbattlax xbattlax commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to the maintainer comment on #33 about generalizing /cmd_vel holding into component health monitoring.

This PR adds a namespaced health-monitor/xbattlax stack-health monitor contribution and records the public draft contract in docs/SOFTWARE_INTERFACES.md.

What it includes:

  • dependency-free stack-health aggregator core, now shared by the ROS2 package and simulator
  • ROS2 oomwoo_health_monitor package with health_monitor_node and launch file
  • JSON-over-std_msgs/msg/String adapter for /oomwoo/health/roster and /oomwoo/health/component
  • /oomwoo/health/stack publication on each monitor tick
  • /oomwoo/health/mcu_heartbeat publication only while the aggregate state is healthy
  • roster contract for critical vs. advisory components
  • freshness and health-level checks for component heartbeats
  • fail-safe arming window before MCU heartbeat emission
  • immediate heartbeat withholding on missing, stale, or unhealthy critical components
  • advisory-fault reporting without stopping the robot
  • deterministic JSONL simulator for boot, arming, stale critical node, recovery, and advisory-only faults
  • unit tests for core deadman behavior and ROS2 node adapter behavior using lightweight stubs
  • docs for the stack watchdog contract and recovery-safety integration

Review Context

This keeps /cmd_vel holding as a motion-quality fix, but moves the robot-wide safety question to a general stack-health deadman: components publish work-path heartbeats, a health monitor checks the active roster, and only then is one MCU-facing stack heartbeat emitted.

Verification

  • python3 -m unittest discover -s contributions/health-monitor/xbattlax/tests -p 'test_*.py'
  • python3 contributions/health-monitor/xbattlax/tools/sim_health_monitor.py
  • python3 -m py_compile contributions/health-monitor/xbattlax/oomwoo_health_monitor/oomwoo_health_monitor/core.py contributions/health-monitor/xbattlax/oomwoo_health_monitor/oomwoo_health_monitor/health_monitor_node.py contributions/health-monitor/xbattlax/tools/sim_health_monitor.py
  • git diff --check

@xbattlax

Copy link
Copy Markdown
Contributor Author

Follow-up update pushed in 24f3cae:

  • moved the health-monitor core into a real ROS2 oomwoo_health_monitor package
  • added health_monitor_node consuming /oomwoo/health/roster and /oomwoo/health/component
  • publishes /oomwoo/health/stack every tick and /oomwoo/health/mcu_heartbeat only when the aggregate state is healthy
  • added launch/package metadata plus ROS2 adapter tests using lightweight stubs
  • updated the PR body and README/docs with build/run details

Review/check sweep before posting:

  • issue comments: none
  • review comments: none
  • reviews: none
  • GitHub checks: none reported for this branch

Verification:

  • python3 -m unittest discover -s contributions/health-monitor/xbattlax/tests -p 'test_*.py'
  • python3 contributions/health-monitor/xbattlax/tools/sim_health_monitor.py
  • python3 -m py_compile contributions/health-monitor/xbattlax/oomwoo_health_monitor/oomwoo_health_monitor/core.py contributions/health-monitor/xbattlax/oomwoo_health_monitor/oomwoo_health_monitor/health_monitor_node.py contributions/health-monitor/xbattlax/tools/sim_health_monitor.py
  • git diff --check

@makers-pet
makers-pet merged commit be86cf5 into makerspet:main Jul 25, 2026
@makers-pet

Copy link
Copy Markdown
Collaborator

xbattlax - thank you again!

@makers-pet

Copy link
Copy Markdown
Collaborator

xbattlax, my Claude has reviewed the PR, found a few things. Please see below.

=== Claude ===

  • QoS mismatch on the roster (real bug). The contract calls the roster "latched," but the node subscribes with default (volatile) QoS. A monitor that starts after the roster is published will never receive it → stays no_roster → never emits the MCU heartbeat → the MCU keeps the robot stopped. It fails safe (won't move) but is non-functional on a startup race. Fix: transient_local durability on the roster subscription (and likely on the stack/mcu publishers if anything latches them).
  • Clock contract is unspecified. Freshness compares component stamp_sec to the monitor's ROS clock. Under use_sim_time, a producer stamping with wall-clock would break freshness. The contract doc should state that stamp_sec must be the same clock (ROS/sim time) as the monitor.

@makers-pet

Copy link
Copy Markdown
Collaborator

xbattlax, would it be possible for you to self-host ROS2 packages (health-monitor, including docs) - and replace the code in contributions/health-monitor/xbattlax with a README having a link to your xbattlax/health-monitor repo?

This would allow me/anyone to integrate it, try it out, vendor it into oomwoo-install Dockerfile.

@xbattlax

Copy link
Copy Markdown
Contributor Author

Thank you for both follow-ups. Done:

  • created the standalone repository: https://github.com/xbattlax/health-monitor
  • published a stable vendoring release: https://github.com/xbattlax/health-monitor/releases/tag/v0.1.0
  • fixed the roster startup race with TRANSIENT_LOCAL, RELIABLE, depth 1 QoS and added an adapter regression test
  • made /oomwoo/health/stack transient-local for late diagnostics subscribers
  • kept component and MCU heartbeat topics volatile intentionally: a late MCU bridge must wait for a new live heartbeat rather than receive a previously healthy latched heartbeat
  • removed the monotonic-clock fallback and documented that component stamp_sec and the monitor must use the same ROS clock; future/mismatched timestamps fail safe as stale
  • installed the detailed contract docs with the ROS package
  • opened Move health monitor to standalone repository #47 to replace the duplicated contribution with the repository/release link and update SOFTWARE_INTERFACES.md

Verification in the standalone repository:

  • 12 unit/adapter tests pass
  • deterministic simulator passes
  • ROS 2 Jazzy colcon build passes
  • installed contract-doc check passes
  • GitHub Actions core and Jazzy jobs pass
  • actionlint passes

Thanks for catching the QoS race and for suggesting the repository split.

@xbattlax

Copy link
Copy Markdown
Contributor Author

One more follow-up to the vendoring request: the runtime integration is now open
at makerspet/oomwoo-install#3 and its CI is fully green.

It pins xbattlax/health-monitor to v0.1.0 in both the Pi Jazzy installer and
the development Dockerfile. The native arm64 job performs the complete install
and build, verifies the exact release, and exercises the startup-race/deadman
path over real ROS 2 topics.

This should complete the path from the prototype here to a reproducible package
that can be installed and tried from oomwoo-install. Thanks again for the
clear direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants