Add objective integration tests for all example_ws configs#534
Add objective integration tests for all example_ws configs#534
Conversation
JWhitleyWork
left a comment
There was a problem hiding this comment.
Is a good start but needs some tweaking.
| moveit_license_key: ${{ secrets.STUDIO_CI_LICENSE_KEY }} | ||
|
|
||
| # Per-config objective integration tests (run in parallel via matrix) | ||
| objective-integration-test: |
There was a problem hiding this comment.
The downside to doing this with a matrix is that it launches a separate Github Actions runner for each invocation of the matrix. If we do these all as a single colcon test --packages-select <list_of_packages>, many of them can still be run in parallel but we only spin up an instance, install dependencies, clone stuff, etc. once. Also, we should be doing this test for at least Humble and Jazzy (in a matrix because they are completely different environments).
| ament_lint_auto_find_test_dependencies() | ||
| ament_add_pytest_test( | ||
| objectives_integration_test test/objectives_integration_test.py | ||
| TIMEOUT 600 |
There was a problem hiding this comment.
10 minutes? We usually give 10 minutes for all of colcon test. Does it actually take anywhere near this long?
Replicate the lab_sim objectives_integration_test.py pattern across all 11 remaining sim config packages: kitchen_sim, april_tag_sim, grinding_sim, dual_arm_sim, factory_sim, hangar_sim, kinova_sim, space_satellite_sim, space_satellite_sim_camera_cal, mock_sim, and multi_arm_sim. Each test auto-discovers objectives via SystemConfigParser and runs them, with per-config skip lists for objectives requiring user input or GPU/ML inference. New objectives are automatically tested — if one requires skipping, CI fails and the PR author adds it to the skip list. Also adds a CI matrix job to run each config's tests in parallel, and adds missing ament_cmake_pytest + moveit_pro_test_utils test deps (including for the existing lab_sim test). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
863b4ca to
a853397
Compare
|
@JWhitleyWork I realistically don't have time to prioritize this any time soon, and I don't think you do either. WDYT: is this worth handing off (maybe have Shaur take over?), or leave it sit for now with the risk of it going stale, or just close this PR and let someone restart the work when there's a new ticket/priority to do this? (or other suggestion?) |
|
I think this is very worthwhile. We have very few tests on the example workspace and no real integration tests. I think @shaur-k would be the perfect person to work on this as most of the issues here are with CI changes (the current failures using the external workflow, matrix vs non-matrix, etc.). Probably for next sprint. |
Summary
objectives_integration_test.pyto 11 config packages (kitchen_sim, april_tag_sim, grinding_sim, dual_arm_sim, factory_sim, hangar_sim, kinova_sim, space_satellite_sim, space_satellite_sim_camera_cal, mock_sim, multi_arm_sim)SystemConfigParserand runs them with per-config skip lists for user-input/ML/GPU objectivesworkspace_integration_testreusable workflowament_cmake_pytest+moveit_pro_test_utilstest deps across all configs (including existing lab_sim)Known TODOs
integration-test-in-studio-containerjob still runs all tests including these, causing double-runs. Consider adding--packages-skipfor config packages to that jobTest plan
main_tree_to_executeXML attributes — all match🤖 Generated with Claude Code