Skip to content

feat(bench): resource-monitor integration for benchmark orchestration (#159)#160

Open
pesap wants to merge 7 commits intomainfrom
feat/torc-benchmark-integration
Open

feat(bench): resource-monitor integration for benchmark orchestration (#159)#160
pesap wants to merge 7 commits intomainfrom
feat/torc-benchmark-integration

Conversation

@pesap
Copy link
Copy Markdown
Collaborator

@pesap pesap commented Apr 13, 2026

Summary

Implements foundation for torc-based benchmark orchestration (issue #159).

Changes

  • Added `torc` feature flag to `arco-bench` with conditional compilation
  • Created runner module structure: `torc.rs` (torc feature) / `local.rs` (default)
  • Added optional dependencies: `torc-client` (git), `reqwest`
  • Created `examples/benchmark-workflow.kdl` showing:
    • 3-job workflow (compile → solve → report)
    • Resource specifications
    • Dependency chaining
  • Verified both compile modes pass:
    • `cargo check -p arco-bench` (without feature)
    • `cargo check -p arco-bench --features torc` (with feature)

Next Steps (follow-up PRs)

  • HTTP client implementation (torc API client)
  • Result adapter (convert torc job results to arco-bench format)
  • CI integration with torc server container

API Documentation

Torc API spec researched and documented at `/tmp/torc-api-summary.md`:

  • POST /workflows - create workflow
  • POST /bulk_jobs - submit jobs
  • GET /jobs/{id} - poll status
  • GET /results - extract metrics

Closes part of #159

@pesap pesap force-pushed the feat/torc-benchmark-integration branch from facbd8f to 0d370e0 Compare April 13, 2026 04:39
@github-actions github-actions bot removed the docs label Apr 13, 2026
@pesap pesap force-pushed the feat/torc-benchmark-integration branch 2 times, most recently from c3e7aa8 to 64febb6 Compare April 13, 2026 04:55
Adds optional torc integration infrastructure:
- torc feature flag in arco-bench with conditional compilation
- runner module structure (torc/local backends)
- example benchmark-workflow.kdl showing dependency chains
- reqwest and torc-client as optional deps
@pesap pesap force-pushed the feat/torc-benchmark-integration branch from 64febb6 to 2299096 Compare April 13, 2026 17:25
@pesap
Copy link
Copy Markdown
Collaborator Author

pesap commented Apr 13, 2026

Code Review Summary

Reviewed by Hermes Agent 🤖

This PR implements the foundation for benchmark orchestration with resource monitoring. All tests pass, code compiles cleanly, and the design is solid.

🔴 Critical

None

⚠️ Warnings

None

💡 Suggestions (non-blocking)

  1. github_action.py:183-208load_monitor_summary() has deeply nested conditionals. Consider early returns or a parsing helper to flatten.

  2. github_action.py:251-282_run_target() builds paths then calls _monitor_subprocess() which imports heavy rmon modules inline. Consider lazy imports at top-level with TYPE_CHECKING guards, or document why inline imports are needed (appears to be for optional dependency handling).

  3. test_github_action.py — Good coverage, but consider adding tests for:

    • Invalid/malformed JSONL in monitor output (error handling)
    • Empty scenario list behavior
    • Command building edge cases
  4. benchmarks.yaml — The timeout-minutes: 20 at job level + timeout-minutes: 30 workflow level means total run could hit 30 min before the step timeout kicks in. Consider aligning or documenting the intent.

✅ Looks Good

  • Clean dataclass-based design in github_action.py
  • Proper fallback from resource_monitor to arco_bench peak RSS
  • Good CLI interface in run_monitored_benchmarks.py
  • Tests pass (4/4 pytest, cargo check clean)
  • Benchmark workflow integration with github-action-benchmark is well configured
  • Tree-sitter KDL grammar changes look correct (math expression support)

Notes

  • Tree-sitter changes include generated parser.c (11k+ lines) — this is expected for grammar changes
  • PR description mentions "torc integration" but this PR is actually the resource monitoring foundation. The torc HTTP client will come in follow-up PRs as noted.

Verdict: Ready to merge


Automated review completed

pesap added 5 commits April 13, 2026 20:34
Removing the arco-bench crate in preparation for simpler CLI-based
benchmark approach using rmon resource monitoring.

Deleted:
- crates/arco-bench/Cargo.toml
- crates/arco-bench/README.md
- crates/arco-bench/src/ (main.rs, comparison.rs, reporting.rs, scenarios.rs, types.rs)
Adds a simple Python-based benchmark runner that wraps arco CLI
commands with rmon resource monitoring.

Added:
- scripts/bench.py - CLI benchmark runner (~80 lines)
- examples/stress-test/README.md - Stress test guide
- Updated .github/workflows/benchmarks.yaml to use new runner

Benefits:
- Tests real user workflows (arco compile/solve)
- Uses external rmon for accurate resource measurement
- Integrates with github-action-benchmark
@pesap pesap changed the title feat(bench): torc integration for benchmark orchestration (#159) feat(bench): resource-monitor integration for benchmark orchestration (#159) Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant