ORBIT-2: honor ORBIT2_MAX_BATCHES on the EDM training path#39
Merged
Conversation
The doc referenced a @OMNIHUB_TOOLS_DIR@ placeholder that does not exist; the template and sbatch sed substitution use @PERF_TOOLS_DIR@. Co-Authored-By: Claude <noreply@anthropic.com>
The EDM path exec'd launch/train_edm.py directly, whose batch loop is inline in main() with no cap — so ORBIT2_MAX_BATCHES was silently ignored (unlike the studio intermediate_downscaling path via run_orbit2_train.py, and HydraGNN). Add run_orbit2_train_edm.py, which wraps train_edm.main() and caps each epoch by patching IterDataModule.train_dataloader. Point both sbatch scripts' EDM exec at the wrapper; the profiler hook-runner call is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
The analyst/verifier prompts read manifest keys that the sbatch launcher does not write (perf_run_dir/jobid/trace_paths/omnistat_db_path) and hardcoded the perf-inspect venv name, so a fresh run needed manual fix-ups. Align to the actual keys (job_dir/job_id/trace_dir/omnistat_db), resolve the venv via OMNISTAT_VENV with the perf-inspect default, document that omnistat-inspect writes its payload to --scratch-dir (stdout is query-stats only), and fix the verifier's PromQL example to use the rmsjob_info join. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR ensures ORBIT-2’s Bayes-CAST EDM training path honors ORBIT2_MAX_BATCHES by introducing a dedicated launcher wrapper, and aligns perf-analysis agent prompts with the manifest schema actually written by the sbatch wrapper (plus a small HydraGNN doc fix).
Changes:
- Add
run_orbit2_train_edm.pyand update ORBIT-2 sbatch scripts to use it soORBIT2_MAX_BATCHEScaps batches/epoch on the EDM path. - Update ORBIT-2 perf-analysis agent instructions to use
manifest.job_dir,manifest.trace_dir,manifest.omnistat_db,manifest.job_id, and to activate the venv viaOMNISTAT_VENV. - Fix HydraGNN perf-analysis launcher docs to use the correct
@PERF_TOOLS_DIR@placeholder.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| material_science/models/HydraGNN/recipes/perf-analysis/agents/launcher.md | Fixes a stale placeholder name in Omnistat config templating instructions. |
| earth_science/models/ORBIT-2/recipes/perf-analysis/agents/tracelens_verifier.md | Updates trace discovery instructions to match manifest.trace_dir. |
| earth_science/models/ORBIT-2/recipes/perf-analysis/agents/tracelens_analyst.md | Uses OMNISTAT_VENV and updates manifest key usage for perf-run + trace discovery. |
| earth_science/models/ORBIT-2/recipes/perf-analysis/agents/omnistat_verifier.md | Updates PromQL guidance/query and adjusts how the VM port is sourced. |
| earth_science/models/ORBIT-2/recipes/perf-analysis/agents/omnistat_analyst.md | Updates manifest keys (omnistat_db, job_id) and venv activation to match sbatch output. |
| earth_science/models/ORBIT-2/examples/sbatch_train_perf_amd.sh | Switches EDM direct launch to run_orbit2_train_edm.py so batch caps are honored. |
| earth_science/models/ORBIT-2/examples/sbatch_train_amd.sh | Switches EDM direct launch to run_orbit2_train_edm.py so batch caps are honored. |
| earth_science/models/ORBIT-2/examples/run_orbit2_train_edm.py | New wrapper that stubs gptl4py, initializes distributed, and caps EDM train batches by patching the dataloader. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
ashwinma
added a commit
that referenced
this pull request
Jun 18, 2026
Brings in the perf recipes landed on main (#37 HydraGNN perf characterization + sysopt loop, #39 ORBIT-2 EDM ORBIT2_MAX_BATCHES cap + perf-analysis prompt fixes). Conflicts resolved by taking main's content and applying this branch's omnihub naming: PERF_TOOLS_DIR->OMNIHUB_TOOLS_DIR, perf-inspect->omnihub-inspect, @PERF_TOOLS_DIR@->@OMNIHUB_TOOLS_DIR@, perf_tools.dir->omnihub.tools_dir, and the omnihub: cluster-config schema. Canonical /shared/omnihub/tools paths preserved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
train_edm.pydirectly, whose batch loop has nocap, so
ORBIT2_MAX_BATCHESwas silently ignored. Addrun_orbit2_train_edm.pyto wrap
train_edm.main()and cap each epoch; both sbatch scripts now exec it.actually writes, and resolve the venv via
OMNISTAT_VENV.@OMNIHUB_TOOLS_DIR@reference in HydraGNNlauncher.md.Validation
Job 14742 (1×8 MI355X,
ORBIT2_MAX_BATCHES=10) completed rc=0 with exactly10 batches/epoch (was ~5112 uncapped); full perf-analysis chain ran on it.
🤖 Generated with Claude Code