feat(workflow): add stage-1 stepwise reward plumbing - #1
Closed
Fyrgo8 wants to merge 36 commits into
Closed
Conversation
…oject#1409) - add WeChat group QR code image to assets/figures - include community engagement section with links to GitHub discussions - reference Community Repository for meeting materials and details Signed-off-by: mingcheng <mingcheng@apache.org>
* docs(cli): sync cli reference for icepop and kpop params * feat(examples): add icepop and kpop configs for gsm8k * style: fix end-of-file newline in icepop/kpop configs * fix(icepop/kpop): detach imp_ratio and KL inputs to prevent gradient leak * fix(kpop): use <= for KL threshold to include boundary * refactor: unify icepop/kpop into rejection_sampling with binary_kl metric - Add binary_kl metric to RejectionSamplingConfig for KPop - Remove enable_icepop/icepop_alpha/icepop_beta/enable_kpop/kpop_phi - Update gsm8k_icepop.yaml to use rejection_sampling with ratio+lower+upper - Update gsm8k_kpop.yaml to use rejection_sampling with binary_kl+upper - Sync CLI docs
areal-project#1408) ## Summary - head_version/tail_version now per-sample, filtered by loss_mask==1 - fixes head_version always being -1 due to input token version placeholders - adds version_rle field (run-length encoded per-token version sequence) - adds _split_trajectory_for_dump helper for correct multi-turn prompt_end - adds segments field for multi-turn agent trajectory analysis ## Context The previous `_dump_trajectory` had three issues: 1. `head_version = min(versions)` always returned -1 because input tokens use -1 as placeholder 2. `prompt_end = seqlen - sum(mask)` is incorrect for multi-turn agent rollouts where loss_mask is interleaved 3. No per-token version granularity was persisted for cross-version analysis --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add IcePop/KPop feature introduction * style: fix mdformat * docs: remove IcePop/KPop from Examples table
…real-project#1429) VLLMBackend.build_generation_request omitted gconfig.frequency_penalty and gconfig.stop, while the sibling SGLangBackend forwards both. Both are GenerationHyperparameters (and neither is in _OPENAI_UNSUPPORTED_ARGS), and vLLM's OpenAI-compatible /v1/completions and /v1/chat/completions endpoints accept them. As a result a user who set frequency_penalty (anti-repetition) or stop strings had them honored on SGLang but silently ignored on vLLM, changing the sampling distribution between backends. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…GROUP (areal-project#1414) On some hardware/driver combos (e.g. certain Ascend variants), torch.distributed.batch_isend_irecv hangs during weight update. Add AWEX_WU_USE_GROUP env var so callers can fall back to per-op send/recv. Usage: defaults to 1 (use_group=True). Set AWEX_WU_USE_GROUP=0 to bypass the group path. Applied at the three adapter call sites: fsdp/megatron/sglang execute_weight_update(). Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…real-project#1393) * feat: disable megatron grad buffers CPU backup to save host memory Add disable_grad_buffers_cpu_backup option to MegatronEngineConfig to skip CPU backup for gradient buffers during offload in colocated training. Gradient buffers are recomputed each training step, so they don't need to be backed up to CPU memory during offload. This saves host memory (∼5.9GB for typical configs) and reduces offload/onload overhead. - Add config flag disable_grad_buffers_cpu_backup (default: False) - Call DDP.offload_grad_buffers() before TMS pause to release grad storage - Call DDP.restore_grad_buffers() after TMS resume to reallocate and zero Tested: numerical equivalence confirmed, host memory saved ≈ grad buffer size * docs: add disable_grad_buffers_cpu_backup to cli_reference * fix: disable synchronize/empty_cache in offload/restore_grad_buffers
…project#1436) available_range subtracted the full exclude_ports count regardless of whether those ports lay within [min_port, max_port]. Excluded ports outside the range deflated the availability count and could raise a spurious ValueError even when every in-range port was free. Intersect exclude_ports with the range before subtracting. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…areal-project#1430) * fix(reward): score non-string answers in clevr_count_70k_reward_fn clevr_count_70k_reward_fn did not str()-coerce its inputs or guard against errors, unlike the sibling reward fns (gsm8k, geometry3k). A non-string answer (e.g. an int) made ans.strip() raise AttributeError, which WorkflowExecutor catches and uses to reject the whole trajectory instead of scoring the sample — and even a matching completion was lost rather than scored 1.0. Coerce completions and answer to str and wrap the body in try/except, matching the sibling reward fns. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(example): apply the same guard to the clevr GRPO example reward fn The example defines its own clevr_count_70k_reward_fn (referenced via workflow_kwargs) that still had the old logic. Mirror the built-in fix: str-coerce completions/answer and guard, so a non-string answer is scored rather than raising and dropping the trajectory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…-project#1403) * feat(megatron): make MTP head opt-in to support Qwen3.6 MoE RL Qwen3.6 MoE checkpoints store their MTP (multi-token-prediction) experts in a fused layout that megatron-bridge 0.4.x cannot export, so update_weights crashes with "Object must exist on at least one PP rank" at step 1. MTP is an inference-only head, not part of the RL objective and unused by the rollout, so it should not be built for RL training. Add MegatronEngineConfig.enable_mtp (default False). In make_mcore_model (megatron-bridge path), drop MTP (provider.mtp_num_layers=None) when the model has one and enable_mtp is False; raise if enable_mtp is True but the model has no MTP head. Set True for SFT or bridge builds that support the model's MTP format. Refs: areal-project#1398 * docs(examples): add Qwen3.6-35B-A3B megatron geometry3k GRPO config GRPO recipe for Qwen3.6-35B-A3B on geometry3k with the megatron actor (megatron-bridge) + vLLM rollout, demonstrating the new megatron.enable_mtp=False switch (MTP dropped for RL). Refs: areal-project#1398 * docs(megatron): refine enable_mtp help + Qwen3.6 example (review) Address PR areal-project#1403 review: - Scope the enable_mtp help to bridge_type=megatron-bridge (the flag is a no-op on the mbridge/registry paths) and drop the unwired SFT claim. - Add PYTORCH_ALLOC_CONF=expandable_segments to the Qwen3.6 example to avoid 35B-MoE OOM. Refs: areal-project#1398 * fix(engine): keep non-MTP weights in HF export when MTP head is dropped With enable_mtp=False the bridge export yields no mtp.* tensors, and megatron-bridge's save_generator with strict=True silently skips every source shard containing an MTP key -- discarding the non-MTP weights packed in those shards (Qwen3.6-35B lost lm_head + 2 layers; 27B lost ~16 layers) while rebuilding a consistent-looking index. Pass strict=False to save_hf_pretrained when the MTP head was dropped so incomplete shards are written with all present keys (only mtp.* is absent, as intended), zero the MTP layer counts in the exported config.json so external loaders do not fabricate an MTP head over missing weights, and rebuild the safetensors index from the shard files' actual contents (the bridge's strict=False path leaves ghost mtp.* entries and a stale metadata.total_size). Refs: areal-project#1398
* feat(ppo): add CISPO advantage estimator (MiniMax-M1) PPO/GRPO clipping zeroes the gradient of any token whose importance-sampling ratio leaves the clip band: `min(r*A, clip(r)*A)` is constant in theta there. MiniMax-M1 (https://arxiv.org/abs/2506.13585, Eq. 4-5) observes those are disproportionately the low-probability "fork" tokens (`However`, `Wait`, ...) that steer reasoning -- exactly the tokens reasoning RL needs gradient on -- and instead clips the IS *weight* under stop-gradient while keeping the gradient on every token's `log pi_theta`. ScaleRL (arXiv:2510.13786 Eq. 4) adopts the same surrogate. AReaL had grpo/gspo/ppo/sapo but no CISPO. Per token, opt-in via `actor.use_cispo_loss=true`: ratio = exp(logprobs - proximal_logprobs) ratio_clipped = clip(ratio, 1 - eps_clip, 1 + eps_clip_higher) # stop-grad pg_loss = -sg(ratio_clipped) * advantages * logprobs Advantages are never clipped. The clip reuses the existing delta-from-1 `eps_clip` / `eps_clip_higher` plumbing (same convention as GSPO); CISPO is canonically single-sided, so the recommended setting is `eps_clip=1.0` (lower bound 0) with `eps_clip_higher=4.0` for the wide MiniMax-M1 range. Shape of the change -- additive, fits AReaL's existing estimator seam: - `functional.py`: `cispo_loss_fn` (+ export), reusing the token-mean reduction and the PPO-compatible stat schema so the existing clip-stat logging path applies unchanged; `clip_mask` reports band-exit on either side (under CISPO no clip zeroes the loss, so band-exit -- not loss-affecting clip -- is the meaningful metric). - `actor.py`: `grpo_loss_fn` dispatches to CISPO before SAPO/PPO. - `cli_args.py`: `PPOActorConfig.use_cispo_loss` with `__post_init__` validation (mutually exclusive with SAPO, requires `eps_clip_higher > 0`, token-level importance sampling only) + regenerated CLI docs (en/zh). Defaults unchanged (`use_cispo_loss=False`) -> byte-identical to before. `tests/test_cispo_loss.py` pins the two defining invariants, each across a PPO-like band (0.2/0.28) and the wide MiniMax band (1.0/4.0): the closed-form surrogate value + clip indices, and gradient routing (`logprobs.grad == -sg(clip(ratio)) * A / N`, zero gradient through the IS-ratio path -- mutation-verified to fail if the stop-gradient detach is dropped), plus config validation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ppo): support decoupled loss for CISPO via rejection sampling The CISPO branch ignored use_decoupled_loss / rejection_sampling, silently dropping the off-policy correction in exactly the async/stale regime CISPO targets. CISPO already anchors its clipped IS ratio at pi_proximal; thread the behavior logp + rejection_sampling through cispo_loss_fn and rescale each token's surrogate by the detached pi_proximal/pi_behave weight, mirroring ppo_actor_loss_fn. Both factors are stop-gradient, so the estimator stays -sg(behave_imp_weight * ratio_clipped) * A * grad(log pi_theta). Addresses the maintainer review on the dispatch bypassing the decoupled check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Default fp32 master weights (PR areal-project#1369) push Qwen2.5-VL-3B beyond A100-40G capacity at the first AdamW step. Pin the test to bf16 storage with adam_bf16 (Kahan summation) so it runs on 40G runners. ## Description <!-- Provide a clear and concise description of what this PR does --> ## Related Issue <!-- Link to the issue this PR addresses. PRs should be related to a well-templated issue. --> Fixes #(issue) ## Type of Change <!-- Select ONE that best describes this PR --> - [x] 🐛 Bug fix - [ ] ✨ New feature - [ ] 💥 Breaking change - [ ] 📝 Documentation update - [ ] ♻️ Refactoring - [ ] ⚡ Performance improvement - [ ] ✅ Test coverage improvement ## Checklist <!-- Mark with 'x' what you've done --> - [ ] I have read the [Contributing Guide](https://github.com/areal-project/community/blob/main/CONTRIBUTING.md) - [ ] Pre-commit hooks pass (`pre-commit run --all-files`) - [ ] Relevant tests pass; new tests added for new functionality - [ ] Documentation updated (if applicable; built with `./docs/build_all.sh`) - [ ] Branch is up to date with `main` - [ ] Self-reviewed via `/review-pr` command - [ ] This PR was created by a coding agent via `/create-pr` - [ ] This PR is a breaking change **Breaking Change Details (if applicable):** <!-- Describe what breaks and how users should migrate --> ## Additional Context <!-- Add any other context, screenshots, logs, or explanations here --> ______________________________________________________________________ **Need help?** Check the [Contributing Guide](https://github.com/areal-project/community/blob/main/CONTRIBUTING.md) or ask in [GitHub Discussions](https://github.com/areal-project/AReaL/discussions)!
areal-project#1440) * feat(cli): add experimental cli scaffold for service-style subcommands Prepares the ground for feat/inference-service-cli, feat/training-service-cli, and feat/agent-service-cli to land on top — each subcommand PR only adds its own click group and one cli.add_command call, instead of duplicating the same plumbing. Key pieces: - empty top-level `cli` click group with version + --help wiring; subcommand modules attach themselves via cli.add_command(...) - `state.areal_home()` + `atomic_write_json()` for AREAL_HOME-rooted local state files (atomic .tmp + os.replace) - `process` module: pid_alive / pick_free_port / signal_pid / kill_pids + spawn_process with start_new_session=True so detached worker processes survive parent SIGHUP and a graceful SIGTERM→grace→SIGKILL teardown sequence is one helper away - pyproject: `[project.scripts] areal = ...cli.main:cli`, optional `[cli]` deps (click, colorlog) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * build(cli): regenerate uv lockfiles for cli optional deps Pre-commit's uv-lock hook expects the lockfiles to track click and colorlog after they were added to [project.optional-dependencies] cli. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli): address gemini review on process/state utilities - pid_alive: try os.waitpid(pid, WNOHANG) before os.kill so a zombie child does not look alive — otherwise kill_pids waits its full grace window and sends a redundant SIGKILL. - spawn_process: close the parent's log_handle copy after Popen dup()s it for the child, so repeated spawns do not leak fds. - atomic_write_json: use tempfile.NamedTemporaryFile + os.fsync, and unlink the tempfile on serialization or rename failure — fixes a tempfile leak on bad input, durability against crashes, and a race between concurrent writers on the same path. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(cli): unify terminate + reuse kill_process_tree, drop pick_free_port Replace the scaffold's bespoke process primitives with shared utilities already in the codebase: - process: kill_pids now delegates to areal.infra.utils.proc.kill_process_tree, which walks the descendant tree via psutil. This fixes the bug TaoZex raised on signal_pid's inconsistent PermissionError handling — signal_pid is gone entirely. - process: drop pick_free_port. Callers should use areal.utils.network.find_free_ports, which draws from a non-ephemeral port range and supports exclude_ports — addresses the TOCTOU race guozhihao-224 flagged on the naive bind(0) implementation. - scheduler: new module with a stateless top-level terminate(ref, *, backend, grace_s) that subcommand CLIs can funnel teardown through. Only the local backend is implemented; future backends extend the dispatch in-module. The free-function shape keeps the dispatcher from drifting into stateful scheduler instances. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: integrate CLI shared files Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style(cli): drop extra blank line after imports in utils.py Local pre-commit auto-fixed the blank line but the fix wasn't re-staged before commit 9401918, so CI's ruff hook caught it. * refactor(cli): collect namespace path helpers into NamespacedStateStore state.py previously exposed 13 free functions (namespace_root, services_dir, logs_dir, service_state_path, …, recover_pids_from_raw_state) that all took ``namespace`` as their first argument. Promote them to methods on a new NamespacedStateStore class so subcommand CLIs construct one instance per namespace and stop threading the namespace string through every call. areal_home, atomic_write_json, SupportsComponentProbe, and ServiceStateBase remain at module level. ServiceLifecycle and ConfigLoader each hold a store instance built from their namespace at __init__ time; LogsCommand resolves log paths via ``lifecycle.store.logs_dir``. No behavior change. Note: downstream branches (inf / agent / train) will need to swap their imports from ``state.service_state_path``/``state.clear_current_service``/ etc. to ``store.<method>`` on the next rebase. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…real-project#1448) * refactor: move 5 experimental modules into areal/v2 for 2.0 release Move agent_service, inference_service, training_service, weight_update, and cli from areal/experimental/ to areal/v2/, and rewrite every reference (Python imports, `python -m` invocations, console-script entry points, CODEOWNERS, docs, review-pr signals) to the new path. - 5 directories migrated via `git mv` (history preserved) - 70 files modified across areal/, tests/, examples/, pyproject{,vllm}.toml, .github/CODEOWNERS, ROADMAP.md, and tooling docs - `areal/v2/__init__.py` added so v2 is an importable package - Build config (tool.uv.build-backend with module-root="") auto-discovers the new package — no pyproject changes beyond the entry point Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style: apply pre-commit auto-fixes after areal/v2 move CI pre-commit job reformatted 54 files automatically: - .github/CODEOWNERS: realign owner columns (32-col) after shorter /areal/v2/ paths broke the previous 40-col alignment - areal/**, tests/**, examples/**, docs/**: ruff isort reorders `areal.v2.*` imports into their new alphabetical slot (between `areal.engine` and `areal.infra`) - markdown/yaml whitespace normalized by mdformat / ruff-format Pure formatting; no logic change. `pre-commit run --all-files` is now green locally. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(tests): point py<3.12 conftest stubs at areal/v2/ Two test conftest.py stubs (Python 3.10/3.11 compat) still had the `areal/experimental/{inference_service,weight_update}` path as the namespace package's __path__, broken by the v2 move. The sed pass missed them because the paths were comma-separated `os.path.join` args (`"areal", "experimental", "X"`), not slash-form path strings. Additionally insert an `areal.v2` stub between `areal` and the leaf package so the parent→child attribute wiring loop (which uses `name.rsplit(".", 1)`) can find a parent module in sys.modules. Without it `setattr(parent, child, ...)` silently no-ops and `unittest.mock.patch` traversal breaks on the new path. Spotted by gemini-code-assist on PR areal-project#1448. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(tests): mirror areal/v2 move under tests/v2 Move agent_service, inference_service, training_service, and weight_update test directories from tests/experimental/ to tests/v2/ to mirror the source-tree layout. tests/experimental/ retains archon/ and openai/ (still-experimental modules). - 50 files migrated via `git mv` (history preserved) - tests/v2/__init__.py added - 9 files rewritten for the new dotted/slashed test paths: `tests.experimental.{4 modules}` → `tests.v2.{4 modules}` `tests/experimental/{4 modules}` → `tests/v2/{4 modules}` (covers integration_utils imports, fake_train_engine engine_class, pytest invocation strings in docstrings, and the tests/v2/weight_update/torchrun/run_nccl_weight_transfer.py path) conftest stubs (Python <3.12 namespace shims) keep working because _REPO_ROOT is computed via "..", "..", ".." from the conftest file — same depth under tests/v2/X/ as under tests/experimental/X/. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* chore: integrate CLI shared files
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(cli): training experiment launcher — `areal train run`
Wraps a user-supplied driver entry point (`module.path:func`) into a
uniform CLI invocation driven by yaml + hydra overrides. Same lifecycle
as `python -m examples.math.gsm8k_rl ...` — the CLI process is the
driver process, attached, all the way down.
Single verb, permanently. No ps / stop / status / logs: a training job
is a process, not a service; OS tooling (`ps`, `kill`) and the cluster
scheduler (`squeue`, `kubectl get`) already cover that, and logs live
at `{fileroot}/logs/...` for `tail -F`. Service-shaped verbs belong
with `areal inf`, whose state is daemon-backed and invisible to OS
tooling.
Usage:
areal train run --config experiments/grpo.yaml \
--driver examples.math.gsm8k_rl:main \
actor.lr=1e-5 trial_name=lr-sweep-3 +actor._version=v2
`--driver MOD:FN` is required; we deliberately don't peek a `driver:`
field out of the yaml — keeps the contract obvious and avoids parsing
yaml in the CLI layer.
Anything after the flags is forwarded verbatim as a hydra override to
the driver (click's `nargs=-1 + UNPROCESSED + ignore_unknown_options`).
Backgrounding is out of scope: users wrap the command in
nohup/tmux/sbatch as appropriate. Adding `--detach` would force state
files, heartbeat, pid tracking — none of which anything else here needs.
Conflicts with `feat/inference-service-cli` are isolated to main.py +
pyproject.toml's [cli] extra; trivial 3-way merge after the inf branch
lands.
* refactor(experimental): split training CLI commands
Move the training CLI entrypoint into a root CLI module and a training command package so future training commands can be added without growing a monolithic command file.
* docs(cli/train): add training service CLI guide
Document the `areal train` subcommand group: basic concepts, the
driver function contract, hydra override conventions, and exit code
behaviour.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(experimental): rebase train CLI onto cli scaffold v2
Train doesn't carry a service / daemon / state, so the scaffold's
ServiceLifecycle / BaseHTTPClient / StatusReporter components don't
apply — train run is a synchronous importlib dispatcher. The change
here is just plumbing alignment:
- Rebase onto feat/experimental-cli-scaffold; scaffold now owns
cli/__main__.py / cli/cli.py.
- Drop the redundant cli/main.py shim (cli/__main__.py covers it).
- Point [project.scripts] areal entry at cli.cli:cli in both
pyproject manifests, matching the agent / inf branches.
- Re-flow training/cli_guide.md through mdformat.
* docs(cli/train): drop the evaluation example from the guide
* docs(cli/train): drop the BOBA-GRPO example from the guide
* fix(cli): resolve config merge leftovers
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(cli): add experimental cli scaffold for service-style subcommands Prepares the ground for feat/inference-service-cli, feat/training-service-cli, and feat/agent-service-cli to land on top — each subcommand PR only adds its own click group and one cli.add_command call, instead of duplicating the same plumbing. Key pieces: - empty top-level `cli` click group with version + --help wiring; subcommand modules attach themselves via cli.add_command(...) - `state.areal_home()` + `atomic_write_json()` for AREAL_HOME-rooted local state files (atomic .tmp + os.replace) - `process` module: pid_alive / pick_free_port / signal_pid / kill_pids + spawn_process with start_new_session=True so detached worker processes survive parent SIGHUP and a graceful SIGTERM→grace→SIGKILL teardown sequence is one helper away - pyproject: `[project.scripts] areal = ...cli.main:cli`, optional `[cli]` deps (click, colorlog) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(cli): unify terminate + reuse kill_process_tree, drop pick_free_port Replace the scaffold's bespoke process primitives with shared utilities already in the codebase: - process: kill_pids now delegates to areal.infra.utils.proc.kill_process_tree, which walks the descendant tree via psutil. This fixes the bug TaoZex raised on signal_pid's inconsistent PermissionError handling — signal_pid is gone entirely. - process: drop pick_free_port. Callers should use areal.utils.network.find_free_ports, which draws from a non-ephemeral port range and supports exclude_ports — addresses the TOCTOU race guozhihao-224 flagged on the naive bind(0) implementation. - scheduler: new module with a stateless top-level terminate(ref, *, backend, grace_s) that subcommand CLIs can funnel teardown through. Only the local backend is implemented; future backends extend the dispatch in-module. The free-function shape keeps the dispatcher from drifting into stateful scheduler instances. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: integrate CLI shared files Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(cli): collect namespace path helpers into NamespacedStateStore state.py previously exposed 13 free functions (namespace_root, services_dir, logs_dir, service_state_path, …, recover_pids_from_raw_state) that all took ``namespace`` as their first argument. Promote them to methods on a new NamespacedStateStore class so subcommand CLIs construct one instance per namespace and stop threading the namespace string through every call. areal_home, atomic_write_json, SupportsComponentProbe, and ServiceStateBase remain at module level. ServiceLifecycle and ConfigLoader each hold a store instance built from their namespace at __init__ time; LogsCommand resolves log paths via ``lifecycle.store.logs_dir``. No behavior change. Note: downstream branches (inf / agent / train) will need to swap their imports from ``state.service_state_path``/``state.clear_current_service``/ etc. to ``store.<method>`` on the next rebase. * feat(experimental): add agent service CLI * refactor(experimental): align agent CLI with Click * refactor(experimental): remove agent rl negotiated flag Use the existing RL session id and API key as the source of truth instead of keeping a derived boolean in agent session state. Key changes: - Drop rl_negotiated from SessionState and tolerate legacy state files - Route agent CLI and demo output through loggers instead of print - Update CLI state tests for the revised session schema * refactor(experimental): avoid creating agent session on run Agent service startup should only launch runtime components. Sessions are now created explicitly through new_session. Key changes: - Remove initial session creation from areal agent run - Drop the run --session-key option - Add CLI tests for session-free startup * refactor(experimental): align agent CLI with inference; drop session verbs Pulled the agent CLI in line with feat/inference-service-cli and removed surface the controller-replacement scope does not need. Style alignment with inference CLI: - adopt 2-step *_cmd -> do_* (drop the redundant handle layer) - table / JSON output via click.echo; hard failures via click.ClickException - Click default_map driven by load_click_default_map(_BINDINGS); drop the hand-rolled cfg_get / resolve_* helpers - single AgentCli logger (registered color in areal/utils/logging.py), exported from new agent/common.py alongside running_state / load_running_state / wait_http_health - AgentCLIHTTPError / AgentCLIUnreachable renamed to AgentHTTPError / AgentUnreachable; per-server clients renamed (GatewayClient, RouterClient, DataProxyClient) to mirror inference/client.py Removed surface: - interactive REPL stub (interactive.py + --interactive / --stop-on-exit / --history-file on run); chat / reward placeholders never shipped - destroy / health aliases of stop / status - new_session / switch_session and session_ops; SessionState / SessionsState / InferenceClient go with them. Session lifecycle is moving to a gateway /v1/sessions REST endpoint; CLI no longer owns it - legacy rl_negotiated migration in state.py (the flag was removed two commits ago; no production state needs the shim) Net diff: -830 lines, 6 tests green, pre-commit clean. * refactor(experimental): drop cli scaffold overlap pending areal-project#1440 The shared cli/cli.py + cli/main.py entry points and the pyproject [project.scripts] + the example invocation tweak duplicated plumbing that the cli scaffold PR (areal-project#1440) now provides. Drop them here so this branch can rebase cleanly on top of areal-project#1440 once that lands; the agent CLI itself (cli/agent/*) is untouched and continues to live in this PR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(experimental): rename agent http.py to client.py for parity with inf inf-cli puts the same kind of HTTP client wrappers in cli/inference/client.py; rename the agent counterpart to match so the two sibling subcommand packages have parallel layouts. Pure rename + three import updates, no behavior change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(experimental): adopt cli scaffold v2 in agent service Rebase onto feat/experimental-cli-scaffold and replace the per-CLI duplicates with the shared base classes/utilities: - state.py now keeps only the agent dataclasses; ServiceState satisfies ServiceStateBase and gateway_alive() is anchored on the gateway PID only (fixes the prior "any pid alive" check that kept reporting running after the gateway died). - config.py / client.py / lifecycle.py shrink to thin subclasses of ConfigLoader / BaseHTTPClient / ServiceLifecycle. - run.py and stop.py route through ServiceLifecycle for double-start refusal, force-replace, and state cleanup. - status.py renders via StatusReporter + ColumnSpec; ps.py emits via json_or_table; the bespoke logs subcommand is dropped in favor of LogsCommand(lifecycle=...).build(). - launcher.py reserves all ports up front through find_free_ports (non-ephemeral, no TOCTOU) and uses the scaffold spawn/wait helpers. Net ~360 LOC dropped from the agent module while every behavioral guarantee carries over. * refactor(experimental): drop empty Gateway/DataProxy client subclasses Both classes only inherited BaseHTTPClient.health() and were never instantiated after the scaffold refactor (StatusReporter probes addrs directly, launcher only uses RouterClient). Callers that ever need a gateway/data-proxy health probe can do BaseHTTPClient(addr) — the variable name carries the role. RouterClient stays because register_proxy is a real method. * fix(experimental): point areal CLI script at cli.py entry main.py was dropped earlier when scaffold landed (its single-line shim was redundant with __main__.py), but the [project.scripts] entry still referenced it, leaving the installed ``areal`` console script broken. Update both pyproject manifests to ``cli.cli:cli``. * docs(cli/agent): add agent service CLI guide Document the `areal agent` subcommand group: launching the gateway/router with N worker/data-proxy pairs, inspecting service state, log management, two-phase shutdown, configuration precedence, and the relationship with `areal inf`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style(docs): apply mdformat to agent CLI guide * refactor(experimental): adopt scaffold NamespacedStateStore in agent scaffold's state.py promoted the namespace-aware free functions onto NamespacedStateStore. Update agent accordingly: hold a module-level ``store = NamespacedStateStore(AGENT_NAMESPACE)`` and route every service_state_path / set_current_service / clear_current_service / logs_dir call through it. No subclassing needed — agent has no extra state files (vs inf's two-file split). Behavior unchanged; 6 tests still pass. * style: sort v2 CLI imports * refactor(v2/cli/agent): drop dead --inf-* options The --inf-addr / --inf-api-key / --inf-model surface was a placeholder for a session-api-key negotiation that never landed: the values were written into ServiceState but no downstream code (launcher, gateway, worker, data_proxy, /v1/responses bridge) reads them. Keeping the options advertises functionality the CLI does not provide. - run.py: drop the three click options and do_run params - launcher.py: drop the inf_* kwargs on launch_agent_stack - state.py: drop inf_* fields from ServiceState; pop them on load() for forward-compat with state files written by older revisions - config.py: drop the [inference] -> run.inf_* TOML bindings - cli_guide.md: drop the --inf-* example, the "Relationship with areal inf" section, and the [inference] TOML block; also drop the "Multiple pairs" example and fix the agent import path syntax from the package:Class colon form to the module.Class dot form (the worker resolves the path via import_from_string which requires dots) - tests: drop inf_* kwargs from test_agent_cli_run; rewrite the config default_map test to assert on a non-inference key Will be reintroduced (with real wiring) when /v1/sessions lands on the gateway and the gateway process needs inf credentials to negotiate session_api_key for RL trajectory association. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(cli): inference service CLI — daemon + 4 verbs
`areal inf` is an ollama-style operator console for the local inference
service. One daemon per user, one OpenAI-compatible gateway endpoint,
models registered against it.
Verbs:
inf run start daemon (gateway + router); inline --model registers
an external (--api-url) or internal (--backend / --model-path)
model in the same call
inf ps list registered models
inf status daemon health + model count
inf stop SIGTERM gateway+router, grace period, then SIGKILL
State is a single ~/.areal/inf/state.json (pid + url + admin key +
started_at). Everything else lives in the gateway/router process
memory; CLI is otherwise stateless.
Layout under areal/experimental/cli/:
main.py / state.py shared scaffold (areal_home, pid_alive,
atomic_write_json), to be reused by
`areal train` in a separate PR
commands/inf/__init__.py all four verbs + register helpers in
a single file (one place to scan
what the user can do)
commands/inf/state.py DaemonState dataclass + paths
commands/inf/launcher.py subprocess spawn helpers, including
base_gpu_id support for sglang dp>1
commands/inf/client.py urllib gateway + router HTTP client
Heavy imports (sglang/vllm/torch via areal.api.cli_args) stay lazy —
inside register helpers only — so `areal inf -h` and `areal -h` parse
the click tree without paying for them.
* fix(cli/inf): persist model worker pids in state.json so stop kills them
`_register_internal` returns the list of sglang+data-proxy pids it
spawned, but `_do_run` discarded the return value. state.json then
only held gateway+router pids, so `areal inf stop` left sglang and
data-proxy processes orphaned.
Add a `worker_pids: list[int]` field to DaemonState, save the list
right after register_internal succeeds, and include those pids in
the stop kill set.
Also fix _do_stop short-circuiting on a dead gateway pid — even when
the daemon front-end is dead, model worker pids in state may still
be alive (the original bug pattern this commit fixes). Always try
to kill every pid we know about, then drop the state file.
* fix(cli/inf): route startup messages through getLogger; align admin key default
Startup-time prints (router/gateway pids, replica spawn lines, daemon
ready, foreground / shutdown notices) were going through click.echo,
so they bypassed the AReaL log formatter — no timestamps, no log
level, no PascalCase tag. Switch all of these to
`getLogger("InfCli").info(...)` so output looks like the rest of the
project:
20260611-09:14:02.157 InfCli INFO: starting inference daemon ...
20260611-09:14:02.342 InfCli INFO: router pid=304116 http://...
20260611-09:14:02.910 InfCli INFO: gateway pid=304118 http://...
Register InfCli in LOGGER_COLORS_EXACT under the launcher (blue)
group, since `areal inf run` is conceptually a process launcher.
Reads kept on click.echo on purpose: `inf ps` / `inf status` plain-text
output is structured (tabular or single-field-per-line) and meant to
be consumed by jq / awk in scripts; piping that through the colored
log formatter would break it.
Also flip `--admin-api-key` default from "areal-admin-key" to
"admin-api-key" — matches the existing inference_service convention.
* feat(cli/inf): standalone `register` and `deregister` verbs
Phase 2 — model lifecycle separated from daemon lifecycle.
areal inf register <name> [external | internal flags]
Register a new model against a running daemon. Same flags as
`inf run --model ...`, just attached to an existing service.
areal inf deregister <name> [--grace] [--force]
Drop the model from the router, unregister its proxy workers,
and SIGTERM/SIGKILL the spawned sglang+data-proxy procs.
DaemonState gains a `models: dict[str, ModelEntry]` mapping each
registered model to its (pids, proxy_addrs). This replaces the
previous flat `worker_pids: list[int]` so deregister can target one
model's processes without touching the others. `inf stop` flattens
across all entries and kills the whole set; the foreground / failure
cleanup paths use the same flatten.
`_register_internal` now returns `(pids, proxy_addrs)` rather than
just pids — both are needed at deregister time (router unregister
takes the proxy addr; SIGTERM takes the pid).
External models also get a state entry now (empty pids/addrs) so
`inf deregister` can find them and drop them from the router.
phase 1 verbs unchanged.
* feat(cli/inf): phase 3 — reward + collect verbs
Two verbs that round out the RL data path:
areal inf reward <session_api_key> <reward> [--model X]
thin wrapper around POST /rl/set_reward. set_reward is the
only thing that flips an active conversation into a ready
trajectory, so any data-collection flow needs to call it (even
with a dummy reward=0 just to "flush"). CLI verb is for users
whose agent is in another language / shell / human raters; agent
authors writing python are free to POST directly.
areal inf collect <model> --batch-size N \
[--sessions-out FILE] [--output FILE] \
[--timeout T] [--poll-interval S] \
[--discount D] [--style individual|concat]
client-side batch orchestrator. start_session(group_size=N) ->
hand sessions to the agent (via --sessions-out FILE) -> poll
/export_trajectories every poll-interval seconds, accumulating
unique trajectories until N are collected or timeout fires ->
one final export with remove_session=True for cleanup -> dump
JSONL.
Mirrors the controller's rollout_batch path but moves the wait
loop to the client so gateway / router stay stateless. Agent
lifecycle is intentionally NOT inside collect (no --agent-cmd):
agent runs in its own process and just reads sessions_out.
JSONL is the only output format on purpose -- the gateway already
serializes trajectories to JSON at HTTP boundary, so .pt would mean
re-decoding tensors only to re-encode them; trainers consuming the
output can torch.tensor(x) when they need it.
8 verbs total (run / ps / status / stop / register / deregister /
reward / collect).
* feat(cli/inf): enrich ps/status with model kind / backend / addrs
Phase 2.5 — bring `inf ps` and `inf status` to the design_inf.md
fidelity (sections 11.5 / 11.6) without bringing back the multi-service
concept.
ModelEntry gains four fields:
kind: 'internal' | 'external'
backend: spec string ('sglang:tp=2,dp=2') for internal, '' for external
api_url: external upstream URL, '' for internal
inference_server_addrs: per-replica sglang/vllm URLs (internal)
`_register_internal` returns (pids, proxy_addrs, inf_addrs); _do_run
and _do_register both fill in the new ModelEntry fields.
inf ps now reads state.models (CLI-side truth) instead of polling
gateway /v1/models. Output:
NAME KIND BACKEND WORKERS
qwen3 internal sglang:tp=2,dp=2 2
gpt-4o external - -
inf status switches to a multi-row table per design 11.5:
COMPONENT STATUS ADDR DETAILS
gateway ok http://127.0.0.1:8080 models=2
router ok http://127.0.0.1:..
qwen3 registered internal backend=sglang:tp=2 workers=2
gpt-4o registered external api_url=https://...
JSON output of both verbs follows suit.
Backwards-compat: ModelEntry's new fields all have defaults, so an
old state.json still loads cleanly.
* feat(cli/inf): add `inf models` verb
`inf ps` currently lists registered models (table: NAME / KIND /
BACKEND / WORKERS). Add `inf models` as a more explicit alias —
docker-style "different verb for different resource". `ps` keeps
its current behavior; both call the same _print_models helper so
output is identical.
This isn't part of the multi-service rollback (which we decided not
to do). It's a small ergonomic addition for the single-daemon shape.
* feat(cli/inf): add `logs` verb + TOML config support + help text for proxy/engine args
Three related additions:
1. `inf logs --component NAME [-f] [-n LINES]`
Tail a log under ~/.areal/inf/logs/. Component defaults to 'gateway';
can be 'router' or a full model log basename like 'qwen3-inf-0'.
-f follows (tail -F), -n sets initial line count (default 200).
Exec's tail directly for stream fidelity.
2. TOML config support (design 12)
- Group-level option `areal inf --config FILE` merges FILE on top
of ~/.areal/inf/config.toml (both optional).
- config.py loads TOML via tomllib (py3.11+), maps [default] /
[launch] / [register.internal] / [collect] sections to CLI
option defaults via click's default_map mechanism.
- Precedence: CLI flag > --config > ~/.areal/inf/config.toml >
hardcoded default.
3. Detailed help for --engine-args and --proxy-args
Users couldn't guess what to pass. Now both flags show inline
hints (common sglang knobs / data-proxy flags + defaults). Help
is a single paragraph so click's wrap_text handles terminal width.
10 verbs total (run / stop / ps / status / models / register /
deregister / reward / collect / logs).
* fix(cli/inf): align `collect` flags with design 11.9
Rename + add + drop options on `inf collect` so the surface matches
the design spec exactly:
rename --discount -> --turn-discount
rename --style -> --export-style
add --format json|jsonl (default jsonl)
add --json progress-events flag (placeholder; not implemented)
drop --task-id (always 'cli-collect' internally)
drop --sessions-out (agents query gateway directly)
drop --poll-interval (always 2.0s internally)
JSON output (--format json) emits {tid: interaction, ...} pretty-printed.
JSONL output (default) emits one trajectory per line, each prefixed
with trajectory_id.
config.toml [collect] keys renamed to match.
* fix(cli/inf): three bugs (admin key, gpu collision, sglang request log)
P1. --admin-api-key default reverts from "admin-api-key" to
"areal-admin-key" — matches the v2 inference_service convention
used everywhere else in the codebase.
P2. Registering a second internal model collided with the first on
GPUs 0..tp-1. Cause: base_gpu_id was always r * tp, computed only
against the current model's dp index, ignoring GPUs already used
by previously-registered models. Fix: track a monotonic
`next_gpu_id` cursor on DaemonState and pass it into
`_register_internal` as `base_gpu_id`. Each ModelEntry now
records its (base_gpu_id, gpu_count) so deregister can roll back
the cursor when removing the *last* registered model (preserves
contiguous allocation; doesn't try to coalesce holes in the
middle, which is fine for the v1 use case).
P3. sglang server logs only had model-load output, no chat requests.
Cause: SGLangConfig.log_requests defaults to False. Fix: spawn
sglang with log_requests=True so /chat/completions traffic shows
up under ~/.areal/inf/logs/<model>-inf-N.log. (data-proxy
access logs are off via uvicorn config inside the data-proxy
package itself; that's not under inf CLI's control.)
* refactor(experimental): split inference CLI commands
Move the inference service CLI out of the monolithic commands package and into command-specific modules. Align collect with the session/export flow by returning session keys, polling exports without revoking the router group, and cleaning up at the end.
* feat(experimental): support multiple inference services
* fix(experimental): harden inference CLI lifecycle
Protect inference service state transitions so register and run do not race or leave orphaned processes behind.
Key changes:
- Track engine and proxy PIDs separately for phased shutdown
- Lock model state during register and startup model setup
- Clean up foreground services on SIGTERM and SIGHUP
- Recover raw PIDs before forced service replacement
- Align inference CLI model and session option names with the design
* feat(cli/inf): add scheduler abstraction for worker placement
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(cli/inf): widen probe timeout, parallelize status, drop default_model
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(experimental): adopt cli scaffold v2 in inference service
Rebase onto feat/experimental-cli-scaffold and replace the per-CLI
duplicates with the shared base classes/utilities:
- state.py keeps the inf-specific dataclasses and the two-file
recover_pids_from_raw_state; RuntimeState now satisfies
ServiceStateBase (gateway_alive + components + .load classmethod).
- config.py / client.py / lifecycle.py shrink to thin subclasses of
ConfigLoader / BaseHTTPClient / ServiceLifecycle. The old
GatewayHTTPError / GatewayUnreachable names are kept as aliases of
ServiceHTTPError / ServiceUnreachable so subcommands swap mechanically.
- InferenceLifecycle overrides force_replace_slot to walk the inf
raw-state helper (which knows about the secondary model-state file)
and to remove both files on cleanup.
- common.py drops scaffold-replaced helpers (running_state /
load_running_state / refuse_if_running / wait_http_health /
wait_client_health / print_services / print_models /
probe_http_health); keeps backend-spec parsing, model registration,
TaskHandle formatters, and terminate_runtime_state (data-flow order
is inf-specific).
- commands/run.py uses ServiceLifecycle for refuse / force-replace and
ForegroundWatcher for the SIGINT/SIGTERM/SIGHUP handling.
- commands/stop / status / ps / models / register / deregister /
reward / collect route through inf_lifecycle; status.py emits via
StatusReporter + ColumnSpec; ps/models via json_or_table.
- launcher.py and scheduler/local.py replace pick_free_port with
find_free_ports (non-ephemeral, no TOCTOU); LocalScheduler tracks
allocated ports across submits.
- commands/logs.py is removed; LogsCommand(lifecycle=inf_lifecycle)
wires the verb in __init__.py.
Net ~290 LOC dropped while every behavioral guarantee carries over.
* docs(cli/inf): add inference service CLI guide
Document the `areal inf` subcommand group: launching the gateway/router,
registering models, RL session flow with rewards, trajectory collection,
log management, and configuration file precedence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style(docs): apply mdformat to inference CLI guide
* refactor(experimental): address PR review on inference CLI
Address garrett4wade's review on areal-project#1435:
1+2) Drop the unfinished `collect` verb and the gateway RPCs it
depended on. `commands/collect.py` is removed entirely; the gateway
client no longer exposes `start_session` / `export_trajectories`
(only `set_reward` remains, used by the surviving `reward` verb).
The cli group, config bindings, cli_guide section, and parser tests
are cleaned up accordingly.
3) Replace the bespoke `sglang:tp=2,dp=2` mini-DSL in
`parse_backend_spec` with `ModelAllocation.from_str`, so the CLI
accepts the same grammar as `InferenceEngineConfig.backend` in YAML
configs (`sglang:d4`, `vllm:d2t4`). Help text, doc examples, and
test fixtures are updated to the new form.
4) Collect the free functions in `state.py` (`models_dir`,
`models_state_path`, `models_lock_path`, `locked_model_state`,
`recover_pids_from_raw_state`) into an `InferenceStateStore` class
and route every caller through a module-level `store` instance.
The dataclasses now obtain paths via `store.<...>`, keeping
on-disk-layout responsibilities in one place.
Net: 13 files, +170 / -524 (mostly from dropping the collect verb).
* style(tests): drop trailing blank lines after removed collect case
* refactor(experimental): subclass scaffold NamespacedStateStore in inference
scaffold's state.py promoted the namespace-aware free functions onto
NamespacedStateStore. Update inference accordingly:
- InferenceStateStore now subclasses NamespacedStateStore, gaining
service_state_path / set_current_service / clear_current_service /
current_service_path / resolve_service_name from the parent. It keeps
the inf-specific models_*, lock_model_state, and overrides
recover_pids_from_raw_state to walk both state files.
- ServiceState.save / .remove route through ``store.set_current_service``
/ ``store.clear_current_service`` instead of the deleted free
functions; commands/run.py / register.py / tests resolve log paths
via ``store.logs_dir`` and the service-state path via
``store.service_state_path``.
Behavior unchanged.
* style: sort v2 CLI imports
* refactor(experimental): drop reward verb + RL session flow docs
Address PR areal-project#1434 review:
- Remove ``areal inf reward`` (commands/reward.py + the ``reward_cmd``
wiring in __init__.py) and the corresponding GatewayClient.set_reward
RPC. The reward flow is server-side only for now; the CLI does not
need to wrap it.
- Drop the "RL session flow" and "Setting reward" sections from
cli_guide.md plus the trailing "For plain inference there is no need
to call /rl/start_session" note.
- Drop the ``reward`` entry from the [default].service binding tuple
in config.py and the reward-related cases in the parser test.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…les (areal-project#1383) * feat(agent_service): add agent service with OpenClaw and Hermes examples Add an end-to-end agent service stack under areal/v2/agent_service with a gateway bridge, data proxy, and worker, plus two example agents (OpenClaw and Hermes) demonstrating session lifecycle, reward setting, and training integration. Key changes: - Extend gateway bridge and data proxy to drive agent sessions - Add OpenClaw and Hermes example agents under examples/agent_service - Add lifecycle demo, run scripts, and config for the Hermes example - Add integration and per-agent tests under tests/v2/agent_service Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(examples): drop redundant hermes session self-check scripts The hermes agent service session flow only needs run_agent_service.py: chat sessions are keyed implicitly by the /v1/responses "user" field, and self-evolution mints the per-session sk-sess-* key internally. The start_session.py / demo_lifecycle.py scripts only probed the inference gateway control plane and duplicated examples/openclaw, so remove them. Key changes: - Delete examples/agent_service/hermes/{start_session,demo_lifecycle}.py - Remove the README "Connectivity self-checks" section and Files rows - Repoint set_reward.py docs to run_agent_service.py for the session key * refactor(examples): hoist hermes example to examples/hermes top level Move the Hermes agent-service example from examples/agent_service/hermes to examples/hermes, putting it at the same level as examples/openclaw, and drop the superseded examples/agent_service/openclaw (the standalone examples/openclaw already replaces it). Key changes: - git mv examples/agent_service/hermes -> examples/hermes - Update agent_cls_path and docs to examples.hermes.hermes.HermesAgent - Rewrite README paths from examples/agent_service/hermes to examples/hermes - Remove obsolete examples/agent_service/openclaw Refs: areal-project#1383 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(examples): split hermes flow into standalone step scripts Replace the monolithic run_agent_service.py with a 5-step flow matching the openclaw example: train.py, `areal agent run`, start_session.py, hermes_loop.py interaction, and set_reward.py. Update start_session.py for the v2 inference service (HTTP 201 + nested session credentials), fold training defaults into config.yaml, and rewrite the README. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(examples): inline CLI formatting helpers and trim hermes README Remove the shared examples/hermes/_fmt.py module by inlining the helpers it provided into the two scripts that used it, so each script is self-contained. Also condense the README prose and use a placeholder model path. Key changes: - Inline formatting helpers into start_session.py and set_reward.py - Delete examples/hermes/_fmt.py and its README entry - Simplify quick-start prose; use actor.path=/path/to/your_model - Switch config.yaml actor backend to megatron:d1 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(examples): clarify session key usage in hermes and openclaw READMEs Annotate which placeholders reuse the sk-sess-* key returned by start_session and distinguish them from the upstream LLM credentials. Trim the openclaw README to focus on the RL training flow. Key changes: - Mark hermes --session-api-key / --api-key as the start_session key - Note HERMES_UPSTREAM_* are your own upstream LLM credentials - Use placeholder values instead of sk-... in env exports - Remove the standalone agent-service section from openclaw README Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style(test): fix import grouping in agent service tests Remove the stray blank line between the pytest import and the examples.* imports so ruff's isort check passes in CI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
) Add a reuse_train_logp option to prox_logp_method for decoupled PPO. It reuses the training forward-pass logprobs (detached) as the proximal logp, skipping the extra proximal forward pass and its memory/compute cost. This is only valid with ppo_n_minibatches=1: with a single minibatch the training forward still reflects the policy that generated the rollout, so its logprobs equal the proximal policy. With multiple minibatches the weights change between steps, so PPOActorConfig.__post_init__ rejects that combination. Add tests for the enum/constant wiring, skips_forward_pass, and the ppo_n_minibatches validation.
…roject#1457) reuse_train_logp requires ppo_n_minibatches=1 so the training forward pass still reflects the policy that produced the rollout. Previously an invalid combination raised ValueError; instead warn and force ppo_n_minibatches=1, making clear this reduces the PPO update to a single optimizer step. Remove obsolete tests that asserted behavior no longer relevant to this follow-up.
* chore: add AReaL 2.0 report paper * chore: rename paper file to AReaL2.0_report.pdf
Run a configurable post-exit shell command after local, Ray, and Slurm launchers stop jobs on timeout, interrupt, or failure paths. The hook receives LOG_DIR, has a 600-second timeout, and logs failures without interrupting shutdown or recovery. Ensure the hook still runs if launcher shutdown raises. Add the config field, unit tests, and regenerated CLI docs.
…-project#1460) Add Megatron context-parallel output gathering for forward-only paths and plumb vocabulary logits statistics through PPO/DPO/SFT losses. Support per-key reduce groups in StatsTracker so CP-local loss and vocab statistics can reduce across the appropriate DP/CP group without changing the default reduction group for unrelated stats. Expose Megatron/MoE configuration knobs for router fusion, auxiliary-loss-free balancing, router z-loss, FP32 lm_head output, and fused cross entropy. Update Bailing MoE defaults and regenerate CLI docs.
…ion (areal-project#1454) Use actual trajectory group sizes when applying group-level reward and advantage normalization so failed or filtered rollout samples do not cause fixed-size slices to cross prompt groups. Pass trajectory metadata through batched_call instead of injecting a sentinel batch key, rename the Normalization argument to group_sizes, and zero singleton leave-one-out groups because they have no peer baseline. Add tests for variable-size groups, singleton leave-one-out behavior, validation, 2D advantage normalization, and batched_call metadata forwarding.
…e chat template (areal-project#1463) Some chat templates (e.g. GLM-5.1) iterate over tool_call arguments with .items(), which fails when arguments is a JSON string as per the OpenAI API convention. Normalize tool_call arguments to dicts before applying the chat template in concat_prompt_token_ids_with_parent and the completions/responses tokenizer paths.
Add a SWE-bench RL training workflow under examples/swe: - train_swe_rl.py: the RL training entrypoint. - agent.py: the AReaL-SWEAgent rollout workflow that runs SWE-bench agents through AReaL's OpenAI-compatible proxy during training. - preprocessors.py / prefix_matchers.py: message preprocessing and interaction-cache prefix matching for the agent proxy. - filter_function.py: rollout group accept/reject filter. - qwen3_30b_a3b_grpo.yaml: a runnable Qwen3-Coder GRPO example config. - README.md: setup guide covering the AReaL-SWEAgent checkout, the AEnvironment backend, and Claude Code (cc) agent training. Also add the SWE dataset loader.
* fix: fix safe-to-test CI workflow * fix(tests): remove test_hermes_agent.py Companion cleanup to the openclaw removal in the previous commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: fix wu ci test --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…upport (areal-project#1458) Extend the OpenAI-compatible proxy rollout server with pluggable message preprocessors and a configurable interaction-cache prefix matcher, add Anthropic-to-OpenAI content handling and tool-call streaming, the Qwen-style tool-call parser, and the accompanying client/proxy/tool-call tests.
…oject#1464) After areal-project#1454, a size-1 leave-one-out group uses the sample itself as the baseline and normalizes to zero instead of passing through the raw advantage. Update test_group_size_edge_cases to assert the new behavior.
swe_sft.py is an SFT-only dataset builder that is not used by the SWE-bench RL training example (examples/swe/train_swe_rl.py loads raw problem instances directly) and is referenced only by its own dataset registration. Remove the module and its registration so the RL example does not ship unused SFT data-processing code.
…er state correctly (areal-project#1468) Since the dist_checkpointing refactor in megatron-core v0.14, flattened_range is legacy, non-reconstructable metadata and is no longer supported as a persistable layout in the checkpoint serialization path; the remaining flattened_range code paths were removed upstream in Megatron-LM PR #2126, and ShardedTensor.validate_metadata_integrity() now rejects any ShardedTensor with flattened_range set. The sharded_state_dict API still defaults to fully_sharded_model_space, which emits flattened_range, so saving a checkpoint with optimizer state fails on the pinned megatron-core 0.17.0. Request dp_reshardable sharding instead, matching the optimizer state format upstream now uses by default. Also pass is_loading=True when building the load-side template. Without it a freshly built optimizer skips megatron-core's state pre-allocation, the template only requests "param", and DCP silently drops exp_avg/exp_avg_sq on resume -- training continues with a reset optimizer state at full learning rate, which we observed to cause gradient-norm spikes and entropy collapse within ~70 steps after recovery in a large-scale RL run.
Introduce a structured reward result that can carry per-step rewards alongside the final scalar reward, align stepwise rewards to completion tokens in RLVRWorkflow, and let PPO consume the injected token-level process rewards without changing the default scalar-only path. Key changes: - add RewardResult and scalar-to-structured normalization helpers - emit step_rewards and step_reward_mask tensors from RLVRWorkflow - inject stepwise rewards into PPO total rewards with logging support - add focused tests for reward normalization and stepwise alignment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR implements a scoped Stage-1 landing for stepwise / process-level reward plumbing.
It does not change the default scalar-reward behavior. Existing reward functions that return a single
floatcontinue to work unchanged.This first landing adds:
RewardResultpayload that can carryfinal_rewardplus optional per-step rewards;RLVRWorkflow;step_rewards/step_reward_maskby injecting them into token-level total rewards;The intent is to make process supervision a first-class path in AReaL while keeping the current default behavior intact.
Related Issue
Fixes areal-project#1381
Type of Change
Checklist
pre-commit run --all-files)./docs/build_all.sh)main/review-prcommand/create-prAdditional Context
Tested on the Aliyun Linux environment with:
tests/test_stepwise_reward_stage1.pytests/test_async_reward_wrapper.pyI did not run the full local Windows test flow because the repository lockfile targets Linux/macOS environments.
This PR intentionally stops at Stage 1 plumbing. It does not yet introduce a broader process-reward API across all workflows, and it does not attempt the algorithmic
IcePop Plusintegration.