Skip to content

feat(workflow): add stage-1 stepwise reward plumbing #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Fyrgo8 wants to merge 36 commits into from
Closed

feat(workflow): add stage-1 stepwise reward plumbing #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
585d074
docs: add community section with WeChat QR code to README 📢 (#1409)
mingcheng Jun 15, 2026
13353cc
Supporting features for IcePop and KPop (#1405)
guojiapub Jun 15, 2026
8b2d221
fix: per-sample version tracking with loss_mask filter and multi-turn…
pyq623 Jun 19, 2026
1f24e1e
docs: add IcePop/KPop feature introduction (#1424)
guojiapub Jun 22, 2026
37645dd
fix(vllm): forward frequency_penalty and stop in generation requests …
EazyReal Jun 23, 2026
f19e00a
fix(awex): allow disabling batch_send_recv use_group via AWEX_WU_USE_…
sitabulaixizawaluduo Jun 23, 2026
e975680
feat: disable megatron grad buffers CPU backup to save host memory (#…
HT-Yuan Jun 23, 2026
b89271e
fix: only subtract in-range excluded ports in find_free_ports (#1436)
EazyReal Jun 23, 2026
bd333e6
fix(reward): guard clevr_count_70k_reward_fn against scoring failures…
EazyReal Jun 23, 2026
4be0c64
feat(megatron): make MTP head opt-in to support Qwen3.6 MoE RL (#1403)
Adiactive Jun 23, 2026
d97d3ae
feat(ppo): add CISPO loss surrogate (MiniMax-M1) (#1412)
EazyReal Jun 23, 2026
58dee1d
fix(CI): fix vlm_grpo CI OOM bug (#1438)
sitabulaixizawaluduo Jun 25, 2026
be8d7cf
feat(cli): add experimental cli scaffold for service-style subcommand…
sitabulaixizawaluduo Jun 29, 2026
ae66992
refactor: move 5 experimental modules into areal/v2 for 2.0 release (…
sitabulaixizawaluduo Jun 29, 2026
62512eb
feat: support v2 weight update disk mode for lora RL (#1450)
sitabulaixizawaluduo Jun 30, 2026
2824f72
feat(cli): add training service cli (#1446)
sitabulaixizawaluduo Jun 30, 2026
0ca30f1
feat(cli): add agent service cli (#1447)
sitabulaixizawaluduo Jun 30, 2026
b891741
feat(cli): add inference service cli (#1434)
sitabulaixizawaluduo Jun 30, 2026
bbc85c7
feat(agent_service): add agent service with OpenClaw and Hermes examp…
IF007 Jun 30, 2026
4ac8c36
feat(ppo): add reuse_train_logp proximal logp method (#1453)
Le8r0nJames Jul 1, 2026
9c6a703
fix(ppo): coerce ppo_n_minibatches to 1 for reuse_train_logp (#1457)
Le8r0nJames Jul 1, 2026
5ea8aae
Chore/add areal 2 report paper (#1461)
sitabulaixizawaluduo Jul 1, 2026
8979fa5
feat(launcher): add best-effort post-exit hook (#1459)
Le8r0nJames Jul 1, 2026
a315ea9
feat(megatron): add CP-safe vocab stats and MoE config support (#1460)
Le8r0nJames Jul 1, 2026
bbc10f0
fix(ppo): handle variable-size trajectory groups in reward normalizat…
Le8r0nJames Jul 1, 2026
87a034e
fix(openai): parse tool_call arguments from JSON string to dict befor…
Le8r0nJames Jul 1, 2026
d8ff8ce
feat(swe): add SWE-bench RL training workflow (#1462)
Le8r0nJames Jul 1, 2026
f2958b6
fix: fix safe-to-test CI workflow (#1456)
sitabulaixizawaluduo Jul 1, 2026
7f3b021
feat(swe): add SWE agent proxy, message preprocessors and tool-call s…
Le8r0nJames Jul 1, 2026
7c1d22d
test(ppo): update singleton leave-one-out group expectation (#1464)
Le8r0nJames Jul 1, 2026
77458d0
chore: update AReaL 2.0 report paper (#1465)
sitabulaixizawaluduo Jul 1, 2026
fee938e
chore: bump v2.0.0 (#1466)
sitabulaixizawaluduo Jul 1, 2026
90cb574
chore(swe): remove unused SWE SFT dataset loader (#1467)
Le8r0nJames Jul 2, 2026
9cf6852
fix(docker): rename zeroclaw installer flag to --skip-quickstart (#1469)
sitabulaixizawaluduo Jul 2, 2026
89e05ac
fix(megatron): use dp_reshardable optimizer sharding and load optimiz…
Le8r0nJames Jul 2, 2026
40e7928
feat(workflow): add stage-1 stepwise reward plumbing
Fyrgo8 Jul 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .agents/skills/review-pr/MIGRATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ ______________________________________________________________________
1. `areal/experimental/models/archon/**` + `areal/experimental/engine/archon_engine.py`
\+ `areal/experimental/engine/archon_checkpoint.py` -> Distributed Runtime + Model
Compute & Attention (CRITICAL/HIGH)
1. `areal/experimental/agent_service/**` -> Service Orchestration (HIGH)
1. `areal/experimental/inference_service/**` -> Service Orchestration (HIGH)
1. `areal/v2/agent_service/**` -> Service Orchestration (HIGH)
1. `areal/v2/inference_service/**` -> Service Orchestration (HIGH)
1. `areal/experimental/engine/archon_weight_sync.py` -> Distributed Runtime (CRITICAL)
1. `areal/infra/rpc/**` -> Launcher & Infrastructure + Distributed Runtime (HIGH)
1. `areal/workflow/**`, `areal/trainer/**` -> Workflow & Trainer Contract (HIGH/MEDIUM)
Expand Down
32 changes: 16 additions & 16 deletions .agents/skills/review-pr/references/review-pr-domains-and-signals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,19 +52,19 @@ ______________________________________________________________________

## Domain 4: Service Orchestration (HIGH)

| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `service_routing_dataflow` | `areal/experimental/agent_service/gateway/`, `areal/experimental/agent_service/router/`, `areal/experimental/inference_service/data_proxy/`, `areal/experimental/inference_service/controller/` | `route`, `gateway`, `router`, `DataProxy`, `controller`, `batch` |
| `session_consistency` | `areal/experimental/agent_service/`, `areal/experimental/inference_service/` | `session`, `affinity`, `history`, `state` |
| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `service_routing_dataflow` | `areal/v2/agent_service/gateway/`, `areal/v2/agent_service/router/`, `areal/v2/inference_service/data_proxy/`, `areal/v2/inference_service/controller/` | `route`, `gateway`, `router`, `DataProxy`, `controller`, `batch` |
| `session_consistency` | `areal/v2/agent_service/`, `areal/v2/inference_service/` | `session`, `affinity`, `history`, `state` |

## Domain 5: Workflow & Trainer Contract (HIGH/MEDIUM)

| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| `workflow_engine_boundary` | `areal/workflow/`, `areal/trainer/`, `areal/engine/` | `RolloutWorkflow`, `arun_episode`, `agenerate` |
| `dataset_surface` | `areal/dataset/` | `DataLoader`, `IterableDataset`, `get_*_dataset` |
| `async_contract` | `areal/workflow/`, `areal/experimental/agent_service/`, `areal/experimental/inference_service/` | `async def`, `await`, `aiofiles`, `asyncio` |
| `weight_version_contract` | `areal/api/engine_api.py`, `areal/workflow/`, `areal/trainer/` | `WeightUpdateMeta`, `set_version`, `weight version` |
| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | --------------------------------------------------------------------------- | --------------------------------------------------- |
| `workflow_engine_boundary` | `areal/workflow/`, `areal/trainer/`, `areal/engine/` | `RolloutWorkflow`, `arun_episode`, `agenerate` |
| `dataset_surface` | `areal/dataset/` | `DataLoader`, `IterableDataset`, `get_*_dataset` |
| `async_contract` | `areal/workflow/`, `areal/v2/agent_service/`, `areal/v2/inference_service/` | `async def`, `await`, `aiofiles`, `asyncio` |
| `weight_version_contract` | `areal/api/engine_api.py`, `areal/workflow/`, `areal/trainer/` | `WeightUpdateMeta`, `set_version`, `weight version` |

## Domain 6: API & Config Compatibility (MEDIUM)

Expand Down Expand Up @@ -95,12 +95,12 @@ ______________________________________________________________________

## Domain 9: Launcher & Infrastructure (HIGH/MEDIUM)

| L2 Signal | File Path Pattern | Code Pattern |
| ------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------- |
| `launcher_resource_match` | `areal/infra/launcher/` | `LaunchConfig`, `RayLauncher`, `SlurmLauncher` |
| `scheduler_contract` | `areal/infra/scheduler/`, `areal/scheduler/` | `Scheduler`, `placement`, `resource` |
| `rpc_transport` | `areal/infra/rpc/`, `areal/experimental/inference_service/data_proxy/` | `RTensor`, `serialize`, `rpc`, `fetch` |
| `runtime_image_config` | `Dockerfile`, `.dockerignore` | `FROM`, `ARG`, `RUN`, `ENV`, `COPY`, `uv sync`, `VARIANT` |
| L2 Signal | File Path Pattern | Code Pattern |
| ------------------------- | ------------------------------------------------------------ | --------------------------------------------------------- |
| `launcher_resource_match` | `areal/infra/launcher/` | `LaunchConfig`, `RayLauncher`, `SlurmLauncher` |
| `scheduler_contract` | `areal/infra/scheduler/`, `areal/scheduler/` | `Scheduler`, `placement`, `resource` |
| `rpc_transport` | `areal/infra/rpc/`, `areal/v2/inference_service/data_proxy/` | `RTensor`, `serialize`, `rpc`, `fetch` |
| `runtime_image_config` | `Dockerfile`, `.dockerignore` | `FROM`, `ARG`, `RUN`, `ENV`, `COPY`, `uv sync`, `VARIANT` |

## Domain 10: Low-Risk Hygiene (LOW)

Expand Down
Loading
Loading