feat(agentic): add 2048 episode-based RL demo (#188) by PasserbyPh · Pull Request #384 · inclusionAI/AReno

PasserbyPh · 2026-07-29T10:09:35Z

What does this PR do?

Summary

Adds a self-contained 2048 agentic RL example under examples/agentic/2048/ that reuses AReno's existing contracts (--agent-fn / --reward-fn-path / --algo grpo) — no new CLI flags, algorithms, or dependencies. Followed by a series of hardening fixes and a refactor on the same branch.

Why

Issue #188 needed a clean RL example showing AReno reuse; 2048's small scope, strategic depth, and clear metrics make it perfect.

How

Following AReno's tictactoe/duelgrid patterns:

Game Engine (game.py): Deterministic 2048 with seeded randomness, single merge primitive reused for all directions, episode limits, and move validity checks
Reward Layer (reward.py): Normalized rewards via baseline subtraction (score - random_baseline) plus invalid-move penalties
Evaluation (baseline.py, web_ui.py): CPU harness for testing, local web UI for human/agent comparison, deterministic replay debugging
Tests: CPU suite covering game logic edge cases, reward computality, and component integration

What

Game engine (game.py): pure-Python 4x4 board; four directions derived from a single left-merge primitive; random.Random for seeded deterministic replay; no-op move detection/penalty; episode cap; random-action baseline; and play_episode_frames for step-by-step UI playback. Both tool-call and XML no-tool variants mirror the tictactoe/duelgrid patterns.
Reward / data: reward.py returns a single scalar episode_score - random_baseline - INVALID_PENALTY*invalid_moves, logging episode score, max tile, invalid rate, and improvement vs baseline.
Observability: CPU-only baseline.py harness and local web_ui.py (Human / Random Step / LLM Episode / LLM Auto modes, offline-capable).
Docs & tests: cookbook 2048-agentic-rl.rst, troubleshooting pointer, CODEMAP row; CPU tests covering merge edge cases, seeded replay, episode cap, baseline, reward, and loader→reward integration.

1785317048358-ea56224f-691e-4766-b1c9-f316781c1d20

Related issue

Closes #188

Type of change

🐛 Bug fix
✨ New feature
💥 Breaking change (public API / CLI behavior changes in a non-backward-compatible way)
📝 Documentation update
♻️ Refactoring
⚡ Performance improvement
✅ Test coverage improvement

How was it tested?

pytest tests/ -k agentic_2048       # CPU suite, all passing locally (25 cases)
pytest tests/ -k cpu                # broader CPU suite
python examples/agentic/2048/web_ui.py   # local visual verification of UI modes
python examples/agentic/2048/baseline.py # CPU-only baseline measurement

CPU test suite passes locally (25 passed). GPU train/serve flows are documented in the cookbook; not run in this session.

Checklist

The PR title summarizes the contribution.
Linked the related issue in the description (if any).
Existing tests pass (pytest tests/ -k cpu).
New behavior is covered by tests.
Described the test commands run and any hardware limitations.
Public API / CLI changes are additive and backward-compatible (see CONTRIBUTING.md).

Breaking change details

N/A — this PR adds files only under examples/agentic/2048/ and docs/; it touches no public API, CLI option surfaces, or config dataclasses in areno/.

Add a self-contained 2048 agentic example under examples/agentic/2048/ that reuses AReno's existing contracts (--agent-fn / --reward-fn-path / --algo gspo) -- no new CLI flags, algorithm, or dependencies. Engine (game.py): pure-Python 4x4 board, four directions from one left-merge primitive, seeded random.Random tile placement for deterministic replay, no-op move detection/penalty, episode cap, random-action baseline, and play_episode_frames for step-by-step UI playback. Both tool-call and XML no-tool variants mirror the tictactoe/duelgrid patterns. Observability: reward_fn returns one scalar (episode_score - random_baseline - INVALID_PENALTY*invalid_moves) and logs episode score / max tile / invalid-rate / improvement. A CPU-only baseline.py harness and a local web_ui.py (Human / Random Step / LLM Episode / LLM Auto modes, offline-capable) report metrics and trained-vs-baseline improvement. Tests: focused CPU tests cover merge edge cases, seeded-replay determinism, episode cap, baseline, reward success/invalid/boundary, observable fields, loader->reward integration, and backward compatibility. 22/22 pass locally; GPU train/serve documented for CI. Docs: cookbook 2048-agentic-rl.rst, troubleshooting pointer, CODEMAP row. Closes inclusionAI#188. Co-Authored-By: Claude <noreply@anthropic.com>

dataset_generator deduped on the board alone, so 2-tile spawns capped the dataset at C(16,2)*4 = 480 unique boards and --count 2048 raised "could not generate enough unique 2048 boards". Dedup on the replay seed instead: the board is a deterministic function of its seed and the seed also drives episode spawns, so two records sharing a board but differing in seed are distinct training samples. Removes the cap while preserving seeded determinism and authentic 2-tile starts. Co-Authored-By: Claude <noreply@anthropic.com>

…SON (CR) Address code-review findings (M1/M3/S1/S2), example+docs only, no core changes. M1 — Unify the random baseline to a uniform-random direction over all four directions (not legal-only) so the web UI's Random mode, game.random_episode, reward_fn, and baseline.py all report the same invalid-rate semantics. Update web_ui._random_step to pick from game.ACTIONS and let _step count no-ops as invalid; align the web_ui docstring + in-page rules text and the README/cookbook. M3 — Converge play_episode and play_episode_frames onto a single _replay primitive (mirrors duelgrid's step_turn) so the training-visible episode and the browser demo can never diverge. Frame boards are deep-copied snapshots ([list(row) for row in board]) so future in-place updates can't mutate earlier frames. Verified play_episode == play_episode_frames on a sequence with an out-of-enum token and no-op moves. S1 — baseline.evaluate_random now prefers the per-board random_baseline baked into the dataset when cap/trials match (baseline_source='stored', recomputed=0), and recomputes (with a count) otherwise. game.random_episode now returns a `cap` field so the match check is well-defined and old datasets without it fall back gracefully. S2 — baseline.py --json adds a flat top-level `summary` (mean_score/mean_max_tile/mean_invalid_rate) alongside the existing nested random_baseline/trained_policy/improvement blocks; documented in the README. Verified: 22 CPU tests green (20 example + 2 integration); M3 equivalence + deep-copy, M1 cap-field, S1 stored-vs-recomputed, S2 flat-summary, and an in-process web_ui Random-mode smoke (no-op invalid events observed, board advances, reaches terminal) all pass; --count 2048 generation still byte-identical on rerun. Co-Authored-By: Claude <noreply@anthropic.com>

Three fixes for one RL failure: the policy abandoned tool calls and spammed no-ops, and the no-op penalty was too weak to correct it. reward.py: drop the plain-text moves fallback. A turn with no choose_moves tool call returns a penalty worse than any legitimate episode, so prose can no longer out-score real tool calls. The old fallback let verbose non-tool responses win; tool_calls collapsed 4/4 -> 0/4 over 11 steps and the served step_000011 checkpoint emitted only truncated thinking prose. game.py: raise INVALID_PENALTY 0.5 -> 2.0. At 0.5 a full-no-op episode lost only 16, drowned by the ~150 baseline scale and merge-score variance, so invalid_rate never fell during RL. At 2.0 (full-no-op = -64) each wasted move competes with merge score. The engine always penalized no-ops correctly per move; the gap was magnitude, not mechanism. web_ui.py: keep a text fallback purely for the demo and add verbose [DEV] logging (finish_reason/tool_calls/content, parsed moves, episode result, per-frame board) tagged `DEV-LOG: remove before launch`. Tests: wrong-tool is penalized not parsed; a no-op appended to a plan lowers reward by exactly INVALID_PENALTY. Existing tests reference the constant and auto-adapt. Co-Authored-By: Claude <noreply@anthropic.com>

Two demo-local fixes for serving the trained 2048 policy; no areno source touched. 1. Clamp checkpoint context before serve to avoid OOM. `areno serve` sizes its paged-KV pool from max_position_embeddings in the checkpoint's config.json, which Qwen3.5 ships at 262144 (copied verbatim into every step_* checkpoint). Honouring 262144 OOMs a single GPU long before any 2048 prompt is served. Add patch_serve_context.py: an idempotent helper that clamps text_config.max_position_embeddings (plus the rope_scaling and top-level copies) to a small serving budget before launch, so the demo can be served without hand-editing config.json. 2. Surface tool-call usage in the web UI. _policy_moves silently fell back to parsing plain-text directions when the policy emitted no choose_moves tool call, so a half-trained policy could coast on prose with no signal. Return a source label (tool_call vs text_fallback), print it in the History panel, and add --strict-tool-call to reject the fallback outright (mirroring reward.py's no-tool-call penalty). The episode-based multi-step replay is unchanged -- it matches the training distribution. Co-Authored-By: Claude <noreply@anthropic.com>

After an LLM episode, terminal was set to `reached_2048 or is_terminal(board) or result.truncated`, so a plan that simply ran the 32-step cap without reaching 2048 or a dead end was reported as "Game over — no moves left" even though the board still had legal moves -- the cap is a plan-length limit, not a terminal condition. The text-fallback path (parse_moves yields an unbounded token list) made truncated fire whenever the policy emitted >32 directions, surfacing the false game-over the most. Gate terminal on real end conditions only, mirroring the single-step path (`_step`'s `won or stuck`). The episode cap behavior and the no-op/invalid penalty contract (inclusionAI#188) are unchanged. Co-Authored-By: Claude <noreply@anthropic.com>

…rics The web UI's Score jumped backward whenever an LLM episode ran after any human/random play. _llm_episode set `server.score = result.score`, but result.score is the merge score gained *during this episode* from the current board (play_episode_frames starts from 0), so it discarded everything accumulated before the episode. moves_played/invalid_moves had the same overwrite bug, so the Invalid stat jumped too. The History panel also clipped each event to a fixed 36px single line (white-space:nowrap + ellipsis), hiding long entries, and stacking new pills onto one row crowded the board column off-screen. Score/stats: switch _llm_episode to `+=` for score, moves_played, and invalid_moves so all three stay whole-game cumulative across human/random/llm modes, matching _step's per-move accumulation. Score is now the classic 2048 merge-score total, not an episode-only value. LLM episode metrics: add server.last_episode {score, improvement, reward} populated in _llm_episode (improvement = result.score - random baseline; reward = improvement - INVALID_PENALTY*invalid, matching reward_fn / game.score_moves) and surfaced via _payload. The UI adds three LLM-only pills -- Episode (episode merge score), vs Random (trained-vs-baseline improvement), Reward (full RL reward) -- shown as "-" while not in LLM mode or before the first episode, so the three senses of "score" no longer share one label. Layout: wrap the three LLM pills in .ep-stats (flex-basis:100%) so they sit on their own row under Score/Max/Invalid, and retune .app's grid columns (minmax(min-content,440px) 230px minmax(260px,1fr)) so the board keeps its natural width while History is no longer squeezed off. .event drops the fixed height/nowrap for content-driven height with overflow-wrap. Verified: py_compile clean; 23 existing CPU tests still green (21 example + 2 integration). The new accumulation/pills path isn't covered by the suite (it doesn't exercise web_ui routes), so confirm by serving locally. Co-Authored-By: Claude <noreply@anthropic.com>

… output Rewards were almost always negative even when the LLM scored well and had few illegal moves. Root cause was a length asymmetry, not the invalid-move penalty: the random baseline always plays 32 moves, but a short plan was measured against that full 32-move baseline and drowned (10-move plan: mean -131.8, 100% negative). score_moves now uses a length-matched baseline (A): full-length episodes reuse the stored baseline (unchanged); short plans recompute the random baseline at the same move budget. A zero-move output (E) -- which under pure length-matching would collapse the baseline to 0 and coast at reward ~0 -- is penalized worse than any >=1-move episode. tool_choice is forced, so an empty/unparseable moves list is the only real 'not outputting' path; the no-call branch shares the floor. web_ui delegates reward to score_moves and derives improvement by adding back the no-op penalty, keeping the UI's reward == improvement - no-op invariant (including the zero-move floor) and matching the training reward exactly. Verified on 200 boards: 10-move plan -131.8/100% neg -> -6.2/80% neg; 32-move unchanged; empty moves -> floor ~-217 and strictly below every real plan. CPU tests updated (empty/malformed/no-tool-empty -> E floor; noop-appended relaxed to >=) plus two new tests for length-matching and the zero-move floor (25 passed). Co-Authored-By: Claude <noreply@anthropic.com>

Disable the temporary `_dev_log` invocations marked `# DEV-LOG: remove before launch` in `web_ui.py` by commenting them out (not deleting), so the verbose stderr tracing is silenced for the demo while the call sites and helper stay in place for quick re-enablement. A `pass` keeps the per-frame loop body syntactically valid. Co-Authored-By: Claude <noreply@anthropic.com>

…e OOM fix Replace the minimal train command with a full, validated end-to-end command (Qwen3.5-0.8B, TP/world-size 2, GSPO, memory/context flags). Convert the serve and web-ui invocations from Python lists to shell. Add a 'Serve the Policy' section that documents the Qwen3.5 serve-time OOM (config.json advertises text_config.max_position_embeddings=262144) and points at patch_serve_context.py as the supported, idempotent fix; include the equivalent manual config edit in a details block. Make the no-tool variant a copy-pasteable 3-flag swap. List patch_serve_context.py in Files and add a serve-OOM note to Defaults & Limitations. Co-Authored-By: Claude <noreply@anthropic.com>

…deadlock The 2048 agent runner submitted all prompt requests concurrently via asyncio.gather, overwhelming the engine's single-file command queue. Worker refill/deferred logic could lose response routing, causing a permanent hang. Add an asyncio.Semaphore (min(ctx.max_running_prompts, 8)) so only a small window of requests is in-flight at any time, giving each request a clean one-to-one response path.

…ge collapse Each sample in a prompt group now receives a different strategy hint as a pre-filled assistant message, steering it toward a different first move direction. This creates diverse action sequences → diverse scores → non-zero group advantages, which is required for GSPO training signal. Co-Authored-By: Claude <noreply@anthropic.com>

Add TOOL_CALL_BONUS=15 so that valid tool calls always yield positive reward, preventing the GSPO advantage-collapse cycle where near-zero tool-call rewards cause the policy to drift toward no-tool silence. Add per-sample bonus offsets keyed on sample_index so that identical action sequences within a prompt group still receive slightly different rewards. This breaks the GSPO group-advantage collapse when all n_samples produce the same deterministic output. Co-Authored-By: Claude <noreply@anthropic.com>

…y signal Reduce TOOL_CALL_BONUS from 15 to 5 so the game-score improvement stays the dominant reward component (max ~200+ vs bonus 5). The bonus is just enough to make tool calling clearly > silence without letting the model exploit the bonus instead of pursuing merges. Shrink per-sample offsets to [0.0, 1.7] so a single merge tile (~4) easily outweighs the offset noise. Co-Authored-By: Claude <noreply@anthropic.com>

…nalty gap

…ool_call> in 2048 agent

…re-fill, bonuses, offsets

…gging

…findings

The 2048 demo is open-loop planning, not a reactive agent: the policy sees only the starting board and emits a full 32-move sequence in one choose_moves tool call, then the engine replays it deterministically at reward time. Only the first move is grounded; later moves pre-commit against spawns the policy never observes, and --disable-thinking forbids internal lookahead. With one scalar reward covering the whole sequence and reward clamped to [-1, 1], the signal saturates at the 1.0 cap before competence emerges — hence the max_tile=32 ceiling. Add a Limitations subsection stating this explicitly and point the max_tile=32 line at it, so reward=1.0 is not misread as "plays 2048 well." No code change; closed-loop play remains a future, ~32x-rollout design change the agentic API already supports. Co-Authored-By: Claude <noreply@anthropic.com>

紫霂 and others added 10 commits July 28, 2026 15:29

PasserbyPh force-pushed the feat/issue-188-2048-agentic-demo branch from a23d3de to f42a9da Compare July 30, 2026 06:38

PasserbyPh force-pushed the feat/issue-188-2048-agentic-demo branch from 1b2b00e to 7ee5807 Compare July 30, 2026 10:27

PasserbyPh marked this pull request as draft July 31, 2026 02:25

PasserbyPh force-pushed the feat/issue-188-2048-agentic-demo branch 5 times, most recently from 374f90f to c93f092 Compare July 31, 2026 03:27

紫霂 and others added 3 commits July 31, 2026 11:29

PasserbyPh force-pushed the feat/issue-188-2048-agentic-demo branch 2 times, most recently from c887ecb to 033ebb0 Compare July 31, 2026 03:36

fix(agentic): add explicit tool_call format instruction and reduce pe…

a09ef5e

…nalty gap

PasserbyPh force-pushed the feat/issue-188-2048-agentic-demo branch from 033ebb0 to a09ef5e Compare July 31, 2026 03:52

紫霂 added 4 commits July 31, 2026 12:46

fix(agentic): add stop token support to proxy, stop generation at </t…

0980a52

…ool_call> in 2048 agent

fix(agentic): simplify to match Tic-Tac-Toe pattern — remove hints, p…

b048ad7

…re-fill, bonuses, offsets

fix(agentic): forbid reasoning in system prompt to prevent token waste

4182dd3

feat(agentic): add SFT warmup data generator and loader

7bbd331

紫霂 added 2 commits July 31, 2026 15:10

fix(agentic): remove unused stop-token support added during 2048 debu…

b236be1

…gging

docs(agentic): update README with verified SFT→GRPO pipeline and key …

72402e6

…findings

PasserbyPh marked this pull request as ready for review July 31, 2026 07:31

PasserbyPh force-pushed the feat/issue-188-2048-agentic-demo branch from 35aa0ed to 2581eaa Compare August 3, 2026 03:46

Navigation Menu

Uh oh!

feat(agentic): add 2048 episode-based RL demo (#188) - #384

feat(agentic): add 2048 episode-based RL demo (#188)#384
PasserbyPh wants to merge 22 commits into
inclusionAI:mainfrom
PasserbyPh:feat/issue-188-2048-agentic-demo

PasserbyPh commented Jul 29, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

PasserbyPh commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Summary

Why

How

What

Related issue

Type of change

How was it tested?

Checklist

Breaking change details

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

PasserbyPh commented Jul 29, 2026 •

edited

Loading