feat(agentic): add logic-circuit diagnosis agentic RL demo - #436
Open
wtbdev wants to merge 29 commits into
Open
Conversation
Add a new agentic RL demo where the agent diagnoses a hidden stuck-at fault in a combinational logic circuit (AND/OR/NOT gates). The agent can set input vectors (free), inspect internal nodes (costs 1 probe), and submit a diagnosis. - game.py: circuit generation (DAG by construction), evaluation, fault injection, brute-force uniqueness verification, tool schemas, scoring - run_agent.py: async multi-turn agent loop with free tool choice - reward.py: outcome + efficiency reward (correct → 0.5–1.0, wrong → 0.0) - dataset_generator.py: seeded batch generation with JSONL output - dataset_loader.py: normalization and prompt building - 25 CPU tests covering generation, evaluation, tool execution, reward scoring, and episode flow Closes inclusionAI#193 Co-Authored-By: Claude <noreply@anthropic.com>
Interactive single-page app showing the circuit topology as a layered DAG. Toggle inputs, set input vectors, click gates to probe values, and submit a diagnosis. Follows the tictactoe web_ui.py pattern with stateful HTTP server and embedded HTML/CSS/JS. Co-Authored-By: Claude <noreply@anthropic.com>
- SVG-based circuit visualization with connection lines between nodes - Layered DAG layout (inputs at bottom, gates in middle, output at top) - Click gates to probe (costs 1 probe) and select for diagnosis - Dedicated fault-type buttons for diagnosis submission - Inline probe values displayed on nodes - Better color palette matching tictactoe style - Clearer UX flow: toggle inputs → set input → probe → diagnose Co-Authored-By: Claude <noreply@anthropic.com>
…indow Replace verbose per-node listing with compact layer-by-layer format: IN0 IN1 IN2 IN3 AND4=AND(IN0,IN1) OR5=OR(IN2,IN3) Reduces prompt from ~400 to ~156 tokens for typical 12-node circuits, keeping total trajectory well under max_context_len=2048. Co-Authored-By: Claude <noreply@anthropic.com>
- _assistant_message now handles dict, Pydantic, and SimpleNamespace response formats from the areno proxy - Harder system/turn prompts requiring tool calls on every turn - Fixes 'no executable tool call' on Qwen3.5 models Co-Authored-By: Claude <noreply@anthropic.com>
- MAX_TURNS=8 keeps trajectory under context window - MAX_PROBES=5 prevents runaway probing without submission - Last turn forces submit_diagnosis Co-Authored-By: Claude <noreply@anthropic.com>
…nses Turn 1 forces set_input_vector, intermediate turns probe freely, final turn forces submit_diagnosis. Following codebreaker pattern eliminates the 'model returns text instead of tool call' problem. Also reduces max_turns to 4 for tighter context budget. Co-Authored-By: Claude <noreply@anthropic.com>
…ose defaults - Revert forced tool_choice per turn back to free choice with all 3 tools - Restore max_turns=8, max_probes=10 - Keep robust _assistant_message (dict/object compat) - Keep compact prompt format The demo is designed for models with tool-calling capability (3B+). Small-model workarounds (forced tool_choice, tiny circuits, minimal turns) are not needed for the intended use case. Co-Authored-By: Claude <noreply@anthropic.com>
New reward structure creates a gradient ladder: -1.0 no interaction at all -0.3 probed some gates but not the faulty one +0.2 probed the faulty gate (even without submitting) 0.0 submitted wrong diagnosis 0.5+ submitted correct diagnosis This gives the model stepping stones to learn from random exploration, solving the cold-start problem where all -1.0 rewards create zero gradient. Co-Authored-By: Claude <noreply@anthropic.com>
Prevents the model from outputting text instead of tool calls, which was the root cause of the policy collapse after a few steps. 0.6B tool calling is brittle — any policy shift can push it over the edge from producing valid tool calls to plain text. Co-Authored-By: Claude <noreply@anthropic.com>
With tool_choice='required', every turn produces tool calls and accumulates context. 8 turns leads to 25x training slowdown at step 8. 5 turns (observe + probe + probe + probe + submit) is sufficient for small circuits. Co-Authored-By: Claude <noreply@anthropic.com>
Each turn exposes exactly one tool with forced tool_choice, eliminating format ambiguity entirely. The model's decisions are in the arguments (which inputs? which node? which fault type?), not in which tool to call. Sequence: set_input_vector -> inspect_node x3 -> submit_diagnosis Co-Authored-By: Claude <noreply@anthropic.com>
Dumps the model's raw response content when no valid tool call is found, so we can see what the model actually outputs during format collapse. Co-Authored-By: Claude <noreply@anthropic.com>
…erence Previously only inspect_node counted as 'interaction'. Now any valid tool call (set_input_vector, inspect_node, submit_diagnosis) moves the reward from -1.0 to -0.2. This gives gradient signal just for maintaining tool-call format, directly combating format drift. Co-Authored-By: Claude <noreply@anthropic.com>
Layer 1 (format): scores raw completion text for JSON proximity. Creates per-sample variance even when tool-call parsing fails, preventing the zero-advantage deadlock that caused periodic collapse. Layer 2 (interaction): rewards valid tool calls and probing the faulty gate (+0.1 per call, +0.2 for hitting the right gate). Layer 3 (outcome): rewards correct diagnosis with efficiency bonus (0.5 + 0.2 * efficiency). Range: [-0.05, 1.0]. Key property: two different model outputs almost never get identical format scores, so advantages are never all-zero and gradients always flow. Co-Authored-By: Claude <noreply@anthropic.com>
This reverts commit e87863d.
When no tool calls are parsed, use record.completion text to assign per-sample penalties [-1.0, -0.3]. Two samples with different output get different penalties → advantages are never all-zero → no deadlock. Penalty structure: -1.0 empty output -0.5 text, no JSON structure -0.3 text with JSON features (quotes, braces, digits) Co-Authored-By: Claude <noreply@anthropic.com>
Turns 2-4 now expose both inspect_node and submit_diagnosis with no forced tool_choice, so the model can submit when confident instead of being forced to probe. Turn 1 and final turn still force single tools for format stability. Co-Authored-By: Claude <noreply@anthropic.com>
- web_ui.py: --agent flag connects to areno serve endpoint, Agent Move button lets the model play one turn at a time - dashboard.py: local training log parser with charts (not pushed) Co-Authored-By: Claude <noreply@anthropic.com>
Previously _handle_agent rebuilt messages from scratch each call, so the model had no memory of prior inputs, probes, or results. Now the server stores conversation history and appends each turn, including assistant messages and tool results. Co-Authored-By: Claude <noreply@anthropic.com>
…replay Replace heuristic process rewards with information-gain shaping: - Replays each valid tool call against candidate fault hypotheses - Computes progress as normalized entropy reduction in the hypothesis set (log-scale information gain) - Correct diagnosis: 0.8 + 0.2 × efficiency (always positive) - Wrong submission: negative but less so when observations were informative (min(0.30×progress − 0.02×probes − 0.35, −0.05)) - No submission: min(0.30×progress − 0.02×probes − 0.55, −0.25) - No valid actions: −1.0 - Never rewards direct access to the hidden fault — every candidate is filtered only through observable outputs Co-Authored-By: wtbdev <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
- run_agent: add probe budget check to _tool_inspect_node - run_agent: fix probes_remaining to use per-episode max_probes - run_agent: narrow except Exception to specific exceptions - web_ui: init and reset _last_probed_value, prevent AttributeError - web_ui: pop orphaned turn prompt on LLM exception - web_ui: add exhaustion guard in _new_game retry loop - web_ui JS: fix probed CSS only on actually probed nodes - web_ui JS: re-enable autoBtn after single agentMove - dataset_loader: log warning on missing fault field - reward: no change per design decision Co-Authored-By: Claude <noreply@anthropic.com>
Accept circuits with 0-2 extra live gates instead of requiring exact match. Quality preserved: brute-force verify, fault-output check, and dedup all still apply. n_gates always reflects actual live count. 2.8x faster generation. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements issue #193: a logic-circuit diagnosis agentic RL demo.
Generates AND/OR/NOT circuits with one injected stuck-at fault. The agent can set input vectors (free), inspect node outputs (1 probe), and submit the faulty gate — while the reference circuit is hidden.
Changes
examples/agentic/logic_diagnosis/:tests/test_agentic_logic_diagnosis_example_cpu.py— 25 CPU tests covering generation, evaluation, tool execution, reward scoring, and episode flow.Design highlights
Training
Closes #193
🤖 Generated with Claude Code