Skip to content

feat(dx): add human-readable loss-mask explainer (#221) - #426

Open
3367337929 wants to merge 2 commits into
inclusionAI:mainfrom
3367337929:feat/loss-mask-explainer
Open

feat(dx): add human-readable loss-mask explainer (#221)#426
3367337929 wants to merge 2 commits into
inclusionAI:mainfrom
3367337929:feat/loss-mask-explainer

Conversation

@3367337929

@3367337929 3367337929 commented Jul 31, 2026

Copy link
Copy Markdown

Summary

Closes #221

This PR adds a human-readable loss-mask explainer that consumes packer output (token ids + loss mask) and maps it back to conversational structure — roles, turns, text previews, and per-span trainable token counts.

What it does

The explainer takes the packer's actual output and maps each contiguous masked/unmasked region back to its conversational role, turn index, and text preview. It does not reimplement mask rules — it reads the packer output directly to ensure the explanation always matches training behaviour.

Key features:

  • Maps token-level loss mask back to roles (system/user/assistant/tool), turns, and text previews
  • Reports per-span token count, trainable count, and mask ratio
  • Terminal table + structured JSON dual output
  • Text previews truncated by default (50 chars); full text via --show-full-text
  • SFT mask: only assistant tokens trainable
  • Agentic mask: tool-result tokens suppressed per LossMaskPolicy

New files

  • areno/api/loss_mask_explainer.pyMaskSpan, LossMaskReport, LossMaskExplainer.explain()
  • tests/test_loss_mask_explainer_cpu.py — 20 CPU tests
  • docs/troubleshooting/loss-mask-explainer.rst — user documentation

Modified files

  • areno/cli/inspect.py — add loss-mask sub-command to existing inspect group

Key design decisions

  • Consume packer output, not reimplement mask rules — ensures consistency with real training behaviour
  • Default omits full text — per acceptance criteria, text previews truncated unless --show-full-text passed
  • No new dependencies — reuses apply_chat_template_with_options, normalize_messages, resolve_model_ref, load_tokenizer
  • No model weights loaded — CPU only, no GPU validation needed
  • Backward compatible — opt-in CLI command, no changes to existing behaviour
  • Code style matches project conventions@dataclass(slots=True), docstrings, comments follow data.py/rewards.py/openai_chat.py patterns

Testing

All 20 CPU tests pass:

Ran 20 tests in 0.010s — OK

Covers: SFT mask (assistant-only), agentic mask (tool-result suppressed), all-masked, none-masked, truncated sample, malformed input (empty messages, mismatched lengths, unknown role), CLI integration (text/json/full-text), backward compatibility.

Verified on real model (Qwen/Qwen3-0.6B) on Kaggle — 31 tokens, 10 trainable (32.3%), system/user 0%, assistant 100%.
combined_screenshot

Acceptance criteria checklist

  • Token-for-token alignment with SFT and agentic packer outputs
  • Cover all-masked and truncated samples
  • Terminal/JSON output omits full text unless explicitly requested
  • Uses existing AReno contracts; no external database or sandbox
  • Default behavior remains backward compatible
  • Focused tests cover success, invalid input, and boundary/failure paths
  • User documentation includes a minimal runnable example and explains observable output

LiuYunhui added 2 commits July 30, 2026 17:58
Add a human-readable loss-mask explainer that consumes packer output
(token ids + loss mask) and maps it back to conversational structure:
roles, turns, text previews, and per-span trainable token counts.

New files:
- areno/api/loss_mask_explainer.py: MaskSpan, LossMaskReport,
  LossMaskExplainer.explain() — consumes packer output, does not
  reimplement mask rules
- areno/cli/inspect.py: CLI entry with chat-template (inclusionAI#220) and
  loss-mask (inclusionAI#221) sub-commands
- tests/test_loss_mask_explainer_cpu.py: 20 CPU tests covering SFT,
  agentic, all-masked, truncated, malformed input, CLI integration,
  and backward compatibility
- docs/troubleshooting/loss-mask-explainer.rst: user documentation

Closes inclusionAI#221
The loss-mask explainer CLI (inspect.py) imports ChatTemplateInspector
from the chat-template module. Include inclusionAI#220 files so the branch is
self-contained and testable on Kaggle.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a human-readable loss-mask explainer

1 participant