First, congrats on the ICLR 2026 acceptance, ExGRPO's experience-management framing is genuinely useful, and we've been building on it.
While porting ExGRPO to our own stack, we hit what looks like three compounding issues in the entropy-based trajectory selection path (trainer.exp_metric=ent, exp_select_mode=argmin, as registered in exp_scripts/run_exgrpo.sh). As far as we can tell, the selection metric that gets argmin'd is not the paper's trajectory entropy H(o) (Sec. 4.1, Experience Selection). We'd love the authors' read on whether this matches the code that produced the paper's results, and what the expected impact is.
1. Inverted padding mask
experience_helpers.py#L137-L146:
mask = ~(ppl_batch.batch['responses'][i % BATCH_SIZE] != self.dataset.tokenizer.pad_token_id)
ent = all_metrics[i][mask].mean().item()
responses != pad_token_id is True at real response tokens; the leading ~ inverts it, so the mean is taken over the complement of the response. The ppl branch (L137–L142) has the same pattern.
2. Layout mismatch with _forward_micro_batch
compute_postcal_metrics builds input_ids by tokenizing prompt + response as one string, left-padded to max_prompt_length + max_target_length, while responses is right-padded to max_target_length. But the entropy window in dp_actor.py#L117-L128 (full_entropy[:, -response_length-1:-1]) assumes verl's canonical layout where window index j aligns with responses[j]. Under the left-padded layout, the response's entropies actually sit at window indices [T−L, T−1], and (with use_remove_padding=True, per the run script) pad_input zero-fills all padded positions. So the positions selected by the mask in (1) are a mixture of exact zeros, prompt-tail entropies, and — coincidentally — the response's entropies, normalized by T−L rather than L.
3. Stale batch in the mask lookup
Candidates are scored in chunks of 8 (L74–L126), but the mask loop runs after the chunk loop, so ppl_batch (L125) still holds the last chunk. ppl_batch.batch['responses'][i % BATCH_SIZE] therefore applies the final chunk's padding layouts to all earlier candidates — with ~64 replay questions per step, most candidates are masked with an unrelated trajectory's length.
Net effect
Worked consequence (arithmetic emulation of the exact ops)
With max_prompt_length=1024, max_target_length=8192, and typical response lengths, zeros dominate the selected span and the metric collapses toward (Σ response entropy + prompt-tail constant) / (8192 − L) — increasing in length in the numerator and denominator both. In simulation:
- two candidates with identical true mean entropy rank differently by length (shorter wins);
- a shorter candidate with worse true entropy still beats a longer, better one;
- borrowed masks (issue 3) shift a candidate's score arbitrarily;
- a candidate with
L == max_target_length selects zero positions → mean() = NaN → np.argmin returns the NaN index, auto-selecting the most-likely-truncated candidate when one exists.
So argmin over this metric behaves approximately like "prefer shorter stored solutions, plus noise," rather than "prefer low-entropy solutions." The argmax mode (used for the w/ Highest Entropy ablation) inherits the same metric.
What looks unaffected: the training objective, policy shaping, recorded behavior log-probs (replace_recorded_old_log_probs), advantages, buckets, and retirement all operate on the canonical batch layout and appear correct — so the blast radius seems confined to which stored trajectory is replayed, not how it's trained on.
Questions
- Can you confirm whether the paper's reported results (Table 1, Table 3, Table 6, incl. w/o T. Selection and w/ Highest Entropy) were produced with this code path, or with a different internal revision?
- If this code path was used: do you agree the trajectory selection ablations effectively compared a length-biased noisy selector against random selection, rather than H(o) against random? What's your expected impact on those rows and on the main results?
- Have you run (or do you plan to run) with a corrected H(o) — mean full-response entropy under each candidate's own mask? Any expected direction of change?
Suggested fix
Compute per-candidate masks inside the chunk loop (or store them alongside metrics), drop the ~ inversion, and align the geometry — either build the metric batch in verl's canonical layout so window index j ↔ responses[j], or select window indices [T−L_i, T−1] per candidate — and guard the empty-selection case to avoid NaN reaching argmin. Happy to open a PR with this fix and a small unit test if useful.
Caveat: this is from static analysis plus arithmetic emulation of the tensor ops (not a run of your full stack), so apologies in advance if we've missed a config path — happy to be corrected.
First, congrats on the ICLR 2026 acceptance, ExGRPO's experience-management framing is genuinely useful, and we've been building on it.
While porting ExGRPO to our own stack, we hit what looks like three compounding issues in the entropy-based trajectory selection path (
trainer.exp_metric=ent,exp_select_mode=argmin, as registered inexp_scripts/run_exgrpo.sh). As far as we can tell, the selection metric that getsargmin'd is not the paper's trajectory entropy H(o) (Sec. 4.1, Experience Selection). We'd love the authors' read on whether this matches the code that produced the paper's results, and what the expected impact is.1. Inverted padding mask
experience_helpers.py#L137-L146:responses != pad_token_idis True at real response tokens; the leading~inverts it, so the mean is taken over the complement of the response. Thepplbranch (L137–L142) has the same pattern.2. Layout mismatch with
_forward_micro_batchcompute_postcal_metricsbuildsinput_idsby tokenizingprompt + responseas one string, left-padded tomax_prompt_length + max_target_length, whileresponsesis right-padded tomax_target_length. But the entropy window indp_actor.py#L117-L128(full_entropy[:, -response_length-1:-1]) assumes verl's canonical layout where window indexjaligns withresponses[j]. Under the left-padded layout, the response's entropies actually sit at window indices[T−L, T−1], and (withuse_remove_padding=True, per the run script)pad_inputzero-fills all padded positions. So the positions selected by the mask in (1) are a mixture of exact zeros, prompt-tail entropies, and — coincidentally — the response's entropies, normalized byT−Lrather thanL.3. Stale batch in the mask lookup
Candidates are scored in chunks of 8 (L74–L126), but the mask loop runs after the chunk loop, so
ppl_batch(L125) still holds the last chunk.ppl_batch.batch['responses'][i % BATCH_SIZE]therefore applies the final chunk's padding layouts to all earlier candidates — with ~64 replay questions per step, most candidates are masked with an unrelated trajectory's length.Net effect
Worked consequence (arithmetic emulation of the exact ops)
With
max_prompt_length=1024,max_target_length=8192, and typical response lengths, zeros dominate the selected span and the metric collapses toward(Σ response entropy + prompt-tail constant) / (8192 − L)— increasing in length in the numerator and denominator both. In simulation:L == max_target_lengthselects zero positions →mean()= NaN →np.argminreturns the NaN index, auto-selecting the most-likely-truncated candidate when one exists.So
argminover this metric behaves approximately like "prefer shorter stored solutions, plus noise," rather than "prefer low-entropy solutions." Theargmaxmode (used for the w/ Highest Entropy ablation) inherits the same metric.What looks unaffected: the training objective, policy shaping, recorded behavior log-probs (
replace_recorded_old_log_probs), advantages, buckets, and retirement all operate on the canonical batch layout and appear correct — so the blast radius seems confined to which stored trajectory is replayed, not how it's trained on.Questions
Suggested fix
Compute per-candidate masks inside the chunk loop (or store them alongside metrics), drop the
~inversion, and align the geometry — either build the metric batch in verl's canonical layout so window indexj↔responses[j], or select window indices[T−L_i, T−1]per candidate — and guard the empty-selection case to avoid NaN reachingargmin. Happy to open a PR with this fix and a small unit test if useful.Caveat: this is from static analysis plus arithmetic emulation of the tensor ops (not a run of your full stack), so apologies in advance if we've missed a config path — happy to be corrected.