Skip to content

Align the chunked-loss config access between SFT and Distillation - #6744

Open
qgallouedec wants to merge 3 commits into
mainfrom
align-chunked-ce-config-access
Open

Align the chunked-loss config access between SFT and Distillation#6744
qgallouedec wants to merge 3 commits into
mainfrom
align-chunked-ce-config-access

Conversation

@qgallouedec

@qgallouedec qgallouedec commented Aug 13, 2026

Copy link
Copy Markdown
Member

Both trainers resolve the same per-model logit post-processing (logit_scale / output_multiplier / final_logit_softcapping) for their chunked loss, but reached it two different ways. DistillationTrainer uses config.get_text_config(); SFT branched on an is_vlm flag.

  • _patch_chunked_ce_lm_head now reads model.config.get_text_config(), so the resolution block is word-for-word identical to the one in distillation_trainer.py.
  • is_vlm stays — it still gates two transformers < 5.0.0 fallbacks (VLMs set base_model_prefix = "", and take config-level MoE aux-loss parameters) — but its docstring no longer claims a config-reading role it no longer has.
  • TestPatchChunkedCELMHead + TestChunkedCrossEntropyLoss: 79 passed.

Note

Low Risk
Narrow config-resolution change in chunked CE training path; behavior should match transformers’ canonical get_text_config() and existing distillation logic, with tests reported passing.

Overview
_patch_chunked_ce_lm_head now resolves logit_scale, output_multiplier, final_logit_softcapping, and MoE output_router_logits via model.config.get_text_config(), matching DistillationTrainer instead of branching on is_vlm (text_config vs top-level config).

The is_vlm argument is unchanged for transformers < 5.0.0 (backbone via model.model, config-level MoE aux-loss fields); its docstring no longer describes config selection. Inline comments were updated to document get_text_config() and the logit_scale / output_multiplier fallback behavior.

Reviewed by Cursor Bugbot for commit 834ec80. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Base automatically changed from add-muse-glimmer-chat-template to main August 18, 2026 03:29
@qgallouedec
qgallouedec force-pushed the align-chunked-ce-config-access branch from fd4a9ec to fb1e276 Compare August 18, 2026 16:24
@qgallouedec
qgallouedec requested a review from cmpatino August 18, 2026 16:27
@qgallouedec
qgallouedec force-pushed the align-chunked-ce-config-access branch from 4ccfdc5 to 834ec80 Compare August 18, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant