Add Muse Glimmer support and testing

[qgallouedec]

Released 2026-08-10: meta-models/Muse-Glimmer-30B — Meta's Apache-2.0 agentic VLM (2B Perception Encoder + 28B dense text decoder), shipped in transformers 5.15.0.

• Tiny model script muse_glimmer_for_conditional_generation.py (MuseGlimmerForConditionalGeneration), pinned to transformers==5.15.0. Keeps one layer of each type on both towers (text: sliding / full-attention, where full-attention layers are the NoPE ones via layer_rope_theta 0; vision: window / full-attention), and scales out_hidden_size to vision hidden_size * merge_size ** 2, which the vision adapter requires. Config diff vs the reference is 14 entries, all scale-downs.
• Tests: VLM param lists in test_sft_trainer.py (3), test_dpo_trainer.py, test_kto_trainer.py.
• Tiny model pushed: trl-internal-testing/tiny-MuseGlimmerForConditionalGeneration.

Not in this PR: the chat template. Muse Glimmer ships a native response_template, so parsing works out of the box, but the template has no {% generation %} markers — SFT assistant_only_loss=True needs a vendored training variant, and its <atem:function_calls> tool syntax is a new family. Follow-up PR.

---

Note

Medium Risk
Changes affect how scaled logits are applied in SFT chunked loss and distillation (including Liger guards) for VLMs; behavior is a targeted fallback but incorrect scaling would skew training objectives.

Overview
Adds Muse Glimmer (MuseGlimmerForConditionalGeneration, transformers ≥5.15.0) to TRL’s tiny-model workflow and VLM test matrix.

A new generator script builds a minimal checkpoint from meta-models/Muse-Glimmer-30B (short text/vision stacks with sliding/window vs full-attention layers, scaled out_hidden_size for the vision adapter) and runs the usual smoke/dtype/push steps. SFT, DPO, and KTO tests now include trl-internal-testing/tiny-MuseGlimmerForConditionalGeneration, gated below transformers 5.15.0.

SFT chunked CE and distillation chunked JSD / Liger compatibility checks now read logit scaling from text_config via get_text_config() on VLMs and treat Muse Glimmer’s output_multiplier like Cohere-style logit_scale, so custom loss paths match the model’s real forward.

^{Reviewed by Cursor Bugbot for commit 5094979. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5de1b84. Configure here.}