Add Weighted Instruction Tuning loss to SFTTrainer

[C-anwoy]

What does this PR do?

Adds Weighted Instruction Tuning (WIT) as an opt-in SFTTrainer loss for text prompt-completion datasets:

training_args = SFTConfig(
    loss_type="wit",
    prompt_loss_weight=0.2,
    completion_loss_weight=0.8,
)

WIT assigns independent weights to prompt and completion token losses. The weighted loss is normalized by the number of tokens whose weight is non-zero, rather than by the sum of weights. Consequently, (0.0, 1.0) recovers completion-only SFT and (1.0, 1.0) recovers full-sequence language modeling.

The implementation:

• preserves prompt/completion boundaries as per-token weights through preprocessing, truncation, packing, padding, and padding-free collation;
• masks zero-weight tokens in labels, allowing current Transformers to compute the exact nonzero-token denominator across gradient-accumulation microbatches;
• consumes num_items_in_batch in the custom loss so Trainer does not apply an additional gradient-accumulation division;
• supports standard and conversational text prompt-completion datasets;
• rejects unsupported combinations explicitly (language-modeling or vision datasets, assistant-only loss, formatting functions, Liger kernels, custom loss functions, and context/sequence parallelism);
• documents the API and adds the paper to TRL's paper index; and
• adds unit, preprocessing, collation, packing, gradient-accumulation, validation, and end-to-end training tests.

Validation performed locally on the current main branch:

• make precommit
• python -m pytest -p no:rerunfailures tests/test_sft_trainer.py — 116 passed, 166 skipped
• focused WIT selection — 18 passed

Fixes #6746

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? See Add Weighted Instruction Tuning (WIT) loss to SFTTrainer #6746.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

AI writing disclosure

We welcome the use of AI tools to help with contributions. For transparency and to help us improve our review process, please indicate the level of AI involvement in this PR.

• No AI usage: the PR was written entirely by a human.
• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
• AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit ac7cce8. Configure here.}

[C-anwoy]

Hi @qgallouedec, when you have a chance, could you please take a look at this PR or route it to the appropriate reviewer?

I'm the co-first author of the WIT paper, which has been peer-reviewed and accepted to the Transactions of the Association for Computational Linguistics (TACL) -- the flagship journal of the ACL community with impact factor of 11.7 in 2025. I am available to explain the objective, validate the mathematical details, and address any requested changes.

The PR adds WIT as an opt-in SFTTrainer loss, including exact nonzero-token normalization under gradient accumulation, compatibility validation, documentation, and tests.

I have reviewed the code thoroughly from my side.
Thank you!