Skip to content

fix(trainer): make compute_flops_per_token invariant to embedding tying - #6731

Open
AuthRan wants to merge 1 commit into
huggingface:mainfrom
AuthRan:fix-flops-tied-untied-lm-head
Open

fix(trainer): make compute_flops_per_token invariant to embedding tying#6731
AuthRan wants to merge 1 commit into
huggingface:mainfrom
AuthRan:fix-flops-tied-untied-lm-head

Conversation

@AuthRan

@AuthRan AuthRan commented Aug 13, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes #6708.

compute_flops_per_token gave different FLOPs estimates for tied vs.
untied word embeddings, even though tying only shares parameters
between the input embedding and the lm_head — it doesn't change
what actually runs in the forward pass:

embed_flops = 2 * V * h
lm_head_flops = 0 if config.tie_word_embeddings else 2 * V * h
  • The input embedding is a lookup (gather), not a matmul, so modeling
    it as 2 * V * h FLOPs overcounts it — it should cost ~0.
  • The lm_head projection is a real matmul (hidden → vocab logits)
    and always runs, whether or not its weight matrix happens to be the
    same object as the input embedding. Zeroing it out when tied is
    incorrect.

This PR drops the embedding-lookup term entirely and makes the
lm_head matmul unconditional, so the FLOPs estimate no longer
depends on tie_word_embeddings, matching the repro in the issue
(f_untied - f_tied == 0).

Also updates the existing test_tied_vs_untied_lm_head test, which
had encoded the old (incorrect) non-zero delta as the expected
result.

Before submitting

  • Updated the regression test to assert the correct invariant; confirmed it fails against the pre-fix code and passes against the fix.
  • Ran ruff check / ruff format on the changed files.
  • Other TestComputeFlopsPerToken / TestComputeMfu / TestAdjustedMfu tests still pass (adjusted_mfu calls compute_flops_per_token internally, so it picks up the fix automatically).

Note

Low Risk
Small change to an MFU estimation helper and its test; no training, auth, or runtime model behavior is affected, though reported MFU values will differ slightly from before.

Overview
compute_flops_per_token no longer treats input embeddings as a 2×V×h matmul or skips lm_head FLOPs when tie_word_embeddings is true. The estimate now always includes the lm_head logits matmul and omits embedding lookup cost (~0 FLOPs), so tied and untied configs return the same per-token FLOPs and MFU-style metrics shift slightly downward versus the old formula.

The test_tied_vs_untied_lm_head regression test now asserts f_untied == f_tied instead of expecting a vocab-sized delta.

Reviewed by Cursor Bugbot for commit 9009eed. Bugbot is set up for automated code reviews on this repo. Configure here.

Tied and untied word embeddings produced different FLOPs estimates,
even though tying only shares parameters between the input embedding
and lm_head and doesn't change what runs in the forward pass:

- `embed_flops = 2 * V * h` treated the input embedding lookup as a
  matmul. It's a gather, not a matmul, so it costs ~0 FLOPs.
- `lm_head_flops` was zeroed out when `tie_word_embeddings` is set,
  but the lm_head projection matmul always runs to produce logits,
  regardless of whether its weight is shared with the input
  embedding.

Drop the embedding-lookup term and make the lm_head matmul
unconditional, so FLOPs no longer depend on `tie_word_embeddings`.
Updates the existing `test_tied_vs_untied_lm_head` test, which had
encoded the old (incorrect) delta as the expected result.

Closes huggingface#6708
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Accounting error in FLOPS calculation

1 participant