fix(trainer): make compute_flops_per_token invariant to embedding tying - #6731
Open
AuthRan wants to merge 1 commit into
Open
fix(trainer): make compute_flops_per_token invariant to embedding tying#6731AuthRan wants to merge 1 commit into
AuthRan wants to merge 1 commit into
Conversation
Tied and untied word embeddings produced different FLOPs estimates, even though tying only shares parameters between the input embedding and lm_head and doesn't change what runs in the forward pass: - `embed_flops = 2 * V * h` treated the input embedding lookup as a matmul. It's a gather, not a matmul, so it costs ~0 FLOPs. - `lm_head_flops` was zeroed out when `tie_word_embeddings` is set, but the lm_head projection matmul always runs to produce logits, regardless of whether its weight is shared with the input embedding. Drop the embedding-lookup term and make the lm_head matmul unconditional, so FLOPs no longer depend on `tie_word_embeddings`. Updates the existing `test_tied_vs_untied_lm_head` test, which had encoded the old (incorrect) delta as the expected result. Closes huggingface#6708
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #6708.
compute_flops_per_tokengave different FLOPs estimates for tied vs.untied word embeddings, even though tying only shares parameters
between the input embedding and the
lm_head— it doesn't changewhat actually runs in the forward pass:
it as
2 * V * hFLOPs overcounts it — it should cost ~0.lm_headprojection is a real matmul (hidden → vocab logits)and always runs, whether or not its weight matrix happens to be the
same object as the input embedding. Zeroing it out when tied is
incorrect.
This PR drops the embedding-lookup term entirely and makes the
lm_headmatmul unconditional, so the FLOPs estimate no longerdepends on
tie_word_embeddings, matching the repro in the issue(
f_untied - f_tied == 0).Also updates the existing
test_tied_vs_untied_lm_headtest, whichhad encoded the old (incorrect) non-zero delta as the expected
result.
Before submitting
ruff check/ruff formaton the changed files.TestComputeFlopsPerToken/TestComputeMfu/TestAdjustedMfutests still pass (adjusted_mfucallscompute_flops_per_tokeninternally, so it picks up the fix automatically).Note
Low Risk
Small change to an MFU estimation helper and its test; no training, auth, or runtime model behavior is affected, though reported MFU values will differ slightly from before.
Overview
compute_flops_per_tokenno longer treats input embeddings as a2×V×hmatmul or skipslm_headFLOPs whentie_word_embeddingsis true. The estimate now always includes thelm_headlogits matmul and omits embedding lookup cost (~0 FLOPs), so tied and untied configs return the same per-token FLOPs and MFU-style metrics shift slightly downward versus the old formula.The
test_tied_vs_untied_lm_headregression test now assertsf_untied == f_tiedinstead of expecting a vocab-sized delta.Reviewed by Cursor Bugbot for commit 9009eed. Bugbot is set up for automated code reviews on this repo. Configure here.