Dev2: pre-flight max_len check for all categories + flag stale eval r… - #37
Merged
sp25-bai-047-wq merged 1 commit intoAug 16, 2026
Conversation
…esults
Objectives v3, Dev 2 (Data & Eval Integrity):
1. Ran Dev 3's gradient max_len check against every other category
(diff, diff w/ multi-term, diff trig/exp/log, integrate, partial,
tangent_line), not just gradient.
scripts/verification/preflight_max_len_check.py tokenizes 2000 samples
per category using the exact same path as train.py's
SlangDatasetLoader._tokenize (serialize_slang_math + vocab_mapping +
[BOS]/[EOS] boundaries on targets only), then reports max token length
vs max_len.
Result at current max_len=32:
diff max_src=20 max_tgt=18 OK
diff_multiterm max_src=20 max_tgt=18 OK
diff_trig_exp_log max_src=23 max_tgt=23 OK
integrate max_src=20 max_tgt=18 OK
partial max_src=20 max_tgt=18 OK
tangent_line max_src=21 max_tgt=21 OK
gradient max_src=25 max_tgt=40 TRUNCATING (2000/2000)
Confirms: gradient is the only category silently truncating today.
diff/integrate/partial/tangent_line are all safely under 32 tokens and
need no max_len change on their own.
Re-ran at max_len=48 (Dev 3's proposed fix): all 7 categories,
including gradient, fit with headroom (worst case 40/48). Supports
Dev 3's objective 2 choice of 48 as sufficient.
Script is reusable per Dev 3 objective 3 ("pre-flight check ... run
before every future training start"): `python
scripts/verification/preflight_max_len_check.py [--samples N]
[--max-len N]`.
2. docs/EVAL_RESULTS.md: flagged as stale rather than regenerated.
Dev 2's regen objective depends on Dev 3's fixed checkpoint, which does
not exist yet in this environment -- checkpoints/ is gitignored, and
train.py's main loop currently cannot complete a run (undefined
src_seq/tgt_in in the second model() call, per Dev 3 objective 1).
Added a note pointing at the exact blocker and the exact regen command
(`python eval/run_eval.py`) so the 21.7% figure isn't mistaken for
current once Dev 3's other fixes land, and isn't hand-edited in the
meantime.
Verified: full pytest suite run (excluding modules requiring torch/
starlette, not installed in this sandbox) -- 51 passed, 4 pre-existing
failures in tests/unit/test_quotient_rule.py (quotient_rule/hessian
op support, outside Dev 2 scope, unrelated to this change).
|
@Momin0000 is attempting to deploy a commit to the seno-quantum-coder's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…esults
Objectives v3, Dev 2 (Data & Eval Integrity):
Ran Dev 3's gradient max_len check against every other category (diff, diff w/ multi-term, diff trig/exp/log, integrate, partial, tangent_line), not just gradient.
scripts/verification/preflight_max_len_check.py tokenizes 2000 samples per category using the exact same path as train.py's SlangDatasetLoader._tokenize (serialize_slang_math + vocab_mapping + [BOS]/[EOS] boundaries on targets only), then reports max token length vs max_len.
Result at current max_len=32:
diff max_src=20 max_tgt=18 OK
diff_multiterm max_src=20 max_tgt=18 OK
diff_trig_exp_log max_src=23 max_tgt=23 OK
integrate max_src=20 max_tgt=18 OK
partial max_src=20 max_tgt=18 OK
tangent_line max_src=21 max_tgt=21 OK
gradient max_src=25 max_tgt=40 TRUNCATING (2000/2000)
Confirms: gradient is the only category silently truncating today. diff/integrate/partial/tangent_line are all safely under 32 tokens and need no max_len change on their own.
Re-ran at max_len=48 (Dev 3's proposed fix): all 7 categories, including gradient, fit with headroom (worst case 40/48). Supports Dev 3's objective 2 choice of 48 as sufficient.
Script is reusable per Dev 3 objective 3 ("pre-flight check ... run before every future training start"):
python scripts/verification/preflight_max_len_check.py [--samples N] [--max-len N].docs/EVAL_RESULTS.md: flagged as stale rather than regenerated. Dev 2's regen objective depends on Dev 3's fixed checkpoint, which does not exist yet in this environment -- checkpoints/ is gitignored, and train.py's main loop currently cannot complete a run (undefined src_seq/tgt_in in the second model() call, per Dev 3 objective 1). Added a note pointing at the exact blocker and the exact regen command (
python eval/run_eval.py) so the 21.7% figure isn't mistaken for current once Dev 3's other fixes land, and isn't hand-edited in the meantime.Verified: full pytest suite run (excluding modules requiring torch/ starlette, not installed in this sandbox) -- 51 passed, 4 pre-existing failures in tests/unit/test_quotient_rule.py (quotient_rule/hessian op support, outside Dev 2 scope, unrelated to this change).