Skip to content

Dev2: pre-flight max_len check for all categories + flag stale eval r… - #37

Merged
sp25-bai-047-wq merged 1 commit into
QuantumLogicsLabs:mainfrom
Momin0000:dev2/preflight-length-check
Aug 16, 2026
Merged

Dev2: pre-flight max_len check for all categories + flag stale eval r…#37
sp25-bai-047-wq merged 1 commit into
QuantumLogicsLabs:mainfrom
Momin0000:dev2/preflight-length-check

Conversation

@Momin0000

Copy link
Copy Markdown
Contributor

…esults

Objectives v3, Dev 2 (Data & Eval Integrity):

  1. Ran Dev 3's gradient max_len check against every other category (diff, diff w/ multi-term, diff trig/exp/log, integrate, partial, tangent_line), not just gradient.

    scripts/verification/preflight_max_len_check.py tokenizes 2000 samples per category using the exact same path as train.py's SlangDatasetLoader._tokenize (serialize_slang_math + vocab_mapping + [BOS]/[EOS] boundaries on targets only), then reports max token length vs max_len.

    Result at current max_len=32:
    diff max_src=20 max_tgt=18 OK
    diff_multiterm max_src=20 max_tgt=18 OK
    diff_trig_exp_log max_src=23 max_tgt=23 OK
    integrate max_src=20 max_tgt=18 OK
    partial max_src=20 max_tgt=18 OK
    tangent_line max_src=21 max_tgt=21 OK
    gradient max_src=25 max_tgt=40 TRUNCATING (2000/2000)

    Confirms: gradient is the only category silently truncating today. diff/integrate/partial/tangent_line are all safely under 32 tokens and need no max_len change on their own.

    Re-ran at max_len=48 (Dev 3's proposed fix): all 7 categories, including gradient, fit with headroom (worst case 40/48). Supports Dev 3's objective 2 choice of 48 as sufficient.

    Script is reusable per Dev 3 objective 3 ("pre-flight check ... run before every future training start"): python scripts/verification/preflight_max_len_check.py [--samples N] [--max-len N].

  2. docs/EVAL_RESULTS.md: flagged as stale rather than regenerated. Dev 2's regen objective depends on Dev 3's fixed checkpoint, which does not exist yet in this environment -- checkpoints/ is gitignored, and train.py's main loop currently cannot complete a run (undefined src_seq/tgt_in in the second model() call, per Dev 3 objective 1). Added a note pointing at the exact blocker and the exact regen command (python eval/run_eval.py) so the 21.7% figure isn't mistaken for current once Dev 3's other fixes land, and isn't hand-edited in the meantime.

Verified: full pytest suite run (excluding modules requiring torch/ starlette, not installed in this sandbox) -- 51 passed, 4 pre-existing failures in tests/unit/test_quotient_rule.py (quotient_rule/hessian op support, outside Dev 2 scope, unrelated to this change).

…esults

Objectives v3, Dev 2 (Data & Eval Integrity):

1. Ran Dev 3's gradient max_len check against every other category
   (diff, diff w/ multi-term, diff trig/exp/log, integrate, partial,
   tangent_line), not just gradient.

   scripts/verification/preflight_max_len_check.py tokenizes 2000 samples
   per category using the exact same path as train.py's
   SlangDatasetLoader._tokenize (serialize_slang_math + vocab_mapping +
   [BOS]/[EOS] boundaries on targets only), then reports max token length
   vs max_len.

   Result at current max_len=32:
     diff                 max_src=20  max_tgt=18   OK
     diff_multiterm        max_src=20  max_tgt=18   OK
     diff_trig_exp_log     max_src=23  max_tgt=23   OK
     integrate             max_src=20  max_tgt=18   OK
     partial               max_src=20  max_tgt=18   OK
     tangent_line          max_src=21  max_tgt=21   OK
     gradient               max_src=25  max_tgt=40   TRUNCATING (2000/2000)

   Confirms: gradient is the only category silently truncating today.
   diff/integrate/partial/tangent_line are all safely under 32 tokens and
   need no max_len change on their own.

   Re-ran at max_len=48 (Dev 3's proposed fix): all 7 categories,
   including gradient, fit with headroom (worst case 40/48). Supports
   Dev 3's objective 2 choice of 48 as sufficient.

   Script is reusable per Dev 3 objective 3 ("pre-flight check ... run
   before every future training start"): `python
   scripts/verification/preflight_max_len_check.py [--samples N]
   [--max-len N]`.

2. docs/EVAL_RESULTS.md: flagged as stale rather than regenerated.
   Dev 2's regen objective depends on Dev 3's fixed checkpoint, which does
   not exist yet in this environment -- checkpoints/ is gitignored, and
   train.py's main loop currently cannot complete a run (undefined
   src_seq/tgt_in in the second model() call, per Dev 3 objective 1).
   Added a note pointing at the exact blocker and the exact regen command
   (`python eval/run_eval.py`) so the 21.7% figure isn't mistaken for
   current once Dev 3's other fixes land, and isn't hand-edited in the
   meantime.

Verified: full pytest suite run (excluding modules requiring torch/
starlette, not installed in this sandbox) -- 51 passed, 4 pre-existing
failures in tests/unit/test_quotient_rule.py (quotient_rule/hessian
op support, outside Dev 2 scope, unrelated to this change).
@vercel

vercel Bot commented Aug 15, 2026

Copy link
Copy Markdown

@Momin0000 is attempting to deploy a commit to the seno-quantum-coder's projects Team on Vercel.

A member of the Team first needs to authorize it.

@sp25-bai-047-wq
sp25-bai-047-wq merged commit c16c7c9 into QuantumLogicsLabs:main Aug 16, 2026
1 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants