Skip to content

Results: Qwen3.5 acceptance v1/v2 (execution failures) - #3

Draft
atimics wants to merge 11 commits into
codex/qwen-ilxyr-experimentsfrom
codex/qwen-experiment-rfc
Draft

Results: Qwen3.5 acceptance v1/v2 (execution failures)#3
atimics wants to merge 11 commits into
codex/qwen-ilxyr-experimentsfrom
codex/qwen-experiment-rfc

Conversation

@atimics

@atimics atimics commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Stacked experiment result

This PR is stacked on the implementation branch behind upstream leCore PR #31. PR AnOversizedMooseWithSocks#31 holds the proposed 0.2.11 implementation; this PR holds the preregistration, terminal executions, ilxyr ledgers, evidence exports, permanent publication receipts, and committee discussion.

Terminal status

COMPLETE — v2 ilxyr outcome: execution_failure — no scientific acceptance/rejection result

The authorized v2 run is over. It will not be retried under the same identity, and no v3 run is authorized.

V2 corrected the v1 interpreter-path bug and completed the substantive CPU work:

  1. exact source/model/corpus preflight passed;
  2. all 12/12 ilxyr admission gates passed;
  3. pre-install tokenizer and reference-logit gates passed;
  4. the original checkpoint was evaluated across 4,097 input tokens;
  5. the experimental installer emitted a 2,104.2 MB checkpoint;
  6. leCore reloaded it and evaluated the same 4,097 input tokens; and
  7. the required official Transformers checkpoint reload failed.

Transformers reported that the two prepended layers contain packed linear-attention tensors where the official model expects split projections, and that their MLP tensors use width 128 where the Qwen configuration declares 3,584:

linear_attn.in_proj_qkvz.weight | UNEXPECTED
linear_attn.in_proj_ba.weight   | UNEXPECTED
linear_attn.in_proj_a.weight    | MISSING
linear_attn.in_proj_z.weight    | MISSING
linear_attn.in_proj_qkv.weight  | MISSING
linear_attn.in_proj_b.weight    | MISSING
mlp.up_proj.weight              | MISMATCH: [128, 1024] vs [3584, 1024]
mlp.down_proj.weight            | MISMATCH: [1024, 128] vs [1024, 3584]
mlp.gate_proj.weight            | MISMATCH: [128, 1024] vs [3584, 1024]

This is the exact interoperability failure the mandatory reload gate was intended to expose. The leCore runtime accepts its own heterogeneous representation; the official Transformers architecture does not.

Because the exception happened before the runner's atomic metric envelope was emitted, ilxyr correctly recorded execution_failure with an empty metric set. Partial internal observations are not promoted into preregistered metrics. Official text generation and the vision-input smoke did not run.

V2 result and permanent evidence

  • Committed result: v2-20c3330d0b3e-execution-failure
  • Permanent Arweave root: f35OgXuOoulk9EcndJBNWIgZ5tmaQ6DRzeINP7ARZTU
  • Canonical publication manifest SHA-256: 21874555c7f9e73d20b9d11599c79236f8f40267ee975f703ffc053eb6b1f9ab
  • Publication receipt: publication-receipt.json
  • ilxyr evidence ref: artifact://sha256/15885a3e5fbb01a504473e3076c25c9d5d938c13f96faa340ed9600a245845cc
  • ilxyr run ref: artifact://sha256/dee0b19888937eb4b4109ef6558ce3e5c6a530469019b94f4905c01d144a6d8f
  • Verified ledger: 15 objects, 16 events, valid: true
  • Arweave verification: all 47 transactions confirmed at 0 winc; publication-manifest.json and ledger/events.jsonl were downloaded through the public path manifest and matched their committed SHA-256 values
  • Original AWS evidence bundle: s3://zero-training-022118847419/qwen35-acceptance/2f32151-2fc06364-a6c9d113/v2-formal/qwen35-evidence.tgz, SHA-256 df2ba31608f9039e9528a43a93d4766c8e8feb023668a1b9ffc1c85553210c1c

The r7i.4xlarge formal execution ran from 00:26:36.737Z to 04:51:28.293Z (4h 24m 51.556s). Estimated on-demand compute is $4.71, plus a few cents of storage. The instance self-terminated, its encrypted root volume was deleted, and the temporary zero-ingress security group was removed. The total remained well below the $10 ceiling.

Execution history

Attempt Instance Outcome Scientific result?
preflight-001 i-0b49b8662723706f0 Rejected a mistyped nonexistent source SHA No formal run
v1 i-0c549643b970e896a Wrong Python interpreter; execution_failure before model evaluation No
v2 i-059aab8bf1d3d87c9 Official Transformers rejected the emitted layer shapes/names after paired evaluation No

Frozen v2 identity

  • Experiment: lecore.qwen35.install.20c3330d0b3e.d6905f043e78.a6c9d1135a04.2f32151c6d44.v2.acceptance
  • leCore source: 2f32151c6d44c80a7e2ff98c0696650cd1d9c6f1
  • ilxyr base identity: e92382ff2a5e8714466533a160f6609b4ef9cee8
  • Model: Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17
  • Installation corpus SHA-256: d6905f043e7856b93b2dd72dac5fa0dc593898c55d6c54c51f3153f4317d6b7f
  • Held-out evaluation SHA-256: a6c9d1135a04d10955fe11d210b7f642e1c2341d4f2c8369b9a832cc97839d94
  • Spectral filtering: disabled
  • Installer: explicitly experimental

Gate record

Gate V2 record
Frozen input preflight Pass
ilxyr admission Pass (12/12)
Tokenizer/reference-logit pre-install gate Pass by execution ordering; exact partial values not promoted
Original paired stream Completed (4,097 input tokens)
Installed paired stream Completed (4,097 input tokens)
Exact metric envelope / statistical result Not emitted
leCore checkpoint reload Pass
Official Transformers checkpoint reload Fail
Official text generation Not reached
Official vision-input smoke Not reached
ilxyr ledger verification Pass
Permanent publication and public hash check Pass

Validation

  • Focused Qwen and selector suite: 22 passed.
  • Every committed JSON document parses.
  • The AWS evidence archive matches its signed SHA-256 sidecar.
  • The ilxyr ledger independently verifies all 15 objects and 16 events.
  • Native, RO-Crate, and in-toto exports were generated from a read-only isolated ilxyr snapshot.
  • The canonical publication manifest rebuilds deterministically with SHA-256 21874555…f9ab.

Committee decision requested

Accept this PR as the terminal evidence record. Keep the Qwen loader, tokenizer, configuration parsing, text runtime, and diagnostics in the 0.2.11 review. Keep spectral filtering research-only and keep the layer-prepending installer experimental; the official-reload failure is direct evidence that it is not ready for promotion.

Any future run should require a new implementation that passes an official Transformers reload fixture before seeking explicit authorization for a new experiment identity.

PR AnOversizedMooseWithSocks#30 remains the separate long-term liblecore track.

@atimics
atimics force-pushed the codex/qwen-experiment-rfc branch 8 times, most recently from 2b38d23 to 54967f0 Compare August 12, 2026 22:10
@atimics atimics changed the title RFC: Qwen3.5 layer-prepending acceptance experiment Results: Qwen3.5 acceptance v1 (execution failure) Aug 12, 2026
@atimics
atimics force-pushed the codex/qwen-experiment-rfc branch from 4bec1b7 to b456c98 Compare August 13, 2026 00:04
@atimics
atimics force-pushed the codex/qwen-experiment-rfc branch from e20f6ea to 3ed8d53 Compare August 13, 2026 00:13
@atimics atimics changed the title Results: Qwen3.5 acceptance v1 (execution failure) Results: Qwen3.5 acceptance v1/v2 (execution failures) Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants