feat: harden Qwen 3.5/3.6 LoRA distributed training - #4

Draft

NolanHo wants to merge 157 commits into

mainfrom

audit/qwen35-qwen36-megatron

Commits on Jul 16, 2026

feat: harden qwen hybrid lora distributed training
NolanHo
committed
1d023d0
feat: persist dynamic lora and add replicated dp
NolanHo
committed
e2a03a9
fix: make native accumulation and multi-lora updates logical-step aware
NolanHo
committed
ae75152
test: cover native micro-step optimizer boundary
NolanHo
committed
0794e6a
fix: invalidate native lora graph between microbatches
NolanHo
committed
0530e79
docs: clarify native loss and parallel reduction semantics
NolanHo
committed
f65af15
docs: describe multi-lora logical update
NolanHo
committed
538cb5e
feat: add Megatron-style parallel topology
NolanHo
committed
4e242ee
fix: restore native optimizer step on checkpoint load
NolanHo
committed
afcf091
feat: isolate dynamic lora optimizer state
NolanHo
committed
5a58d6f
feat: add latent-rank tensor parallel lora
NolanHo
committed
e75b3c5
feat: accumulate native lora gradients in fp32
NolanHo
committed
5aa4ea4
feat: add rank-aware tp lora checkpoints
NolanHo
committed
ed25daa
fix: reject unsafe native parallel topologies
NolanHo
committed
ff382a0
docs: record abi11 tp checkpoint evidence
NolanHo
committed
2702bf0
fix: reject native pp and cp topologies
NolanHo
committed
76eace6
docs: record native pp cp rejection
NolanHo
committed
56a6ea1
fix: make dynamic lora dp and ep numerically correct
NolanHo
committed
b7050ea
docs: record dp ep and optimizer verification
NolanHo
committed
d6f897c
feat: prototype variable split ep token a2a
NolanHo
committed
bfa3d8e
feat: add sharded fixed lora ep a2a
NolanHo
committed
1a72183
docs: add native ep benchmark evidence
NolanHo
committed
b51e9c1

Commits on Jul 17, 2026

fix: normalize dynamic sharded EP LoRA gradients
NolanHo
committed
0f39f07
feat: add frozen dense MLP tensor parallelism
NolanHo
committed
54bcce5
feat: add qwen attention tensor parallelism
NolanHo
committed
e5bd9c4
feat: add qwen gdn tensor parallelism
NolanHo
committed
69c4153
feat: add qwen tensor data parallelism
NolanHo
committed
99aecef
feat: add vocabulary tensor parallelism
NolanHo
committed
b13bbbd
feat: compose Qwen LoRA tensor and expert parallelism
NolanHo
committed
f6da76a
feat: pack Qwen expert-parallel dispatch
NolanHo
committed
f19a1d1
feat: shard Qwen experts across tensor parallel ranks
NolanHo
committed
e21dd13
feat: add distributed LoRA checkpoint v5
NolanHo
committed
b1c51a5
feat: add distributed Qwen adapter export and resume
NolanHo
committed
ed3a510
feat: validate Qwen tri-axis LoRA training
NolanHo
committed
e439f5c

Commits on Jul 18, 2026

feat: add resilient source-sharded server dispatch
NolanHo
committed
de5887c
feat: add transactional distributed Qwen checkpoints
NolanHo
committed
fbf0e70
feat: add binary tensor slab transport
NolanHo
committed
c51561f
fix: restore heterogeneous LoRA checkpoints
NolanHo
committed
a3edf09
feat: make dynamic Adam transactional
NolanHo
committed
bbe0aa6
feat: train heterogeneous dynamic adapters
NolanHo
committed
9532174
perf: unify heterogeneous LoRA finalization
NolanHo
committed
6111ae5
fix: enforce distributed LoRA optimizer invariants
NolanHo
committed
dbb313b
perf: group LoRA gradient collectives
NolanHo
committed
9fb792b
perf: slab LoRA gradient accumulators
NolanHo
committed
887fb72
fix: guard LoRA registry mutations
NolanHo
committed
ac3700b
fix: make LoRA registration transactional
NolanHo
committed
9b627c0
perf: scale EP dispatch metadata packing
NolanHo
committed
a3be24e
perf: fuse GDN alpha beta projection
NolanHo
committed
474a8df
fix: align GDN QK normalization epsilon semantics
NolanHo
committed
97ce513
fix: reject unsafe GDN padding layouts
NolanHo
committed
546afc7
perf: scope multi-LoRA checkpoint projection caches
NolanHo
committed
684eadc
perf: skip strict right padding in GDN kernels
NolanHo
committed
7f96893
perf: gate multi-LoRA training trace logs
NolanHo
committed
cdebe91
fix: enumerate dynamic LoRA registries without truncation
NolanHo
committed
6c1602f
fix: exclude fixed adapter from train-all selection
NolanHo
committed
1de20b5
perf: pack LoRA gradient collectives and guard batch registration
NolanHo
committed
e45e11a
perf: add optional GDN state checkpoints
NolanHo
committed
d1ee7e0
perf: bound EP dispatch and preserve async runtime
NolanHo
committed
d254836
perf: batch heterogeneous LoRA tenants by padded rank
NolanHo
committed
2561489
perf: fuse GDN recurrent backward sweep
NolanHo
committed
4f7069e
perf: coalesce opt-in multi-LoRA server batches
NolanHo
committed
d7181bd
perf: add chunkwise GDN and tenant loss reports
NolanHo
committed
b6efba1
feat: add five-axis Qwen communicator foundation
NolanHo
committed
778359c
feat: add pipeline stage ownership ABI
NolanHo
committed
1a6a2fb
feat: wire pipeline-local qwen training ownership
NolanHo
committed
5d12729
fix: preserve global dynamic lora layer identity
NolanHo
committed
381062d
feat: wire pipeline-local qwen server contexts
NolanHo
committed
d37005d
feat: enable pipeline stage training and stage-union checkpoints
NolanHo
committed
ba2645d
fix: validate multi-lora source geometry before dispatch
NolanHo
committed
623adfd
feat: isolate dynamic tenant learning rates
NolanHo
committed
ac196fe
fix: accumulate pipeline microsteps across stages
NolanHo
committed
45d9528
fix: include pipeline microstep weight in stage consensus
NolanHo
committed
153890c
feat: stage PP2 pipeline and dynamic LoRA optimizer LR
NolanHo
committed
2e53bdb
test: verify PP2 LoRA optimizer parity
NolanHo
committed
c8434f5
fix: attach router aux loss to fixed LoRA training
NolanHo
committed
87a797a
feat: add isolated dynamic LoRA evaluation
NolanHo
committed
b5aa2fa
feat: expose selected multi-lora evaluation routes
NolanHo
committed
8672f54
feat: guard dynamic LoRA optimizer steps
NolanHo
committed
cf16db0
docs: record H20 adapter step validation
NolanHo
committed
36a0eaa
feat: add TP2 sequence-parallel collectives
NolanHo
committed
99aa1db
feat: enable guarded TP2 sequence parallel training
NolanHo
committed
1a93a75
test: add TP2 sequence parallel native smoke
NolanHo
committed
6d5e1ac
docs: record guarded sequence parallel scope
NolanHo
committed
ccf99f1

Commits on Jul 19, 2026

test: assert sequence parallel LoRA update
NolanHo
committed
ccc4e6f
fix: validate every pipeline microbatch contract
NolanHo
committed
c538540
fix: quarantine failed dynamic lora recovery
NolanHo
committed
711ea45
test: verify distributed lora recovery quarantine
NolanHo
committed
110fcf9
fix: fail stop poisoned lora worker groups
NolanHo
committed
f66e2aa
perf: pack dynamic lora preflight collectives
NolanHo
committed
19b8b4a
feat: generalize fixed lora pipeline parallelism
NolanHo
committed
db65b44
fix: synchronize first pipeline metadata errors
NolanHo
committed
48263fb
feat: isolate dynamic tenant Adam hyperparameters
NolanHo
committed
2515a85
fix: reject undersized linear attention chunks
NolanHo
committed
b29ac1d
feat: add CP2 headwise GDN training
NolanHo
committed
7ab8ece
perf: reuse fixed LoRA cache across pipeline window
NolanHo
committed
6dbf430
bench: add CP2 GDN performance coverage
NolanHo
committed
ded23a0
fix: inherit optimizer defaults in EP benchmark
NolanHo
committed
391a0c3
fix: harden qwen LoRA distributed training
NolanHo
committed
45ecd50
feat: add distributed LoRA gradient clipping
NolanHo
committed
e1460cd
fix: harden dynamic LoRA pipeline lifecycle
NolanHo
committed
998d863
docs: correct H20 toolchain status
NolanHo
committed
c67232f
feat: add guarded dynamic LoRA pipeline window
NolanHo
committed
58f49fa
docs: record guarded dynamic pipeline scope
NolanHo
committed
7c3199a
merge origin/main into qwen audit branch
NolanHo
committed
ef32dff
test: cover CUDA allocator fraction validation
NolanHo
committed
0133f6e
fix: guard dynamic pipeline adapter composition and commit
NolanHo
committed
598af90
Add selected dynamic PP loss reporting
NolanHo
committed
387fbe6
Harden selected PP requests and add opt-in fused QKV
NolanHo
committed
c618cd1
Enable configurable dynamic PP microbatches
NolanHo
committed
73948d8
Validate dynamic PP microbatch tensor shapes
NolanHo
committed
63da777
Include fused QKV mode in distributed identity
NolanHo
committed
811b906
Format dynamic pipeline server wiring
NolanHo
committed
2ddd902
Test PP multi-LoRA row admission
NolanHo
committed
3761f12
Fix native fused QKV compilation
NolanHo
committed
e1c174a
Test dynamic PP isolation and fused QKV oracle
NolanHo
committed
8069506
Add opt-in fused MLP FC1 path
NolanHo
committed
d0c24c9
Enable dynamic GDN context parallel LoRA
NolanHo
committed
418faad
Enable dynamic LoRA sequence parallel mask path
NolanHo
committed
b56b12f
Harden heterogeneous dynamic PP windows
NolanHo
committed
00ea92a
Add opt-in fused GDN CP exchange
NolanHo
committed
0bfb82a
Enable guarded dynamic MTP normalization
NolanHo
committed
adf9613
Add DP2 dynamic MTP oracle
NolanHo
committed
5892152
Enable guarded TP2 dense MTP LoRA path
NolanHo
committed
cb7ab3b

Commits on Jul 20, 2026

Extend Qwen3.6 MTP to guarded MoE and EP plumbing
NolanHo
committed
d4ea960
Validate Qwen3.6 MTP across EP and vocabulary TP
NolanHo
committed
b67495b
Remove non-beneficial MoE overlap prototype
NolanHo
committed
165a6a2
Reduce dynamic LoRA host synchronization fences
NolanHo
committed
cb5d81c
Optimize sparse heterogeneous LoRA registry selection
NolanHo
committed
ff5613a
Negotiate dynamic multi-LoRA HTTP capabilities
NolanHo
committed
29f97f7
Reduce large-world MoE metadata transfers
NolanHo
committed
2dfe8fe
Add binary multi-LoRA training ingress
NolanHo
committed
feacc81
Bound dynamic LoRA rank inflation
NolanHo
committed
7a0bedd
Record H20 PP and CP boundary validation
NolanHo
committed
734dff7
Keep bounded multi-LoRA layout buckets open
NolanHo
committed
25d0373
Record A2A async overlap benchmark result
NolanHo
committed
717c902
Add opt-in pinned host staging
NolanHo
committed
f553d54
Lock PP scheduler fail-closed contract
NolanHo
committed
add3eb1
Document PP scheduler guard
NolanHo
committed
033a093
Overlap multi-LoRA batching deadline with dispatch
NolanHo
committed
40e52ca
Add guarded full-attention CP2 bridge
NolanHo
committed
9baecde
Document CP2 full-attention verification
NolanHo
committed
6552570
perf: fuse shared QKV LoRA input projection
NolanHo
committed
3431c8b
perf: add guarded MoE shared expert overlap
NolanHo
committed
4da25a0
perf: persist dynamic adapter ID index
NolanHo
committed
7692e3e
perf: pack MoE A2A metadata
NolanHo
committed
282fd4a
perf: lazy dynamic optimizer state and trim GDN barriers
NolanHo
committed
140cdc8
perf: reuse EP A2A prefix plans
NolanHo
committed
baedae0
perf: fuse packed MoE weighted unpermute
NolanHo
committed
330b470
perf: pool dynamic Adam transaction shadows
NolanHo
committed
11343a1
perf: fuse MoE local expert permutation
NolanHo
committed
04befa4
perf: overlap EP count metadata with packing
NolanHo
committed
43e46e6
feat: snapshot dynamic Adam state on host
NolanHo
committed
4a501a8
perf: keep restored Adam state on host
NolanHo
committed
fac291a
perf: page cold dynamic Adam state to host
NolanHo
committed
6c160bb
test: compare paged Adam snapshots on CPU
NolanHo
committed
73ba2a4
docs: record dynamic Adam host paging evidence
NolanHo
committed
a8f1909
feat: add arbitrary context parallel ring attention
NolanHo
committed
0305f94