Skip to content

codegen: byte-load/store merging + bswap — no wide-access collapse exists; crypto measured −11-12% #367

Description

@kromych

Byte-assembly readers/writers never collapse to wide accesses: no Inst::Bswap exists (ir.rs), neither backend has a rev/bswap encoder, and the byte-loop forms (tweetnacl dl64/ts64/st32, tweetnacl.c:35-52) are 4-block phi loops the inliner rejects ("disallowed inst Phi"), so each dl64 call costs ~70 dynamic insns vs clang's ldr+rev.

Measured (host arm64, crypto fixture): dl64+ts64 self-time = 12.2% of samples; source-level simulation of the merge = −10.3% (output-correct ALU swap) to −12.4% (verifier re-run, wide load/store); a hardware rev lands ~−11-12% (CI arm64 1056 -> ~940 ms). Corpus reach: 11 snapshot fixtures match the load-side shape; sqlite3Get4byte/Put4byte compile the byte OR-tree fallback under badc; miniz MZ_READ_LE16/32 are byte trees on arm64 (111 sites).

Design (three parts, in order):

  1. Inst::Bswap{value, kind U16/U32/I64}: aarch64 rev16/rev(w)/rev(x) encoders, x64 bswap r32/r64 (0F C8+rd) + ror r16,8; VM arm (swap_bytes) for --interp parity; mechanical additions to the exhaustive Inst matches.
  2. Bounded byte-loop unroll via the unroll-as-inlining machinery (mid-end: const-trip loop unrolling (as inlining) + array scalar promotion; crypto's copy loops are 72% of its round #360), gated to byte-assembly candidates: single-body-block loop, phi-pair header, Bz on BinopI{ult/lt,i,K}, step +1, trip <= 8, body <= ~12 insts, no calls, a U8 Load or I8 Store feeding a shift/or accumulate. Trip evaluation via the shared VM evaluator (mid-end: post-inline constant folding via the VM SSA evaluator; Extend(Imm) blocks the immediate-form encoders #347).
  3. passes/byteload.rs merge (rotate.rs-style in-place rewrite): linearize an Or root into (disp,shift) byte terms (OR-tree and Horner forms), require same base value, complete distinct shifts, consecutive disps, single-use intermediates, non-volatile, no intervening Store/Call/Mcpy (store_forward hazard discipline); disp-vs-shift order selects plain wide Load (LE) or wide Load + Bswap (BE); dual store side (runs of Store{I8} of Shr(src,8k), store ids use_count==0). Unaligned wide scalar access is legal on both targets.

ORDER CONSTRAINT (load-bearing, measured): the merge must run BEFORE inline — the collapsed dl64/ts64 become single-block and pass the existing Phi-free inline gate; the simulation's end state (1 bl left in hashblocks) depends on it. Pipeline: mem2reg -> unroll(bytes) -> byteload -> inline.

Fixtures: dl64/ts64/ld32/st32 shapes, the sqlite3Get4byte shape, a volatile-byte negative, an intervening-store negative.

Metadata

Metadata

Assignees

No one assigned

    Labels

    leverage:mediumSubsystem- or ABI-scoped fixperformanceEmitted-code performance

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions