feat(7z): coder-graph depth — multi-folder, BCJ/Delta, Deflate/BZip2/Zstd, AES-256 [RM-303]#71
Merged
Merged
Conversation
Replace the single-folder 7z seek-reader model with a Vec<Folder>: - parse_streams_info now yields one FolderInfo per folder. Drops the num_pack != 1 and NumFolders == 1 refusals; a single coder still means one pack stream per folder, so pack streams and folders pair 1:1 and each folder gets its own absolute pack offset from the shared base. - Generalize SubStreamsInfo across folders: per-folder substream sizes (single-substream folders omit their stored size) and the CRC digest count rule numDigests = Σ (n==1 && folderCrc ? 0 : n). Folder CRC presence is tracked per folder via read_digests_defined. - FileRec carries (folder_index, offset_in_folder); content files draw down a flat folder-then-file substream list, resetting the per-folder offset on each folder change. - SevenZSeekReader keeps exactly one folder decoder live at a time. It is built lazily on first use of each folder and torn down (reclaiming the raw source, freeing the codec workspace) before seeking to and building the next — bounded memory regardless of folder count. Each folder is finalized (exact size, no trailing bytes) before switching and at end of stream. - Encoded next-header decoding keeps its single-folder restriction. Still LZMA/LZMA2, single coder per folder; the writer is unchanged (one solid LZMA2 folder). #![forbid(unsafe_code)] holds; no new deps. Differential tests: non-solid archives (each file its own folder) from sevenz-rust2 via push_archive_entry, for both LZMA2 and plain LZMA, with interleaved directory/empty-file entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Landing step 2 of the 7z coder-graph work. Pure infra: no behavior change for existing archives (differential/interop tests unchanged). Parser: - Replace the single-coder FolderCoder model with a spec-complete coder graph (Coder/BindPair/Folder). read_folder now parses any number of coders, bind pairs, and packed-stream indices; read_coder handles complex (multi-in/out) coders and coders without attributes; classify_method keeps only LZMA/LZMA2 decodable, every other id lists as Unsupported. - parse_streams_info reads one CodersUnpackSize per output stream across all folders, assigns each folder its own back-to-back pack streams (multi-pack aware), and derives each folder's final-output (unpack) size. - resolve_folder computes a topological decode order via post-order DFS with in-progress cycle detection, and rejects stream overlaps (an input fed by more than one source or none, an output bound twice) as typed Malformed. Decode: - SevenStage: a recursive, statically dispatched Read chain (Source pack stream wrapped by each coder's reader). build_folder_reader folds decode_order into the chain; only linear LZMA/LZMA2 folders over one pack stream are built, so BCJ2/PPMd/multi-in-out list but report Unsupported at build. Still exactly one decode chain live at a time (bounded memory). Codec infra: - Generalize PipelineRead into codec_read::CodecReader<Inner: Read, C: Codec>; filtered_io reuses it for zstd/LZ4 via a thin PipelineRead alias, and PipelineCodec now implements Codec. Verified: build -p libarchive_oxide with sevenz, "sevenz aes", --no-default-features --features sevenz, and default (zstd/lz4); xtask codec-policy (portable stays C/FFI-free) and no-dyn (static dispatch) pass; clippy clean; sevenz_differential, interop_foundation, and fuzz_replay green. #![forbid(unsafe_code)] holds. Cross/qemu/macOS matrices deferred to CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Landing step 3 of the 7z coder-graph work: decode-only delta and BCJ
byte filters, wired into the folder decode chain so a linear graph that
chains them with LZMA2/LZMA over one pack stream now extracts (previously
listed as Unsupported). Exactly one folder decoder stays live; bounded
memory and static dispatch are unchanged.
Filters (new filter/delta.rs, filter/bcj.rs, under the sevenz feature):
- DeltaDecoder: per-byte reconstruction, distance 1..=256 over a 256-byte
history ring. Fully streaming (no held state), so any chunking matches.
- BcjDecoder: BranchKind covering x86 (stateful E8/E9 with running mask and
IP), the 4-byte-stride RISC families (ARM/ARM64/PPC/SPARC), 2-byte ARM
Thumb, 16-byte IA-64, and variable-width RISC-V. Instructions straddling a
chunk boundary are held back and re-scanned; at end of input the remaining
tail is emitted untransformed, exactly as the encoder left it. Decode is a
bit-for-bit inverse of the reference encoders.
- Both implement the sans-I/O Codec and are driven by the shared CodecReader,
so dispatch stays static. codec_read is now compiled under sevenz too, and
its struct bounds moved onto the impls so SevenStage can nest it.
Parser/decode (sevenz.rs):
- classify_method recognizes the delta id (0x03, distance = prop + 1) and the
BCJ ids (x86/PPC/IA64/ARM/ARMT/SPARC long-form, ARM64 0x0A, RISC-V 0x0B),
mapping them to CoderMethod::Filter(SevenFilter::{Delta,Bcj}). Unknown
property shapes still list as Unsupported.
- SevenStage gains Delta/Bcj stages wrapping CodecReader over the stage below;
wrap_coder builds them. A filter coder is single-in/single-out, so a folder
chaining it with LZMA2/LZMA stays a linear decodable chain.
Tests:
- Differential vs sevenz-rust2: delta+LZMA2 (distances 1/4/256) and BCJ+LZMA2
(x86/ARM/PPC/SPARC) archives are produced by sevenz-rust2 and decoded by
arca to byte equality, cross-checked against sevenz-rust2's own reader.
- Filter unit tests decode every branch kind and every delta distance against
lzma_rust2's independent encoder/decoder, across tiny (1/3-byte) and bulk
output buffers and small/large input chunks, plus near-window-boundary
lengths — proving chunk-size independence.
Verified: build -p libarchive_oxide with sevenz, "sevenz aes", and
--no-default-features --features sevenz, plus default; sevenz_differential
and filter unit tests green on each; clippy clean; xtask no-dyn and
codec-policy pass (no new deps; portable stays C/FFI-free). #![forbid(
unsafe_code)] holds. Cross/qemu/macOS matrices deferred to CI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Step 4 of the 7z coder-graph landing: decode the general-purpose 7z content coders by reusing arca's existing codecs through the shared PipelineCodec dispatch, wrapped as a new SevenStage::Pipeline chain stage (no second parser; ADR-0004 preserved). One active folder decoder, static dispatch, and #![forbid(unsafe_code)] all hold — the codecs live in existing deps. - core: add FilterId::Deflate (raw RFC 1951, no framing, no probe signature). - gzip: add RawInflateDecoder, a bare raw-DEFLATE Codec over the same miniz_oxide raw-inflate core GzipDecoder sits on, with a decoded_total guard. - pipeline_codec: add the FilterId::Deflate arm (sevenz-gated). - sevenz: classify methods 04 01 08 (Deflate), 04 02 02 (BZip2), 04 F7 11 01 (Zstd) into SevenFilter::Pipeline; BZip2/Zstd list-only when their features are off. wrap_coder builds the pipeline stage and bounds BZip2/Zstd working sets against codec_memory (validate_pipeline_memory), mirroring the LZMA dictionary check; the native zstd backend additionally caps the window. - tests: differential decode of sevenz-rust2 Deflate/BZip2/Zstd folders, the latter two gated on arca's bzip2/zstd features; dev-dep sevenz-rust2 gains the deflate + zstd features to produce those folders. Verified: cargo build -p libarchive_oxide --features sevenz, "sevenz aes", and --no-default-features --features sevenz; cargo build -p xtask; xtask codec-policy (portable stays C/FFI-free) and no-dyn; cargo test --features sevenz (and "sevenz aes") --test sevenz_differential (11 passed); clippy clean on touched files. Cross/qemu and macOS matrices deferred to CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the 7z AES-256/SHA-256 encryption coder (method 06F10701) to the single seek parser's coder graph, gated on the `aes` feature. * New `filter/aes7z.rs`: parses the coder properties (numCyclesPower/salt/IV; 0x3F external-key marker and out-of-range work factors list but do not decode), derives the key with 7-Zip's own KDF (a single SHA-256 context iterating salt || password || counter 2^ncp times, password encoded as UTF-16LE — the ZIP AE-2 path instead hashes raw bytes via PBKDF2-HMAC-SHA1), and decrypts with cbc::Decryptor<Aes256> and the stored IV. Trailing ciphertext padding is truncated against the coder's declared output size rather than erroring. * Thread Option<SecretBytes> from SeekArchiveReader through SevenZSeekReader::new and the seek dispatch site (previously dropped) down to the coder build. A missing password on an AES folder is a typed Unsupported error; secrets are never logged. * Retain per-substream CRC-32 (previously discarded in the digest reader) and verify it after decode; a mismatch is ErrorKind::Integrity, reported as "wrong password or corrupt data" on an encrypted folder. * Add `cbc` to the `aes` feature. Differential coverage vs sevenz-rust2 AES256SHA256 + Password: correct password decodes the encrypted header and content to plaintext; a wrong password trips Integrity; a missing password is Unsupported. Static dispatch, bounded memory, and #![forbid(unsafe_code)] are preserved; portable-codecs stays C/FFI-free. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close RM-303 with the accountability + fuzz layer over the 7z coder graph: - Support matrix: rewrite the 7z row to list the full coder-graph method set (LZMA/LZMA2, Delta, BCJ family, Deflate, BZip2, Zstandard; multi-folder and general graphs), AES-256 in its own encryption column, and PPMd/BCJ2 as structured Unsupported. Add PPMd and BCJ2 deficit rows to the deficit table. - ADR-0012: add tracked-deficit rows for PPMd (new decoder deferred) and BCJ2 (multi-stream decode deferred) as typed Unsupported with declared resolution paths; refresh the "entire ledger" note. - fuzz: add the structured read_7z_graph target. It synthesizes a random 7z StreamsInfo (coder count, complex flags, arities/props, bind pairs, packed indices, coder/substream sizes) from `arbitrary`, frames it under correct start/next-header CRCs so the graph parser is always reached, and asserts the RM-303 invariant: no panic, bounded work, and only typed Malformed/Unsupported/ Integrity/Limit errors — stressing cycles, stream overlaps, and truncation (recomputed under the header CRC). Body lives in the portable fuzz_lib and is wired into TARGETS/run_target, the libFuzzer shim, and fuzz/Cargo.toml. - Seed fuzz/corpus/read_7z_graph with six differential-scenario archives (arca solid, sevenz-rust2 multi-folder, delta+LZMA2, BCJ+LZMA2, LZMA, Deflate). Verified: cargo build -p libarchive_oxide --features sevenz / "sevenz aes" / --no-default-features --features sevenz; cargo build -p xtask; codec-policy (portable stays C/FFI-free); sevenz_differential (14/14, incl. aes); fuzz_replay (3/3 — 76960 mutants + arbitrary batch across 18 targets uphold the invariant). Building the cargo-fuzz bin needs nightly + sanitizer and is deferred to CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…base [RM-303] The DEV-124 async poll_process dispatch (merged as #70) matches PipelineCodec variants explicitly; the sevenz Deflate variant added on this branch left that match non-exhaustive once rebased. Add the sevenz-gated Deflate arm, delegating to the default poll_process (RawInflateDecoder is an in-process codec). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
P4suta
enabled auto-merge (squash)
July 23, 2026 20:33
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
P4suta
disabled auto-merge
July 23, 2026 20:37
Two hardenings for the security-sensitive 7z AES path (CodeQL flagged the area): - Enable the `zeroize` feature on the `aes` crate so the expanded AES-256 key schedule held inside the CBC decryptor is wiped on drop. Previously the derived key was zeroized but the cipher's round keys lingered in freed memory for both the 7z CBC and ZIP AE-2 CTR paths. - Rewrite `password_utf16le` over `slice::utf8_chunks` instead of `String::from_utf8_lossy`, so a password with invalid UTF-8 is never copied into an intermediate owned `String` that escapes zeroization; `chunk.valid()` borrows from the caller's buffer. Behaviour is byte-identical (one U+FFFD per maximal invalid subsequence), locked by a new equivalence test against the lossy path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
P4suta
enabled auto-merge (squash)
July 23, 2026 21:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deepens 7z from a single folder/coder to general coder graphs. Linear: DEV-109 (RM-303), parent RM-300. Honors ADR-0004 (only the single
SevenZSeekReaderparser is touched; no second parser), keeps#![forbid(unsafe_code)], static dispatch, and bounded memory (exactly one folder decoder live).What lands (squashed from 7 commits)
Vec<FolderInfo>, lazy per-folder decoder torn down before the next, file→folder mapping, generalized SubStreams/CRC rules.Coder/BindPair/Folder;resolve_foldercomputes decode order via post-order DFS with in-progress cycle detection and rejects stream overlaps as typedMalformed; recursive statically-dispatchedSevenStagedecode chain; newcodec_read.rs::CodecReader<Inner: Read, C: Codec>generalized fromPipelineRead.filter/delta.rs,filter/bcj.rs) as sans-I/OCodecs — x86 (stateful E8/E9 + running mask/IP) and RISC/ARMT/IA-64/RISC-V, with instruction-window buffering across chunk boundaries.SevenFilter::Pipeline; adds a raw-DEFLATEFilterId::Deflatearm (RawInflateDecoderover the sameminiz_oxideraw-inflate core).aesfeature,filter/aes7z.rs) — coder06F10701prop parse, 7-Zip KDF (SHA-256 × 2^ncp oversalt || password_utf16le || counter),cbc::Decryptor<Aes256>with stored IV; password threaded throughSeekArchiveReader; missing password → typedUnsupported; per-substream CRC verified so a wrong password tripsErrorKind::Integrity. Addscbconly under theaesfeature (portable stays C/FFI-free).Unsupported, still listable); structuredread_7z_graphfuzz target over random StreamsInfo.The final commit covers the
Deflatevariant in DEV-124's asyncpoll_processdispatch (rebase follow-up after #70).Tests
sevenz_differential= 14 passed (multi-folder ×2, delta, bcj, deflate, bzip2, zstd, AES correct/wrong-password/missing-password) — all differential against independent producersevenz-rust20.21.3.fuzz_replayupholds typedMalformed/Unsupported/Integrity/Limitonly (no panic, bounded). Feature gates (sevenz,sevenz aes,--no-default-features sevenz, default),codec-policy, andno-dynall green locally.🤖 Generated with Claude Code