Zstd.compress currently emits a single zstd block per frame, so its input is bounded by zstd's Block_Maximum_Size (128 KiB). Larger inputs are rejected up front with a ZstdException (see the guard in PureZstdEncoder.encode) — because a single block exceeding 128 KiB produces a frame that neither libzstd nor kzstd's own decoder can read (verified empirically: >128 KiB failed both directions before the guard was added).
Goal
Chunk the input into ≤128 KiB blocks and emit a multi-block frame (with Last_Block set only on the final block), lifting the cap while staying libzstd-conformant. The decoder already handles multi-block frames; only the encoder's block-emission loop needs to change, and the >128 KiB guard can then be removed.
Notes
- A simple version (independent per-chunk blocks, explicit offsets — the encoder never uses repeat-offset codes) is straightforward and correct but loses cross-chunk match ratio. A windowed version that lets matches span chunk boundaries is the better long-term form.
- Add large-input tests: kzstd round-trip and libzstd-decodes-kzstd for multi-MB inputs.
- Priority follow-up for a general-purpose codec (target 0.2.0).
Zstd.compresscurrently emits a single zstd block per frame, so its input is bounded by zstd'sBlock_Maximum_Size(128 KiB). Larger inputs are rejected up front with aZstdException(see the guard inPureZstdEncoder.encode) — because a single block exceeding 128 KiB produces a frame that neither libzstd nor kzstd's own decoder can read (verified empirically: >128 KiB failed both directions before the guard was added).Goal
Chunk the input into ≤128 KiB blocks and emit a multi-block frame (with
Last_Blockset only on the final block), lifting the cap while staying libzstd-conformant. The decoder already handles multi-block frames; only the encoder's block-emission loop needs to change, and the >128 KiB guard can then be removed.Notes