Skip to content

Multi-block encoding: lift the 128 KiB single-block input limit #1

Description

@jamesarich

Zstd.compress currently emits a single zstd block per frame, so its input is bounded by zstd's Block_Maximum_Size (128 KiB). Larger inputs are rejected up front with a ZstdException (see the guard in PureZstdEncoder.encode) — because a single block exceeding 128 KiB produces a frame that neither libzstd nor kzstd's own decoder can read (verified empirically: >128 KiB failed both directions before the guard was added).

Goal

Chunk the input into ≤128 KiB blocks and emit a multi-block frame (with Last_Block set only on the final block), lifting the cap while staying libzstd-conformant. The decoder already handles multi-block frames; only the encoder's block-emission loop needs to change, and the >128 KiB guard can then be removed.

Notes

  • A simple version (independent per-chunk blocks, explicit offsets — the encoder never uses repeat-offset codes) is straightforward and correct but loses cross-chunk match ratio. A windowed version that lets matches span chunk boundaries is the better long-term form.
  • Add large-input tests: kzstd round-trip and libzstd-decodes-kzstd for multi-MB inputs.
  • Priority follow-up for a general-purpose codec (target 0.2.0).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions