Add ZL_bitExtract and ZL_bitDeposit helpers#519
Open
terrelln wants to merge 6 commits intofacebook:devfrom
Open
Add ZL_bitExtract and ZL_bitDeposit helpers#519terrelln wants to merge 6 commits intofacebook:devfrom
terrelln wants to merge 6 commits intofacebook:devfrom
Conversation
Summary: Improve the ability to catch OOB-access and use-after-free bugs in OpenZL's arenas by using memory poisoning. Differential Revision: D96393907
Summary: Refactor `processStream` and rename to `runDecoder`. As part of the refactor I: 1. Moved all validation that can be done at header decode time to `fillStoredStreams()` 2. Split the function up into smaller helpers 3. Removed a check that the regenerated streams aren't filled, because we already check this in `fillStoredStreams()` 4. Deleted the `transformInputStreams` vector and use the `workspaceArena` instead Differential Revision: D96396998
Summary:
This diff adds a new `decoder_fusion.[hc]` module to OpenZL without hooking it up the the decoding process, so in itself is a no-op.
The key APIs are:
1. `ZL_DecoderFusionDesc` describes the decoder fusion with two parts:
a. The `ZL_DecoderFusionPattern` describes the codec pattern to search for
b. The `ZL_DecoderFusionFn` implements the fusion given the `ZL_DecoderFusion* state`.
2. An empty static registry of built-in decoder fusions in `decoder_registry.c`.
3. `ZL_DecoderFusionState_registerFusion()` allows registering decoder fusions at runtime for unit testing.
4. `ZL_DecoderFusionState_maybeFuse() ` efficiently determines whether fusion can be applied given the parent codec during chunk header decoding in `fillStoredStreams()`.
Differential Revision: D96491175
Summary: This diff hooks up the decoder fusion module added in D96491175 to the decompression process. 1. In `fillStoredStreams()` we check for fusion opportunities 2. We add `runFusedDecoder()` which is called by `DCTX_runDecoder()` for fused codecs Differential Revision: D96491173
Summary: Add test cases for decoder fusion using the `DCTX_registerDecoderFusion()` API to register custom fusions. These test cases are 100% AI generated. I've validated what is tested is correct, and ensured we have the test coverage we need, but the implementation of the codec fusions is a bit of AI slop. Differential Revision: D96491172
Summary: Add portable implementations of PDEP and PEXT. Currently only supports x86-64 and fallback, but more platforms can be added as needed. Differential Revision: D96518564
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary: Add portable implementations of PDEP and PEXT. Currently only supports x86-64 and fallback, but more platforms can be added as needed.
Differential Revision: D96518564