Hi,
We ran into a compatibility problem with compressed content produced by the C implementation using a custom dictionary.
If the length of the raw content + the dictionary length is greater than the defined compression window, the Rust decompressor fails with an error (InvalidData) while the C library (tested via linux cli) can still decompress the same file.
I have created a sample project that uses the (installed) C brotli cli to produce such a file and attempts to decompress it with this library.
I've tested it on Linux, with version 5.0.0 and brotli cli version 1.1 as well as 1.2.
To my (very limited) understanding, this happens because the Rust implementation prepends the dictionary into the buffer. If the data exceeds the window size, the ring buffer wraps around and overwrites the dictionary. The C implementation appears to keep the dictionary in a separate buffer, allowing references to it even after the ring buffer has wrapped.
We can currently workaround this by forcing a larger window size to fit both in the compressor, but for content larger than the maximum 16MB window there is no workaround to still use a custom dictionary.
I am not sure whether this is a known limitation, a bug or if I am just doing something wrong here.
Hi,
We ran into a compatibility problem with compressed content produced by the C implementation using a custom dictionary.
If the length of the raw content + the dictionary length is greater than the defined compression window, the Rust decompressor fails with an error (InvalidData) while the C library (tested via linux cli) can still decompress the same file.
I have created a sample project that uses the (installed) C brotli cli to produce such a file and attempts to decompress it with this library.
I've tested it on Linux, with version 5.0.0 and brotli cli version 1.1 as well as 1.2.
To my (very limited) understanding, this happens because the Rust implementation prepends the dictionary into the buffer. If the data exceeds the window size, the ring buffer wraps around and overwrites the dictionary. The C implementation appears to keep the dictionary in a separate buffer, allowing references to it even after the ring buffer has wrapped.
We can currently workaround this by forcing a larger window size to fit both in the compressor, but for content larger than the maximum 16MB window there is no workaround to still use a custom dictionary.
I am not sure whether this is a known limitation, a bug or if I am just doing something wrong here.