-
Notifications
You must be signed in to change notification settings - Fork 110
Open
Description
While comparing LZ4 implementations, it was found that lz4.frame and cramjam.lz4 are compatible with each other, but numcodecs.LZ4 is not compatible with them.
From the tests :
- Data compressed with lz4.frame can be decompressed by cramjam.lz4 and vice versa.
- Data compressed with lz4.frame or cramjam.lz4 cannot be decompressed using numcodecs.LZ4.
- Data compressed with numcodecs.LZ4 cannot be decompressed using lz4.frame or cramjam.lz4.
import cramjam
import numcodecs
import os
import lz4.frame
initial_data = os.urandom(128)
compressed = {}
compressed["lz4"] = lz4.frame.compress(initial_data)
compressed["cramjam"] = bytes(cramjam.lz4.compress(initial_data))
compressed["numcodecs"] = numcodecs.LZ4().encode(initial_data)
libraries = ["lz4", "cramjam", "numcodecs"]
for compressor in libraries:
for decompressor in libraries:
print(f"Decompressing data compressed by {compressor} using {decompressor}... ", end="")
try:
if decompressor == "lz4":
decompressed = lz4.frame.decompress(compressed[compressor])
elif decompressor == "cramjam":
decompressed = bytes(cramjam.lz4.decompress(compressed[compressor]))
elif decompressor == "numcodecs":
decompressed = numcodecs.LZ4().decode(compressed[compressor])
assert initial_data == decompressed, f"Decompression failed for {decompressor} with data compressed by {compressor}"
print("Success!")
except Exception as e:
print(f"Failed: {e}")Outputs :
Decompressing data compressed by lz4 using cramjam... Success!
Decompressing data compressed by lz4 using numcodecs... Failed: LZ4 decompression error: -13
Decompressing data compressed by cramjam using lz4... Success!
Decompressing data compressed by cramjam using cramjam... Success!
Decompressing data compressed by cramjam using numcodecs... Failed: LZ4 decompression error: -9
Decompressing data compressed by numcodecs using lz4... Failed: LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using cramjam... Failed: LZ4 error: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using numcodecs... Success!
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels