Skip to content

numcodecs.LZ4 not compatible with lz4.fram and cramjam.lz4 #816

@Rachit931

Description

@Rachit931

While comparing LZ4 implementations, it was found that lz4.frame and cramjam.lz4 are compatible with each other, but numcodecs.LZ4 is not compatible with them.

From the tests :

  1. Data compressed with lz4.frame can be decompressed by cramjam.lz4 and vice versa.
  2. Data compressed with lz4.frame or cramjam.lz4 cannot be decompressed using numcodecs.LZ4.
  3. Data compressed with numcodecs.LZ4 cannot be decompressed using lz4.frame or cramjam.lz4.
import cramjam
import numcodecs
import os
import lz4.frame

initial_data = os.urandom(128)

compressed = {}

compressed["lz4"] = lz4.frame.compress(initial_data)
compressed["cramjam"] = bytes(cramjam.lz4.compress(initial_data))
compressed["numcodecs"] = numcodecs.LZ4().encode(initial_data)

libraries = ["lz4", "cramjam", "numcodecs"]

for compressor in libraries:
    for decompressor in libraries:
        print(f"Decompressing data compressed by {compressor} using {decompressor}... ", end="")
        
        try:
            if decompressor == "lz4":
                decompressed = lz4.frame.decompress(compressed[compressor])
            elif decompressor == "cramjam":
                decompressed = bytes(cramjam.lz4.decompress(compressed[compressor]))
            elif decompressor == "numcodecs":
                decompressed = numcodecs.LZ4().decode(compressed[compressor])
            assert initial_data == decompressed, f"Decompression failed for {decompressor} with data compressed by {compressor}"
            print("Success!")
        except Exception as e:
            print(f"Failed: {e}")

Outputs :

Decompressing data compressed by lz4 using cramjam... Success!
Decompressing data compressed by lz4 using numcodecs... Failed: LZ4 decompression error: -13
Decompressing data compressed by cramjam using lz4... Success!
Decompressing data compressed by cramjam using cramjam... Success!
Decompressing data compressed by cramjam using numcodecs... Failed: LZ4 decompression error: -9
Decompressing data compressed by numcodecs using lz4... Failed: LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using cramjam... Failed: LZ4 error: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using numcodecs... Success!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions