EmbdC: Lossy Compression for Visual Embeddings in Video Large Language Models

Overview

EmbdC is a lossy compression framework for visual embeddings generated by video large language models (Video LLMs). The compression pipeline consists of:

1. Mean subtraction - Remove per-sample mean (lossless)
2. Quantization - Round to INT8 with configurable error bound (lossy)
3. Huffman encoding - GPU-accelerated entropy coding (lossless)
4. Zstandard compression - Fast dictionary-based compression (lossless)

Supported models: Qwen2-VL (2B/7B), Qwen2.5-VL (3B/7B), VideoLLaMA3 (2B/7B), Video-LLaVA (7B)

Installation

git clone https://github.com/BoJiang03/EmbdC.git
cd EmbdC

# Install dependencies
pip install -r requirements.txt

# Install flash-attn (pre-built wheel for CUDA 12 + PyTorch 2.7)
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl

# Build CUDA extension (GPU Huffman codec)
cd embedding_cuda_ext
python setup.py develop
cd ..

# Install modified transformers and lmms-eval (included in this repo)
cd transformers && pip install -e . && cd ..
cd lmms-eval && pip install -e . && cd ..

Quick Start

Simple compression (standalone, no GPU required)

# Compress an embedding binary file
python scripts/sr1_simple_embdc.py <input_file> <abs_error_bound> <shape>

# Example
python scripts/sr1_simple_embdc.py embedding_fp32_16_20_36_3584.bin 0.75 [16,20,36,3584]

Generate embeddings from video

python scripts/sr0_gen_embedding.py

Project Structure

EmbdC/
├── plugin/                     # Core compression library
│   ├── compression.py          # Compression/decompression pipeline
│   ├── embedding_process_plugin.py  # Model integration hook
│   ├── lmms_eval_warp.py      # Evaluation wrapper
│   ├── error/                  # Error bound configuration
│   ├── models/videollama3/     # VideoLLaMA3 model adaptation
│   ├── utils/                  # Embedding extraction, evaluation, etc.
│   └── sz_setting/             # SZ baseline configuration
├── embedding_cuda_ext/         # CUDA extension for GPU Huffman codec
├── huffman-cuda-standalone/    # GPU Huffman library
├── scripts/
│   ├── s0-s10, s_utils.py     # Main experiment scripts
│   ├── sr0, sr1               # Standalone demo scripts
│   ├── extra1-3               # Extended experiments (throughput, bandwidth)
│   ├── profile_*.py           # Profiling utilities
│   └── figure/                # Paper figure generation
├── transformers/               # Modified HuggingFace Transformers
└── lmms-eval/                  # Modified LMMS-Eval framework

Experiments

Accuracy vs. compression ratio (Table III-VII in paper)

python scripts/s7_model_abs_error_accuracies.py

GPU compression throughput (Table VIII)

python scripts/extra2_compressor_throughput.py

End-to-end bandwidth case study (Table IX)

python scripts/extra3_real_bandwidth_case_study.py

Generate paper figures

cd scripts/figure
python e3_quant_scale_accuracy.py
python e4_sparsity_cr.py
python e6_sparsity_accuracy_cr.py
# ... etc.

Benchmarks

Evaluated on:

• MVBench (episodic_reasoning, object_existence, scene_transition)
• VideoMME (with subtitles)
• PerceptionTest

License

This project is licensed under the MIT License - see the LICENSE file for details.