IIBalance

Overview

This repository provides the ICME 2026 code implementation of "Beyond Forced Modality Balance: Intrinsic Information Budgets for Multimodal Learning".

Getting started

Create environment:

conda env create -f environment.yml
conda activate iibalance-cremad
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu118

Prepare cache:

python -m src.iibalance.data.prepare_cremad \
  --raw-root data/raw/test_videos \
  --cache-root data/cache/test_videos \
  --split-seed 3407

Run training and evaluation:

python scripts/train_unimodal.py \
  --manifest data/cache/test_videos/manifests/cremad_manifest.csv \
  --modality audio \
  --output-dir artifacts/unimodal_audio

python scripts/train_unimodal.py \
  --manifest data/cache/test_videos/manifests/cremad_manifest.csv \
  --modality video \
  --output-dir artifacts/unimodal_video

python scripts/estimate_iib.py \
  --manifest data/cache/test_videos/manifests/cremad_manifest.csv \
  --audio-checkpoint artifacts/unimodal_audio/best_audio.pt \
  --video-checkpoint artifacts/unimodal_video/best_video.pt \
  --output-path artifacts/iib_prior.json

python scripts/train_iibalance.py \
  --manifest data/cache/test_videos/manifests/cremad_manifest.csv \
  --iib-prior artifacts/iib_prior.json \
  --audio-checkpoint artifacts/unimodal_audio/best_audio.pt \
  --video-checkpoint artifacts/unimodal_video/best_video.pt \
  --output-dir artifacts/iibalance

python scripts/eval.py \
  --manifest data/cache/test_videos/manifests/cremad_manifest.csv \
  --checkpoint artifacts/iibalance/best_iibalance.pt \
  --mode av

Cite our work

If you use this repository, please cite our ICME 2026 paper. BibTeX will be added after publication.