Score Based Multimodal Autoencoders

This is the code for the paper Score Based Multimodal Autoencoders

Please refer to the official repo at https://github.com/rooshenasgroup/sbmae if you have questions.

Get started (Extended PolyMnist)

• Download the dataset and the FID model* !-- Download the preprocessed dataset from here Download the FID model from here -->

    pip install -r requirements.txt
    mv pt_inception-2015-12-05-6726825d.pth ./pytorch_fid/
    tar xvzf upd_poly.tar
    ./train_poly.sh

Get started (CelebMaskHQ)

• Download the dataset and the FID model* Download the preprocessed dataset from here Download the FID model from here

    pip install -r requirements.txt
    mv pt_inception-2015-12-05-6726825d.pth ./pytorch_fid/
    tar xvzf celm_hq.tar
    ./train_cel.sh

To train the EBM model, please look at train_cel_clwithtime_ebm_NOIND.py script

For sampling and other calclations, please look at the scripts.

If you find this work useful, please cite our paper:

@article{
wesego2024scorebased,
title={Score-Based Multimodal Autoencoder},
author={Daniel Wesego and Pedram Rooshenas},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2024},
url={https://openreview.net/forum?id=JbuP6UV3Fk},
note={}
}