Flexible Multitask Learning with Factorized Diffusion Policy

Paper | Webpage

Chaoqi Liu¹, Haonan Chen¹, Sigmund H. Høeg², Shaoxiong Yao¹, Yunzhu Li³, Kris Hauser¹, Yilun Du⁴

¹University of Illinois Urbana-Champaign   ²Norwegian University of Science and Technology   ³Columbia University   ⁴Harvard University

IEEE Robotics and Automation Letters (RA-L), 2026

---

Installation

# Clone the repository
git clone https://github.com/Chaoqi-LIU/fdp.git
cd fdp

# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Sync dependencies and install the package
uv sync
uv pip install -e .

Repository Structure

fdp/
├── fdp/
│   ├── common/                  # Shared utilities (checkpointing, logging, replay buffer, etc.)
│   ├── config/                  # Hydra config files
│   │   └── train_factorpolicy.yaml
│   ├── dataset/                 # Dataset loading utilities
│   ├── env/                     # Environment implementations
│   │   └── rlbench/             # RLBench environment wrapper
│   ├── env_runner/              # Environment runners for evaluation
│   ├── gymnasium_util/          # Gymnasium wrappers (async/sync vector envs, video recording)
│   ├── model/
│   │   ├── common/              # Normalizers, LR schedulers, etc.
│   │   └── diffusion/           # Transformer and UNet backbone implementations
│   ├── perception/              # Observation encoders (vision + state)
│   ├── policy/
│   │   └── factorpolicy.py      # FactorizedDiffusionTransformerPolicy (main model)
│   └── workspace/
│       └── train_policy.py      # Training entrypoint

Citation

@ARTICLE{liu2026factorizeddiffusionpolicy,
    author={Liu, Chaoqi and Chen, Haonan and Høeg, Sigmund H. and Yao, Shaoxiong and Li, Yunzhu and Hauser, Kris and Du, Yilun},
    journal={IEEE Robotics and Automation Letters},
    title={Flexible Multitask Learning With Factorized Diffusion Policy},
    year={2026},
    volume={11},
    number={4},
    pages={4697-4704},
    doi={10.1109/LRA.2026.3664611}
}