CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Rendering

Zhengqing Wang¹ , Yuefan Wu¹ , Jiacheng Chen¹ , Fuyang Zhang¹ , Yasutaka Furukawa^1,2

¹ Simon Fraser University ² Wayve

(arXiv, Project page)

demo_with_audio.mp4

Table of Content

• Introduction
• Installation
• Data preparation
• Getting started
• Citation
• License

Introduction

This paper proposes a neural rendering approach that represents a scene as "compressed light-field tokens (CLiFTs)", retaining rich appearance and geometric information of a scene.

CLiFT enables compute-efficient rendering by compressed tokens, while being capable of changing the number of tokens to represent a scene or render a novel view with one trained network.
Concretely, given a set of images, 1) multi-view encoder tokenizes the images with the camera poses, 2) latent-space K-means selects a reduced set of rays as cluster centroids using the tokens, 3) the multi-view "condenser" compresses the information of all the tokens into the centroid tokens to construct CLiFTs.
At test time, given a target view and a compute budget (i.e., the number of CLiFTs), the system collects the specified number of nearby tokens and synthesizes a novel view using a compute-adaptive renderer.

Installation

Please refer to the installation guide to set up the environment.

Data preparation

Please refer to the data preparation guide to download and prepare the RealEstate10K and DL3DV datasets.

Getting started

Pretrained checkpoints for both datasets are available on Hugging Face.

Please follow the test guide for model inference, evaluation, and visualization.

Please follow the training guide for details about the training pipeline.

Citation

If you find CLiFT useful in your research or applications, please consider citing:

@inproceedings{Wang2025CLiFT,
  author    = {Wang, Zhengqing and Wu, Yuefan and Chen, Jiacheng and Zhang, Fuyang and Furukawa, Yasutaka},
  title     = {CLiFT: Compressive Light-Field Tokens for Compute Efficient and Adaptive Neural Rendering},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2025},
}

Our method is deeply inspired by LVSM and DepthSplat, and benefited from their open-source code. Please consider reading these papers if interested in relevant topics.

License

This project is licensed under GPL, see the license file for details.