PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics

---

Overview

PokeFlex is a comprehensive real-world dataset of volumetric deformable objects designed to advance robotics research and applications. It provides a unique combination of multimodal data and tools, offering several key contributions:

• Rich Multimodal Dataset: Includes 18 deformable objects with synchronized RGB-D data from eye-in-hand and eye-on-hand sensors, high-resolution 3D reconstructed textured meshes, and detailed robot data such as joint angles, end-effector poses, and force-torque measurements.
• Simulation-Ready Assets: Provides assets and resources for integrating the dataset into common physics simulators, enabling physics-based simulations.
• Benchmarks for Deformation Prediction: Features benchmarks for real-time deformation prediction from diverse modalities.

Examples

Action	Foam Dice	Plush Dice	3D Printed Bunny	Foam Cylinder	Plush Octopus
Poking
Dropping

Objects

---

Getting Started

1. Clone the repository

git clone https://github.com/pokeflex-dataset/reconstruction

2. Python Environment Setup

Follow these steps to set up the required environment:

Create and activate the conda environment:

conda create -n pokeflex python=3.11
conda activate pokeflex

Install PyTorch with CUDA support:

conda install pytorch=2.1.0 torchvision pytorch-cuda=12.1 -c pytorch -c nvidia

Install additional dependencies:

pip install -U 'git+https://github.com/facebookresearch/iopath'
conda install jupyter
pip install scikit-image matplotlib imageio plotly opencv-python
conda install -c fvcore -c conda-forge fvcore
pip install black usort flake8 flake8-bugbear flake8-comprehensions

Install PyTorch3D:

conda install pytorch3d -c pytorch3d

Install remaining dependencies:

pip install trimesh pyvista meshio siren-pytorch open3d pyyaml numpy==1.26.4 transformers

3. Get the data

The dataset can be found here.

Supplementary material can be downloaded here.

4. Preprocess the data for training

1. Configure the preprocessing settings in the config/preprocess.yml file.
2. Run the following command to preprocess the data:

python3 dataset/preprocess.py

5. Training

1. Set training parameters in the respective .yml config file according to the chosen modality.
2. Start training using the following command:

python3 main.py --modality <modality> 

6. Testing

1. Specify the paths to the processed dataset and pretrained weights in the configuration file config/testing.yml.
2. Run the evaluation with the following command. You can optionally include the --save_loss flag to save the evaluation metrics:

python3 test/evaluate.py --modality <modality> [--save_loss]

Citation

@article{obrist2024pokeflex,
      author    = {Obrist, Jan and Zamora, Miguel and Zheng, Hehui and Hinchet, Ronan and Ozdemir, Firat and Zarate, Juan and Katzschmann, Robert K. and Coros, Stelian},
      title     = {PokeFlex: A Real-World Dataset of Deformable Objects for Robotics},
      journal   = {Under review},
      year      = {2025}
      url       = {https://arxiv.org/pdf/2410.07688}
      }