Franka Imitation Learning Pipeline

Franka Research 3 imitation learning pipeline: teleoperation data collection (TactAR / Meta Quest 3), training (ACT / 3DFA policies), and real-world deployment. Built on neuromeka-il, adapted for Franka robots with Deoxys as the low-level controller.

Installation

# Full environment (train + deploy)
conda env create -f environment.yaml -n openteach

# Training only
conda env create -f environment_train.yaml -n openteach

# Install packages (editable)
conda activate openteach
pip install -e utils/              # davian_utils (shared utilities)
pip install -e train/davian_il/    # davian_il (ML policies)

Python 3.10, PyTorch 2.6, CUDA 12.4. Config management uses Hydra.

Important Notes

1. Disk usage (critical). During data collection, RGB and depth images are saved at every timestep, so train/data/ grows very quickly. After collecting and preprocessing the data, move it to your training server and delete it from this machine. A full disk will block further data collection and experiments.
2. This repository is a template. Do not edit it in place. Fork or clone it and work in a separate directory. Changes intended for the template should be discussed and merged through a pull request.
3. Teleoperation safety. To avoid collisions with walls or the table, set motion thresholds on the robot's reachable range from the start of data collection.

TODO

1. (Teleop) Collect datasets in LeRobot format. Note: depth is not part of the LeRobot dataset format, and video is stored as MPEG.
2. (Cleanup) Single-command entry points (e.g., daro.teleop, daro.deploy).
3. (Integration) Unify the currently separate data-collection, training, and deployment stages into a single interface, in the style of LeRobot.

Robot Startup and Shutdown

Power on and connect the desktop to the robot

1. Turn on the switch on the control box (black unit).
2. Open https://172.16.0.2/desk/ in a browser.
3. Go to Joints, unlock them, set EXEC mode, and click Activate FCI. The robot is now ready.

Shutdown (important)

1. Go to Joints, lock them, and Deactivate FCI.
2. From the top menu, select Shut down.
3. When the completion notice appears, turn off the control box switch.

Note: turning off the control box switch without first completing steps 1 and 2 causes errors that are troublesome to recover from.

How to run Pi

• /home/davian/byungkunlee/davian_robotics/deploy/middle_level_controller/franka_act_pi --> Refer to this folder

Documentation

Document	Contents
data_collect/README.md	Data collection via teleoperation (TactAR / Meta Quest 3)
train/README.md	Training configs, preprocessing, training commands, synthetic data examples
deploy/README.md	Policy deployment, third-party models, adding task controllers

Repository Structure

├── data_collect/                    # Teleoperation data collection
│   ├── collect_data.py              # Entry point
│   ├── collector.py                 # Collection scheduler + teleop collector
│   ├── config.py                    # Collection profiles (franka_1cam, etc.)
│   └── device/                      # Teleop device drivers (TactAR, SpaceMouse)
├── train/                           # Training pipeline
│   ├── imitate.py                   # Main training entry point (Hydra)
│   ├── preprocess.py                # HDF5 data preprocessing
│   ├── config/                      # Hydra YAML configs
│   └── davian_il/                   # Core ML library (pip install -e .)
│       └── src/davian_il/policies/  # ACT, 3DFA policy implementations
├── deploy/                          # Policy deployment on robot
│   ├── run_deploy.py                # Deployment entry point
│   ├── controllers/                 # Task controllers (see franka_act/ as reference)
│   │   └── franka_act/              # config, model, controller, run_nn_server
│   └── verify_train_deploy_consistency.py
├── calibration/                     # Camera extrinsics calibration (ChArUco hand-eye)
├── utils/                           # Shared utilities (pip install -e .)
│   └── src/davian_utils/
│       ├── robot/                   # FrankaRobot (Deoxys wrapper), gripper, robot_cluster
│       ├── camera/                  # RealSense camera interface
│       ├── hardware/                # Camera/robot config, franka_deoxys.yml
│       ├── config_utils.py          # Base dataclasses (ROBOT_CONFIG, TASK_CONFIG)
│       ├── controller_utils.py      # Base controller classes
│       └── remote_utils.py          # ZMQ server-client (PickleServer/PickleClient)
├── capture_realsense.py             # RealSense capture utility
├── environment.yaml                 # Full conda environment
└── environment_train.yaml           # Training-only conda environment

Calibration

# Camera extrinsics calibration (ChArUco hand-eye)
# See calibration/ for details

# Save ArUco marker reference (right after calibration)
python calibration/verify_calibration.py --save --cameras left_back

# Check camera drift against saved reference
python calibration/verify_calibration.py --cameras left_back

# Live preview with marker detection
python calibration/verify_calibration.py --live --cameras left_back

# Single-camera drift check with visual adjustment guide
python calibration/check_drift.py --camera_serial 347622076599

If --cameras is omitted, every camera registered in CAMERA_SERIAL is opened, so specify only the cameras that are actually connected.

Utilities

# Capture RealSense frames (RGB + depth + point cloud)
python capture_realsense.py

# Validate training/deployment config consistency 
python deploy/verify_train_deploy_consistency.py

Acknowledgments

• neuromeka-il (Apache 2.0) — Original IL framework by Neuromeka AI team
• deoxys_control — Real-time Franka control by UT Austin RPL
• LeRobot (Apache 2.0) — ACT policy implementation

License

This project is licensed under the terms of the license agreement found in LICENSE.