This repo contains a series of components for Unity XR teleoperation with ROS integration. It is designed to be run on a Quest 3 VR headset, and support OpenXR, hand tracking, Unity input system, and is built with Unity 6000.2.15f1. _Newer versions of Unity no longer work on ROS 1 versions of Ubuntu, so this will be the latest offically supported version of Unity for ROS 1.
For information on setting up Unity and opening this project Unity Quickstart, and for Quest information see Quest Quickstart. To install the git-hooks, and link this project to a Linux Unity Hub, run the setup.sh script. To side load apps to the Quest see SideQuest Quickstart, and for streaming the app to a linux device see our streaming script.
On the ROS side, the custom TCP Endpoint needs to be run somewhere on the ROS network (For ROS 2 systems, make sure to switch to the main-ros2 branch, although the current Unity app supports both versions of ROS out of the box). This node is pretty lightweight so it can be run directly on a robot. Once the node is running the device running this app needs to connect to the ROS network, and the IP of the device running the TCP Endpoint needs to be set in the menu (see Menu for more information). If everything works, the menu should turn green, and data should be streaming between the app and ROS. For custom robots, check out the documentation on adding new robot models. If you are interested in contributing to the project, check out the contributing guidelines.
| Component | Description | Location | Preview |
|---|---|---|---|
| Audio Streamer | Implements bidirectional audio stream over ROS | Assets/RSL/Sensors/Audio |  |
| Camera Viewer | Renders a ROS image stream to a floating image window | Assets/RSL/Sensors/Camera |  |
| Grid Map | Renders a 2.5D grid map such as elevation maps from Anymal | Assets/RSL/Sensors/GridMap |  |
| Hands | Hand tracking and pose publishing over ROS, compatible with Ability hand models | Assets/RSL/Telemetry/Hands |  |
| Haptic | Bhaptic glove support plus controller haptics | Assets/RSL/Core/Haptics |  |
| Headset Publisher | Publishes headset and hand poses on TF and Pose | Assets/RSL/Telemetry/Headset |  |
| Lidar | GPU rendering for LiDAR and PointCloud2 point viz from ROS | Assets/RSL/Sensors/Lidar |  |
| Markers | Displays various ROS markers in the scene (supports points, cubes, spheres) | Assets/RSL/Sensors/Markers |  |
| Menu | Palm menu for interaction and toggling | Assets/RSL/Core/Menu |  |
| NeRFViewer | Handheld viewer for rendering NeRFs and scene interaction | Assets/RSL/Sensors/NeRF |  |
| PathStreamer | Displays nav paths with a line and series of arrows | Assets/RSL/Sensors/Path |  |
| PosePublisher | Publishes poses and Joystick commands for locomotion | Assets/RSL/Telemetry/Pose |  |
| PoseStreamer | Displays stamped poses | Assets/RSL/Sensors/Pose |  |
| Robots | Robot model manager, holds the meshes, materials and the scripts to swap between Anymal, ALMA, Tytan, Standalone Dynaarm and Franka Panda | Assets/RSL/Core/Robots |  |
| Service Caller | Creates a floating button that triggers an Empty service call when pressed | Assets/RSL/Sensors/Service |  |
| Stereo | Stereo camera rendering, renders to each eye for human depth perception | Assets/RSL/Sensors/Camera |  |
| TF | Updates for the Unity-ROS TF system for managing robots and reorientation including publishing headset and hands over TF | Assets/RSL/Core/TF | |
| TF Viz | Debug viewer to show currently streamed TF links | Assets/RSL/Core/TF |  |
| Voxblox | Voxel mesh rendering | Assets/RSL/Core/VoxBlox |  |
| VR Debug | Debugging tools for VR, namely a console | Assets/RSL/Core/VRDebug |  |
| VR Streamer | Streams the VR view to a ROS topic (w/o AR view) | Assets/RSL/Telemetry/VRStreamer |  |
- Bumped to Unity 6.2 for improved OpenXR and Quest 3 support (note this is the last version of Unity that will support ROS 1 on Ubuntu 20.04, if you are looking for a pre Unity 6 build check out v0.1.1)
- Now supports ROS 1 and ROS 2 in the same build allowing dynamic switching with a new version of the ROS-TCP-Endpoint
- Added Gaussian Splatting as an option for PointCloud messages
- New Robot manager that moves robot models to external repos (see unity_ros_robots for more info)
- Added support for spatial anchors ensuring persistent localization and mapping across sessions
- Improved package naming within the app for cross package imports
- Various bug fixes and improvements
- Added support for 360 stereo image streams
- Added setup scripts and git hooks for easier project setup
- Fixed some image rendering issues with Image Messages
- Added passthrough toggle
- Improved TF modes for Lidar
- Added State buttons for toggles
- Fixes for ROS 2
- Updated docs for setup and usage with streaming
- Added Robots
- B2W
- GR2
May need to reset the repo as LFS has been removed from history
- Added 2D version of the app
- Refactored new Manager-Streamer system for standardized message visualizations
- Added new visualizations
- Markers (without meshes and text)
- Paths
- Stamped Poses
- TF Visualization for frames
- Grid Maps/Elevation Maps
- Added floating button to trigger Empty service calls
- Redid TF system so the root frame can be dynamic and not locked to odom (also allows functioning without a robot model)
- Improved Pose publishing and allows to work with 2D interface
- Added debug menus to display force
- Overhauled PointCloud rendering for better performance and shader keywords for customized coloring
- Added mesh shaders for efficently render large numbers of procedural meshes
- Improved layout serialization allowing for saving of custom configurations
- Simplified dynaarm model (down to 98 draw calls and 330k tris)
- Fixed TF pose publishing so it should work even without a model
- Removed some duplicated publishers/gameobjects
- Added versioning with display in debug mode
- Added tytan, standalone dynaarm, and franka panda robots
- Updated dynaarm for newer model (may still need simplification)
- Added robot manager for switching between robots
If you use this project in your work please cite this paper:
@INPROCEEDINGS{wildersmith2024rfteleoperation,
author={Wilder-Smith, Maximum and Patil, Vaishakh and Hutter, Marco},
booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title={Radiance Fields for Robotic Teleoperation},
year={2024},
pages={13861-13868},
doi={10.1109/IROS58592.2024.10801345}
}