Immersive Mixed Reality Training Concept for Mastering Surgical Knot-tying - Prototype Implementation
This repository contains a prototype implemtation of our mixed reality training concept designed to enhance medical students’ acquisition of surgical knot-tying skills. Utilizing a virtual reality headset with video passthrough functionality, the system provides adaptive visual instructions tailored to the user’s hand movements during the knot-tying process. This prototype is implemented in Unity3D Version 2022.3 LTS fpr the Varjo XR-3 headset. The prototype can be adapted to other headsets as well, but these systems need to be equipped with a Ultraleap Handtracking sensor unit.
- Git with Git LTS :
- Volumetric Video sequences downloaded and unzipped :
- Latest Unity 2022 LTS version
- Varjo XR-3 Headset
- Varjo Base installed and hand tracking enabled
- A thread for tying the knots
- Ideally, a knot tying bench, or a similar object which you can use to tye the thread around
- Best case: The Digital knot tying bench
- Clone this repository with Git
- Put the folder "PointClouds" from the unzipped volumetric video sequences into the "Assets/Streamingassets" folder of the Unity Project
- Open the project with Unity 2022 LTS
- Open the scene "Assets/Scenes/BMBF_KnotbAR_INTRO.unity"
- Click play! Make sure to have audio enabled, as the voiceover contains important instructions
M. Queisner, C. Remde, R. Luzsa and I. M. Sauer, "Immersive Mixed Reality Training Concept for Mastering Surgical Knot-tying," 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France, 2025, pp. 1010-1015, doi: 10.1109/VRW66409.2025.00204.
Licensed under the GNU General Public License Version 3