BeyondMimic Motion Tracking Inference

[Website] [Arxiv] [Video]

This repository provides the inference pipeline for motion tracking policies in BeyondMimic. The pipeline is implemented in C++ using the ONNX CPU inference engine. Model parameters (joint order, impedance, etc.) are stored in ONNX metadata, and the reference motion is returned via the forward() function. See this script for details on exporting models.

This repo also serves as an example of how to implement a custom controller using the legged_control2 framework.

Installation

Dependencies

This software is built on the ROS 2 Jazzy, which needs to be installed first. Additionally, this code base depends on legged_control2.

Install legged_control2

Pre-built binaries for legged_control2 are available on ROS 2 Jazzy. We recommend first reading the full documentation.

Specifically, For this repo, follow the Debian Source installation. Additionally, install Unitree-specific packages:

# Add debian source
echo "deb [trusted=yes] https://github.com/qiayuanl/unitree_buildfarm/raw/noble-jazzy-amd64/ ./" | sudo tee /etc/apt/sources.list.d/qiayuanl_unitree_buildfarm.list
echo "yaml https://github.com/qiayuanl/unitree_buildfarm/raw/noble-jazzy-amd64/local.yaml jazzy" | sudo tee /etc/ros/rosdep/sources.list.d/1-qiayuanl_unitree_buildfarm.list
sudo apt-get update

# Install packages
sudo apt-get install ros-jazzy-unitree-description
sudo apt-get install ros-jazzy-unitree-systems

Build Package

After installing legged_control2, you can build this package. You’ll also need the unitree_bringup repo, which contains utilities not included in the pre-built binaries.

Create a ROS 2 workspace if you don't have one. Below we use ~/colcon_ws as an example.

mkdir -p ~/colcon_ws/src

Clone two repo into the src of workspace.

cd ~/colcon_ws/src
git clone https://github.com/qiayuanl/unitree_bringup.git
git clone https://github.com/HybridRobotics/motion_tracking_controller.git
cd ../

Install dependencies automatically:

rosdep install --from-paths src --ignore-src -r -y

Build the packages:

colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=RelwithDebInfo --packages-up-to unitree_bringup
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=RelwithDebInfo --packages-up-to motion_tracking_controller
    source install/setup.bash

Docker Installation

Alternatively, you can use Docker to run this repository without installing dependencies directly on your system.

Build the Docker Image

docker build -t motion_tracking_controller .

Run the Container

Important - X11 Display Forwarding:

Before running the container (especially for MuJoCo simulation), you need to allow X11 display forwarding on your host system:

xhost +local:docker

This command allows Docker containers to access your X11 display server. To revoke access later, you can use:

xhost -local:docker

Note: If you encounter display-related errors, make sure:

• Your DISPLAY environment variable is set (usually :0 or :0.0)
• X11 forwarding is enabled (xhost +local:docker)
• The /tmp/.X11-unix socket is accessible

For interactive use:

docker run -it --rm \
  --network host \
  -e DISPLAY=$DISPLAY \
  -v /tmp/.X11-unix:/tmp/.X11-unix:rw \
  motion_tracking_controller

For running with a local ONNX file:

docker run -it --rm \
  --network host \
  -e DISPLAY=$DISPLAY \
  -v /tmp/.X11-unix:/tmp/.X11-unix:rw \
  -v $(pwd)/your_policy.onnx:/workspace/data/policy.onnx:ro \
  motion_tracking_controller \
  bash -c "ros2 launch motion_tracking_controller mujoco.launch.py policy_path:=/workspace/data/policy.onnx"

For running with WandB (requires API key):

docker run -it --rm \
  --network host \
  -e DISPLAY=$DISPLAY \
  -e WANDB_API_KEY=your_wandb_api_key \
  -v /tmp/.X11-unix:/tmp/.X11-unix:rw \
  motion_tracking_controller \
  bash -c "ros2 launch motion_tracking_controller mujoco.launch.py wandb_path:=your_user/project/run_id"

Verify Installation

Inside the container, you can verify the installation:

# Check ROS 2 environment
echo $ROS_DISTRO

# List installed packages
ros2 pkg list | grep -E "(legged|unitree|motion_tracking)"

# Verify workspace compilation
ls -la /workspace/colcon_ws/install/

# Test launch file
ros2 launch motion_tracking_controller --help

Basic Usage

Sim-to-Sim

We provide a launch file for running the policy in MuJoCo simulation.

# Load policy from WandB
ros2 launch motion_tracking_controller mujoco.launch.py wandb_path:=<your_wandb_run_path>

# OR load policy from local ONNX file (should be absolute or start with `~`)
ros2 launch motion_tracking_controller mujoco.launch.py policy_path:=<your_onnx_file_path>

Real Experiments

⚠️ Disclaimer
Running these models on real robots is dangerous and entirely at your own risk.
They are provided for research only, and we accept no responsibility for any harm, damage, or malfunction.

1. Connect to the robot via ethernet cable.
2. Set the ethernet adapter to static IP: 192.168.123.11.
3. Use ifconfig to find the <network_interface>, (e.g.,eth0 or enp3s0).

# Load policy from WandB
ros2 launch motion_tracking_controller real.launch.py network_interface:=<network_interface> wandb_path:=<your_wandb_run_path>

# OR load policy from local ONNX file (should be absolute or start with `~`)
ros2 launch motion_tracking_controller real.launch.py network_interface:=<network_interface> policy_path:=<your_onnx_file_name>.onnx

The robot should enter standby controller in the beginning. Use the Unitree remote (joystick) to start and stop the policy:

• Standby controller (joint position control): L1 + A
• Motion tracking controller (the policy): R1 + A
• E-stop (damping): B

Code Structure

This section will be especially helpful if you decide to write your own legged_control2 controller. For a minimal starting point, check the legged_template_controller.

Below is an overview of the code structure for this repository:

• include or src

  • MotionTrackingController Manages observations (like an RL environment) and passes them to the policy.

  • MotionOnnxPolicy Wraps the neural network, runs inference, and extracts reference motion from the ONNX file.

  • MotionCommand Defines observation terms aligned with the training code.

• launch

  • Includes launch files like mujoco.launch.py and real.launch.py for simulation and real robot execution.

• config

  • Stores configuration files for standby controller and state estimation params.