aria2robot

Teleoperating the WUJI dexterous hand with Aria Gen 2. Subscribes to the hands topic published by aria2mano's stream_pub, retargets via wuji-retargeting, and drives either a sim hand or real hardware. A web-based viser viewer renders the Aria glasses mesh, head-pose trajectory, wrist frames, raw Aria hand keypoints, the scaled-target overlay (what the optimizer sees post-rotation/scaling/pinch-blend), and the WUJI hand URDF — plus a live-tuning panel for every retargeting parameter.

Edge-friendly: no projectaria-client-sdk / projectaria_tools dependency on the subscriber side, so this repo runs on ARM Linux (e.g. robot onboard compute, Jetson) — where the Aria SDK is not supported (currently x86 Linux and ARM macOS only). aria2mano is intentionally not installed here; run it on a separate publisher host (any supported platform).

Installation

Installation tested on Ubuntu 22.04 LTS / 24.04 LTS with Python 3.12.

# 1) Clone and install dependencies
git clone --recursive git@github.com:KevinyWu/aria2robot.git && cd aria2robot
conda env create -f environment.yaml && conda activate aria2robot
pip install -e external/wuji-retargeting

# 2) Install aria2robot
pip install -e .

Quick Start

# 1) On glasses host (separate machine, x86 Linux or ARM MacOS — aria2mano installed there):
aria_gen2 streaming install-certs
python -m aria2mano.stream_pub --wifi --hands-only

# 2) Unplug glasses

# 3) On subscriber (here):
python -m aria2robot.stream_sub                       [--pub-host "<publisher-ip>"]   # both hands, viewer-only
python -m aria2robot.stream_sub --real --hand right   [--pub-host "<publisher-ip>"]   # right hand on real hardware

# 4) Open the viewer: http://localhost:8080

# 5) Stop: Ctrl+C (or press action button on glasses to stop publisher)

Flags

flag	meaning
--hand left / --hand right / --hand both	Which hand(s) to teleop (default both).
--real	Drive wujihandpy hardware (single hand only for now).
--pub-host --pub-port	aria2mano publisher endpoint (default localhost:5555).
--config-path	Retargeting YAML (default config/adaptive_analytical_aria2.yaml).
--viser-port	viser viewer port (default 8080).
--share	Request a public share URL for the viser viewer.
--no-enable-viser	Disable the viser viewer entirely.

📨 Pub/Sub details

Subscribes to the hands topic over commlink (RPC on ZeroMQ). Each message is a dict with htd: HandTrackingDict (left, right: each HandTrackingData with landmarks (21, 3) in Aria's device frame). We reorder Aria's landmark indices into MediaPipe's, then call Retargeter.retarget(...) to get (20,) joint angles (radians).

aria2mano is not installed here — it lives on the publisher host. The subscriber accesses payload fields duck-typed (getattr(htd, "left", ...)) so it doesn't need the dataclass module on its path. This keeps the Aria SDK out of the edge-device install.

🎚️ Live tuning

The viser viewer exposes a "Retargeting" GUI folder that mirrors the YAML config — every parameter is a slider or checkbox:

• Loss / weights — huber_delta, huber_delta_dir, w_pos, w_dir, w_full_hand, norm_delta
• Smoothing — lp_alpha (output low-pass)
• Rotation (deg) — mediapipe_rotation.{x,y,z} (extrinsic XYZ Euler in MANO frame)
• Segment scaling — global scaling + per-finger {PIP, DIP, TIP} (×5 fingers)
• Pinch thresholds (cm) — {d1, d2} per non-thumb finger (controls TipDirVec ↔ FullHandVec α blend)

Slider drags silently mutate an in-memory config; nothing rebuilds until you click an action button:

• Apply Now — atomically swaps in fresh Retargeters for every active side. Next tick uses the new config (works in both sim and --real). Resets the LP filter, so commanded qpos may step once.
• Save to YAML — writes the current config back to <repo>/config/adaptive_analytical_aria2.yaml (comments and key order preserved via ruamel.yaml round-trip).
• Reload from YAML — re-reads the file from disk, re-syncs every widget, and auto-applies. Use this after editing the YAML in your editor.

Geometry-affecting parameters (mediapipe_rotation, scaling, segment_scaling, pinch_thresholds) are reflected in the pink scaled-target overlay so you can preview what the optimizer is being asked to fit before clicking Apply.

🤖 Hardware notes

--real requires wujihandpy. Install with pip install -e ".[real]". Hardware loop sends qpos.reshape(5, 4) to set_joint_target_position at the rate the publisher delivers (~30 Hz with teleop_profile.json). On Ctrl+C the driver writes a 0-rad target and disables the hand. Dual-hand --real --hand both is not yet supported; run two single-hand instances on separate processes.

🤙 Pause / resume with shaka

Retargeting starts PAUSED at boot — the WUJI hand stays still until you engage. Hold a shaka (thumb + pinky extended, three middle fingers curled into the palm) for 1 second to toggle ON. Same gesture toggles back to PAUSED. The Rich table shows per-side state; the console logs each transition.

• Per-hand: in --hand both, each hand toggles independently.
• Pause skips retarget() and driver.step(). URDF/sim freezes at last qpos; real WUJI hardware holds its last commanded pose.
• Release the gesture between toggles — holding past 1 s latches; it does not repeat-toggle.
• Detection uses bend-angle cosines at PIP/IP joints (straight ≈ −1, curled ≈ +1). Tune _EXT_COS / _CURL_COS at the top of src/utils/gesture.py.
• Set ARIA2ROBOT_DEBUG_GESTURE=1 to log per-finger cosines live for threshold tuning.

🧪 Tests

pytest