SkateArm

A two-armed robot you can drive from your browser — a 3D simulation today, built to switch to the real R.Botic Skate when the hardware arrives.

Status — simulation today. Everything here is sim-validated in MuJoCo; the real Skate is en route, so no real-hardware performance is claimed yet — hardware bring-up is Phase 2.

▶ Drive the twin in your browser — no install needed
Jog the joints live over a recorded scene — the fastest way to feel the cockpit.

Prefer to watch or read? product film · live demo & full write-up

_{Built by a mechatronics student open to junior robotics-software roles ↓}

Left: Phase 1 complete — the autonomous bimanual assembly cycle (a GRAFCET sequencer — the IEC 60848 step/transition chart industrial cells are written in, here steps S0–S7 — plus camera QC). Right: Skate Commander — mirror-mode bimanual jog, then teach-in: move the arms by hand and the cockpit writes the rbt program itself.

Prior real-hardware work — SO-101 bring-up

Why this is here: SkateArm is deliberately sim-first, but sim-to-real isn't foreign ground — on an earlier project I'd already brought up a real robot on the same ROS 2 / MoveIt / LeRobot stack.

On a real SO-101 / SO-ARM101 follower-leader arm pair I ran the whole physical stack on native Ubuntu 24.04 · ROS 2 Jazzy · MoveIt · ros2_control · LeRobot: follower-leader teleop, multi-camera capture, dataset record / replay, and an ACT policy trained and run online on the physical arm — then drove the real controllers end-to-end through MoveIt, verifying joint-by-joint that all six commanded channels moved the actual arm.

_{Real arm ↔ RViz / MoveIt model. The honest lesson from real hardware: a green MoveIt state does not mean the arm is calibrated or safe to move — real-pose ↔ model reconciliation was the hard part — exactly the sim-to-real gap SkateArm is built to close.}

→ SO-101 · real-hardware ROS 2 + MoveIt bring-up — the full engineering log: bring-up, controller validation, and the calibration / state-mismatch debugging lessons.

What are you here for?

You want to…	Go to
Drive the robot (twin or real) from a browser	Skate Commander
Connect a ROS 2 / MoveIt 2 stack to a Skate	skate_ros2
See the autonomous assembly cell	Work-cell
See contact-force manipulation (F/T · compliance · gripper)	Manipulation core
Get the control-ready model & collision layer	Sim foundations
Run it yourself	Quick start

New to the jargon? — a 20-second glossary   click to expand

• MuJoCo — a physics simulator; the robot "lives" here virtually before any real hardware exists.
• ROS 2 — the standard open-source middleware (the robot's "operating system").
• URDF — the file describing the robot's links, joints and limits.
• FK / IK — forward kinematics ("where the hand is for these joint angles") / inverse kinematics ("which joint angles put the hand there").
• TCP — tool center point: the exact tip of the tool the robot controls.
• Jog — nudging a joint or the tool one small step at a time (hold a button or drag a slider).
• Digital twin — a 3D copy of the real robot, driven by the same commands.
• Deadman / E-STOP — safety: motion stops if the connection goes silent or you hit emergency-stop.

Skate Commander — web cockpit

Early access · under active development — v0.8.5 is sim-first; drive the twin in your browser now, real-Skate support lands with the hardware.

A browser cockpit for the Skate: a 3D digital twin built from the official URDF, driven over the same UDP wire the real robot speaks. Starts E-stopped, arms at the robot's measured pose, deadman drops in 0.3 s if the tab closes.

Manipulability cloud · live telemetry plots (30 Hz) · the workstation shell (menu bar, tool rail, Stage / Property dock) · ghost-preview Approve / Cancel gate

Open-source integrations

Skate Commander integrates best-in-class open-source robotics tools — each opt-in, permissively licensed (so it can actually ship), and validated in the sim before it lands. Every one falls back cleanly: a missing optional dependency never disturbs the numpy drag-IK, the browser twin, or the cockpit tick loop.

Integration	What it is	Role in the cockpit	Licence	Enable
mink	MuJoCo differential-IK QP solver	Self-collision-avoiding drag-IK backend — keeps a safe standoff from torso / legs / other arm as a hard constraint (+13 mm vs −90 mm penetration)	Apache-2.0	--ik mink
rerun.io	Multimodal 3D + time-series viewer	Scrub-able telemetry beside the twin — the meshed robot in 3D plus per-arm joint / IK / manipulability plots	Apache-2.0 / MIT	--rerun
LeRobot	Hugging Face robot-learning stack & dataset standard	Export teach-in / teleop demos as a LeRobotDataset v3.0 → train ACT / Diffusion Policy / π0	Apache-2.0	⤓ LeRobot

_{rerun.io integration — the meshed twin + scrub-able telemetry, live beside the browser cockpit}

_{Each lands behind a flag or a button and is documented in the roadmap; more integrations are on the way.}

Deep-dive · From cockpit to policy — an ACT visuomotor imitation-learning pipeline

The LeRobot integration taken end-to-end: a scripted DLS-IK expert (damped least squares — the inverse-kinematics solver that stays well behaved near a singularity by trading a little accuracy for a bounded joint speed) produces bimanual-reach demos, exported as a LeRobotDataset v3.0; an ACT policy is then behaviour-cloned from them on a single 4 GB laptop GPU and rolled out closed-loop in the same MuJoCo twin. At inference it sees only pixels + joint angles — never the target coordinates — so it reaches purely from vision. An imitation-learning pipeline run end-to-end on a laptop, not a claim that reaching was discovered from scratch.

  
_{Left — the trained policy driving both arms to the targets from a single camera. Right — reach error vs two no-vision baselines (a learned state-only policy and a fixed mean pose); only vision clears the 8 cm success line.}

flowchart LR
  A["Skate Commander<br/>sim twin · skt_v3"] -->|"render + DLS-IK teach"| B["LeRobotDataset v3.0<br/>40 episodes · front camera"]
  B -->|"lerobot-train (ACT)"| C["ACT policy<br/>ResNet18 + Transformer · 52 M"]
  C -->|"closed-loop rollout"| D["Reaches from pixels<br/>≈ 5 cm mean error"]

Loading

1 · Dataset — bimanual reach from a front camera

 

Dataset spec — fields, sizes, task string   click to expand

Field	Value
Format	LeRobotDataset v3.0 (mp4 video + parquet)
Episodes / frames	40 / 1 880 @ 30 fps
Camera	fixed front view, 256×256, neutral-gray robot
Targets	orange (right hand) + blue (left hand) floating handles, randomized in the reachable workspace
observation.images.front	RGB video, 256×256
observation.state / action	14-DoF arm pose (rad) / next commanded pose (ALOHA convention)
Task string	reach the orange (right hand) and blue (left hand) targets
On disk	≈ 5 MB

Every episode both hands glide from the home pose to two random targets via damped-least-squares IK — a straight Cartesian line with a smootherstep speed profile. Rejection sampling keeps only target pairs both hands can reach (< 1.2 cm residual), so every demonstration lands cleanly on its marker. Written with the real lerobot writer, so it loads with LeRobotDataset(...) anywhere.

2 · Training — ACT on a laptop RTX 3050

Training config — model, hardware, hyperparameters   click to expand

Setting	Value
Policy	ACT — ResNet18 vision backbone + Transformer, deterministic (no VAE)
Trainable params	52 M
Hardware	NVIDIA RTX 3050 Laptop · 4 GB via WSL2 CUDA passthrough
Batch · steps	4 · 20 000
Wall-clock	≈ 32 min (~10 steps/s)
Peak VRAM	0.62 GB
Final L1 loss	0.070
Image norm	ImageNet stats via the LeRobot processor pipeline

Chunked action prediction (chunk_size = 32), ImageNet-pretrained backbone, and use_vae = false — a deterministic policy is the right fit for a deterministic reach and removes the VAE train/inference gap on a small dataset. The whole run sits comfortably under 1 GB of VRAM.

3 · Rollout — reaching from pixels

_{Home → reach, four episodes. The policy sees only the camera frame + joint angles.}

Reach eval — 24 unseen rollouts · identical targets	ACT (vision)	State-only (learned, no camera)	Mean pose (no learning)
Mean reach error — right hand	5.6 ± 0.6 cm	13.7 cm	19.6 cm
Mean reach error — left hand	5.2 ± 0.3 cm	16.5 cm	19.7 cm
Both hands within 8 cm	69 % ± 9 %	0 %	0 %
Median worst hand (pooled)	6.9 cm	17.4 cm	—
Target marker diameter	12 cm	—	—

Closed loop: each step the policy receives the current camera frame + 14-DoF joint state and predicts the next pose; the sim applies it kinematically (forward kinematics — the loop is closed on vision, not on actuator dynamics or contact), re-renders, and feeds it back. The arms converge on targets whose coordinates the policy is never given — a pure visuomotor imitation of the reach mapping. Evaluation is in-distribution (same twin and target distribution as training, on held-out target draws), so it validates the learned vision→motion mapping — not sim-to-real transfer, which is Phase 2.

_{ACT numbers span 3 independent training seeds (24 held-out rollouts each; the chart's error bars are the seed-to-seed spread). Two no-vision controls on the same targets bracket it: a fixed mean pose and a learned state-only policy — a small MLP mapping joint state → next commanded pose, trained on the same demos with no camera. The learned control settles near the workspace centroid, so it beats the fixed pose (13.7 / 16.5 cm vs 19.6 / 19.7 cm) — but with no target signal it still never lands (0 % success). Vision is what turns ~15 cm into ~5 cm.}

Does it hold outside the training box? No — and here's the honest measurement. Re-running the same checkpoint on targets shifted beyond the training reach volume (further forward and out) — but all still IK-reachable to <2 cm — the reach collapses:

Same checkpoint · 24 rollouts	In-distribution	Out-of-distribution
Reach error — right / left	5.6 / 5.2 cm	16.8 / 12.9 cm
Both hands within 8 cm	67 % [47–82]	0 % [0–14]
Targets IK-reachable < 2 cm	100 %	100 %

Every out-of-distribution target was physically reachable (mean IK residual 7 mm), so this is a real generalization gap, not an artifact: the learned vision→motion map interpolates inside its training reach volume and does not extrapolate past it. ood_reach.py (MODE=indist|ood) reproduces both columns from the one checkpoint; raw numbers in eval_data/ood.json. Bracketed ranges on the success rates (here and in the robustness and dynamics tables below) are 95 % Wilson score intervals for n = 24 — wide by design, so read cross-condition gaps as indicative, not exact.

Further stress tests — camera robustness & real actuator dynamics (same checkpoint): degrades gracefully under camera jitter, survives full mj_step dynamics   click to expand

Does it tolerate a shifted camera and lighting? Mostly — and it fails gracefully. Re-running the same checkpoint on the same targets under domain randomization — camera extrinsics jittered (az ±12° · el ±6° · dist ±8% · look ±3 cm), plus lighting and robot/floor appearance, task cues (orange/blue targets) kept fixed:

_{The three render conditions (episode 0): clean · camera jitter · full DR.}

Same checkpoint · 24 targets	clean	camera jitter	full DR
Reach error — right / left	5.7 / 5.0 cm	7.2 / 7.4 cm	7.3 / 7.4 cm
Both hands within 8 cm	71 % [51–85]	38 % [21–57]	42 % [24–61]

The reach degrades gracefully — it doesn't collapse (≈ 7 cm, not the ~16 cm OOD failure): a mis-calibrated camera raises the error ~40 % and roughly halves success, but the arms still localize. Full DR ≈ camera-only, so the sensitivity is dominated by camera geometry, not lighting/appearance — a useful sim-to-real signal. robust_reach.py (clean|cam|dr) reproduces it; raw numbers in eval_data/robust.json.

Does the reach survive real actuator dynamics? Yes. The rollout above is kinematic — each predicted pose is written straight to the joints. Re-running the same checkpoint but commanding every pose through the model's torque-limited position servos and integrating full rigid-body dynamics under gravity (mj_step), the reach holds:

Same checkpoint · 24 rollouts	Kinematic (teleport)	Dynamic (servos + mj_step)
Reach error — right / left	5.7 / 5.0 cm	5.2 / 4.6 cm
Both hands within 8 cm	71 % [51–85]	88 % [69–96]
Unstable / diverged	—	0 / 24

_{Same policy, driven through the position servos under gravity (mj_step) rather than teleported — four episodes.}

The servos track each command to ~2° (0.034 rad) and every episode stays stable, so the kinematic number wasn't hiding a dynamics cliff — the commanded poses are physically realizable. (The kinematic column is the matched same-24-targets baseline — the clean condition of the robustness eval, 71 %; the dynamic column's small edge over it is servo settling smoothing the final pose plus 24-rollout noise, not evidence that dynamics helps — the honest point is the reach doesn't degrade under mj_step.) dynamic_reach.py reproduces the dynamic column (eval_data/dynamic.json); the matched kinematic baseline is the clean row of eval_data/robust.json — same checkpoint and targets. Contacts are disabled in the control scene (the raw meshes self-jam at the shoulders), so this adds gravity, inertia and torque limits — not self-collision.

How this was actually debugged: the policy trained to 0.070 loss but first rolled out 0.65 m — worse than home. The culprit was a silent normalization-contract bug, not the weights. Full story → The ACT policy that reached for garbage.

Reproduce & artifacts — full commands, eval harness, release, gotchas   click to expand

Full scripts: tools/skate_commander/examples/act_reach/ — run from that directory.

# 1 · generate the dataset from the MuJoCo twin (osmesa offscreen render)
MUJOCO_GL=osmesa python gen_reach_dataset.py 40 256

# 2 · train ACT on the RTX 3050 (~32 min)
lerobot-train \
  --dataset.repo_id=skate/reach_act \
  --dataset.root=../../lerobot_datasets/reach_act \
  --policy.type=act --policy.use_vae=false --policy.device=cuda \
  --batch_size=4 --steps=20000 --save_freq=5000 \
  --output_dir=../../act_reach

# 3 · roll the trained policy out closed-loop in the twin
MUJOCO_GL=osmesa python rollout_act.py ../../act_reach/checkpoints/020000/pretrained_model 6

Artifacts & eval harness. The eval that produces the numbers above is in-repo, not just a claim: baseline_reach.py (the mean-pose baseline), state_baseline_reach.py (the learned state-only baseline) and aggregate_reach.py rebuild the mean ± std table and the chart from the per-seed rollouts, and the raw eval_data/ (3 seeds + baseline JSONs) is committed — so the headline is one command to check. The 40-episode dataset and trained checkpoint (with normalization processors) are also published as release ⤓ act-reach-v1. The one non-redistributable piece is the skt_v3 model (Rbotic/skate_teleop), fetched by sim/make.py --clone.

Notes & gotchas — the non-obvious bits   click to expand

• Reach toward the robot's real front (+Y). The arm workspace opens up fully only in front of the chest; reaching behind is cramped and self-occluded. The fixed camera looks at the true front, so both handles always sit between camera and torso and stay visible.
• Normalization lives in the processors, not the policy. In lerobot 0.6.0 the ACT model has no built-in normalize / unnormalize — inference must be wrapped with make_pre_post_processors(...): preprocessor(obs) → select_action → postprocessor(action). Skip it and the policy silently receives un-normalized inputs and drives the arms to garbage.
• VAE off for small data. With only 40 demos the VAE's latent-conditioned decoder behaves poorly at inference (latent = 0); a deterministic policy trains cleaner and reaches far better.
• 4 GB is enough. Batch 4 holds peak VRAM at 0.62 GB; PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True avoids fragmentation stalls on the laptop GPU.
• Eval scripts write <name>_metrics.json to $ACT_TMP. Each *_reach.py (ood / robust / dynamic / state_baseline …) writes its metrics as <name>_metrics.json into $ACT_TMP; the copies committed under eval_data/ are that JSON renamed to <name>.json (e.g. robust_metrics.json → eval_data/robust.json).

---

The cockpit is a full teleoperation workstation — drag-IK and mirror-mode bimanual motion, RRT-Connect collision-routing, Python + teach-in programs, a Stage / Property shell, live telemetry plots, a TF tree, diagnostics, and scene markers with keep-out obstacles. The full catalogue:

Full cockpit feature catalogue — motion · programs · vision · safety · observability · scene tools   click to expand

Motion, IK & manipulability

Feature	What it does
Jog + sliders	Hold −/+, drag the thumb, or jump straight to a limit; amber = your command, azure = actual position
Cartesian jog	Step the TCP along world X/Y/Z in mm — server-side IK, auto-stops on arrival
Drag-IK (3- or 6-DoF)	Grab a wrist sphere in 3D — server-side DLS IK (damped least squares inverse kinematics) glides all 7 arm joints. The tool rail's rotate mode adds wrist orientation for a full 6-DoF pose target (position is held while the wrist reorients)
Singularity awareness	Live manipulability readout; a SING chip warns near a wrist singularity, where a small cartesian move would need huge joint speeds
Manipulability map	A DEX toggle renders a coloured point cloud of the arm's reachable workspace — warm where the arm is dexterous, blue near its singular reach limits
Mirror mode	Bimanual: jog/slider/IK on one arm is reflected onto the other — the sign map is measured from the model's FK, not guessed
Dual-arm carry	CARRY — both wrists hold one object and move together via an X/Y/Z pad, preserving their separation (a true two-handed carry)
Jerk-limited motion	Jog, replay and Home use acceleration-limited / trapezoidal profiles — motion eases in and out instead of snapping (E-STOP still stops instantly)

Programs & teaching

Feature	What it does
Python programs	Built-in editor + rbt API (movej/pose/movel/home/waypoints); Click-to-Step runs one motion at a time; E-STOP or any manual input kills the program
Control flow	A + FLOW snippet bar inserts indent-aware repeat / while / if / wait skeletons, with rbt.ok() / blocked() / contact() / near() condition helpers — loops and conditionals run on the same guarded bridge
Natural-language programs	Describe a task in plain English — a safe offline parser writes the rbt program into the editor (AST-validated; optional LLM fallback), which you then Click-to-Step through the same guarded bridge
Teach-in recording	Press ● REC, move the robot by hand — every settled pose becomes a line of rbt code, ready to replay
Waypoint sequencer	Record poses, play with pause/loop, save/load named sequences

Tools & traces

Feature	What it does
Tool / TCP offsets	Named end-of-arm tools (mm offsets); FK, IK, traces and the gizmo all follow the active TCP
TCP traces	Colored tool-center-point trajectories drawn in the viewport

Vision & autonomy — validated in simulation, returning with the depth camera

These camera-derived tools were built and validated against the MuJoCo render, then parked behind a "Camera tools — under development" stub in v0.8.0 — the live camera must be a real connected depth sensor, not a rendered one, so they re-enable when the hardware arrives (the vision backend stays in the tree as reference). The sim numbers below are real: they are sim-validated, not live cockpit toggles today.

Capability (sim-validated)	What it does
On-board camera	A camera view rendered from the model (MuJoCo) and streamed into the cockpit (MJPEG), switchable between viewpoints
Work-camera point cloud	A PCL toggle back-projects the work camera's depth into the twin — a coloured 3D point cloud of what it sees (table, target), the input the grasp planner consumes
Vision-guided pick	DETECT finds the workspace target and back-projects its centroid to a world pose (~2 mm vs ground truth); PICK drives the right arm to it through the same IK + collision guard and closes the gripper
Smart pick (multi-object)	A GRASP toggle synthesises a top-down parallel-jaw grasp on the point cloud for every object (RANSAC removes the table, clusters the rest, fits a grasp — centre, measured height, footprint, yaw, width check — to each object's own geometry, rejecting the robot's own limbs). A pluggable detector labels each by colour + shape (opt-in YOLO backend for real objects); an object selector + SMART pick the chosen one by name through the IK + guard
Closed-loop visual servoing	SERVO locks the gripper onto the target in image space as it descends — robust to camera-calibration error (open-loop misses ~43 mm, IBVS ~5 mm in sim)

Safety & modes

Feature	What it does
Collision guard	Every target checked for self-collision before it is sent — including along interpolated paths; capsule / box collision model
Contact reflex	A torque spike on a stalled arm joint (loaded but not moving — i.e. pushing into something) latches a soft-stop; clear it from the CONTACT chip
Planned routing	When a straight move (Home or a waypoint goto/play) would clip a self-collision, an RRT-Connect planner (bidirectional RRT) routes the arms around it (collision-free) instead of stalling — the legs / balance chain are left untouched
SIM / REAL toggle	Same protocol either way; switching always re-latches the E-STOP

Observability & operator tools

Feature	What it does
Live telemetry plots	Foxglove-style scrolling strip charts (joint angle / velocity / temperature / TCP / link RTT) at 30 Hz — colour-coded legend, click-to-toggle lines, pause, current-value markers
Live TF frame tree	RViz2-style transform tree (world ▸ base_link ▸ arm flanges) with world-mm readouts and eye-toggled RGB axis triads that track the kinematics
Diagnostics panel	RViz robot_monitor-style status tree (system link, E-STOP, overtemp, guard, contact, RTT + per-joint temp / vel / load) with OK / warn / error dots and a worst-status badge
Joint-limit meters	Each joint's slider edge and value tint amber near a limit (red at the hard stop), with an amber bounding box on the link in 3D
Collision-mesh display	A collision-mesh toggle (key B) renders the guard's actual capsule / box model in 3D and reddens any contacting pair — see exactly what the guard sees
TCP-force overlay	A TCP-force toggle (key F) draws a per-arm end-effector force arrow estimated from the joint torques ((J·Jᵀ)⁻¹·J·τ), low-pass filtered, amber when straining (> 12 N)
Trajectory replay + scrub	A 45 s rolling record of joint motion with a scrubber and Play — drag to freeze the twin at any past instant; an amber playhead tracks it on the strip charts
CSV export	One-click ↓ CSV of the current plot signal or the full 26-DoF recorded trajectory (degrees, real timestamps)
Global speed override	A SPD slider scales all motion server-side — jog and every glide (home, sequences, RRT routes)
Sim transport & inspection	Play / Pause / Step / Reset of the autonomous motion with a run clock; a two-point measure tool; a viewport stats HUD (FPS / draw-calls / triangles); Stage search + a 3D selection outline
External telemetry (rerun.io)	Optional --rerun streams the live twin into a rerun viewer — the full meshed robot in 3D beside scrub-able joint / drag-IK / manipulability time-series; opt-in, off by default

Scene, markers & planning

Feature	What it does
Stage hierarchy & inspector	An Isaac-Sim-style STAGE tree (World ▸ Skate ▸ arms ▸ joints + overlays / grid) with visibility eyes; click any node for a live PROPERTY inspector (name, type, world pose)
Viewport display settings	A gear popover toggles grid / axes, sets camera FOV, swaps the background, and flips render quality
Scene markers	Spawn a target in reachable space and drag its X/Y/Z gizmo; each marker shows live reachability (green / red), one-click →L / →R go-to (server-side IK), →P to append rbt.moveto(…) to a program, and ⇄ both for a simultaneous bimanual reach
Virtual obstacles	Spawn keep-out boxes and place them freely with a 3D gizmo, sized to any W×D×H — the RRT-Connect planner and the collision guard route the arms around them
Planning preview	Before a Home or waypoint move, a translucent ghost robot shows the destination pose and a blue trail shows the planned collision-free route, gated behind Approve / Cancel
Save / load scene	Save the placed markers + obstacles to a JSON scene file and reload them later

v0.8.5 cockpit — an Isaac-Sim-style workstation: a menu bar, a left tool rail, the 3D MuJoCo twin, a STAGE / PROPERTY dock and live telemetry plots. Mirror mode, dual-arm carry, jerk-limited motion and teach-in all live here. ▶ Live preview (drive the joints — no install) · full docs: tools/skate_commander/

skate_ros2 — the wire

A ROS 2 driver over Skate's native UDP protocol (documented packet layout, deadman semantics, 26-DoF ordering) plus a MuJoCo sim endpoint speaking the same protocol — develop your stack before the robot arrives, then swap 127.0.0.1 for r.local. Safety mirrors the firmware: arm-at-measured-pose, command-freshness deadman, 58 °C overtemp latch. The wire & safety logic is unit-tested without ROS; end-to-end verified over real sockets. On top of the wire sits a MoveIt 2 planning stack (below).

A scripted client drives the MuJoCo endpoint over real UDP packets. At t = 11 s it goes silent — the watchdog dampens the robot. HD video: ros2_wire_demo.mp4

On the wire (sim endpoint)	Result
Command rate	60 Hz sustained (configured target)
Telemetry	~190 packets/s
Tracking error	0.015 rad (vs the MuJoCo model)
Watchdog dampen after silence	< 0.3 s (configured timeout)

These are sim-endpoint figures: command rate and watchdog timeout are configured targets confirmed in simulation, and tracking error is against the MuJoCo model. Real-hardware numbers come once the Skate arrives.

MoveIt 2 motion planning

On top of the wire, skate_moveit_config adds MoveIt 2 planning for the two arms — left_arm / right_arm / both_arms groups, an SRDF generated from the URDF, OMPL. Built & end-to-end-verified on ROS 2 Jazzy: move_group loads the config, MoveItPy plans collision-free bimanual trajectories, and the full loop executes with the sim arm moving to the planned pose. A FollowJointTrajectory bridge streams the plan to the driver, so MoveIt inherits the same deadman / e-stop / overtemp safety instead of re-implementing it:

MoveIt 2 (skate_moveit_config)  →  FollowJointTrajectory bridge  →  skate_driver  →  UDP  →  MuJoCo sim / real Skate

RViz MotionPlanning on this stack — set a goal, Plan, Execute; the Skate arm drives to it live over the UDP bridge. (HD clip)

Prefer the standard controller stack? skate_ros2_control provides a C++ ros2_control SystemInterface plus per-arm JointTrajectoryControllers under the same controller names — MoveIt executes through controller_manager with zero config changes (verified end-to-end on Jazzy, no Python bridge in the loop).

The cockpit can attach to the same sim endpoint as a pure observer — toggle OBSERVE and a MoveIt execution renders live in the browser twin, with an EXTERNAL chip while it moves (details).

Full docs + a Windows/WSL2 setup guide are in tools/skate_ros2/ and tools/skate_moveit_config/.

Rates & tracking from the demo run · full docs: tools/skate_ros2/

Autonomous work-cell (Phase 1 — complete)

The demonstrator task, end to end in simulation: the left arm fixtures a base part in the air, the right arm aligns a peg by relative servoing and inserts it with a torque-guarded descent — a joint-torque (τ) watchdog in the sim, not a wrist force/torque sensor. A GRAFCET sequencer (the IEC step-sequencer standard used in industrial soft-PLCs) runs the full cycle on sensor-based transitions — no timers — and two fixed cameras with classical CV deliver the accept/reject verdict that drives it. Every transition is logged to JSON and fed into a Flask + SQLite SCADA dashboard.

Left: the bimanual insert (τ-watchdog guarded, depth 18.5 mm, peg tilt ≤ 2°). Right: the overhead QC camera's annotated verdict. HD video: cell_cycle_demo.mp4 · cell_assemble_demo.mp4

Key number	Result
Cycle time	42.6 s (takt target ≤ 60 s)
QC residual, alignment (camera vs sim oracle)	±1.6 mm
QC residual, insertion depth	±3.4 mm
Accept rate	functional — only 2 cycles logged so far (sample too small for a true rate; tracked live on the dashboard)

Dashboard live previews: overview · cycle detail — code in dashboard/, sequencer in sim/sequencer.py, QC in sim/qc.py.

_{→ Phase 1's τ-watchdog insert is since replaced by real contact-force control — the cycle above runs the force-regulated S4. The numbers above are the weld cycle, which stays the default and is the one the QC fixture's cameras verify. The actuated gripper is now wired into the live cycle too, on both hands: make_cell_scene.py --gripper builds an opt-in scene where S1's receptivity is a measured grasp force rather than a weld toggle, the left hand picks the base off the table and sets it down instead of holding it in mid-air, the peg is carried on friction (0.42 mm slip), and S4 releases by opening the jaws — a cell with no weld active in it anywhere, 22.1 mm insert, 75.8 s, sim/test_cell_gripper.py in CI. That cell is camera-gated at the assembly station, not at the QC fixture: converting the left hand put the tool in the overhead camera's line of sight, so the gate was restored by adding a second pair that reads the finished unit in situ — in the one instant both hands are off it, which a welded cell structurally never has. See the Manipulation core below.}

Manipulation core (contact-force control)

The review's single largest gap — now closed in sim. Phase 1's insert was a scripted descent guarded by a joint-torque (τ) watchdog, and grasping was a magnetic weld stand-in. The manipulation core (Phase 1.5) replaces both with genuine contact-force manipulation — a wrist force/torque sensor, a force-regulated insertion, Cartesian compliant (admittance) control, and a real actuated gripper — each sim-first, each shipped with a CI test. Full scope & honesty rules: docs/MANIPULATION.md.

Left — M3 compliant control: an external push on the wrist makes the TCP yield at a commanded stiffness and return. Right — M4 actuated gripper: a full weld-free pick-and-place — grasp a part under grasp-force control, carry it, and open the jaws to place it in the bin, held by friction throughout.

Phase	What shipped	Key result (sim)	In CI
M1 · wrist F/T sensing	a 6-axis force/torque sensor on each wrist — the real contact wrench, not the τ proxy — streamed to the cockpit as telemetry id 6 and drawn per arm, tagged with which backend is live	sensor vs analytic < 0.05 N / N·m, and the streamed value pinned to that reading bit-for-bit. Against the same known loads the joint-torque estimate is 0.76–4.26 N off at a working pose and misses by 9.7 N (and invents 26.5 N) at the near-singular home pose — so the measurement leads and the estimate is the fallback	test_ft_sensor.py · test_wrist_wrench.py · eval · data
M2 · force-regulated insertion	axial admittance + spiral bore-search — replaces the "1.4 mm/cycle + τ-watchdog" descent	misalignment tolerance 2–4 mm 6/6, 6 mm 5/6, 8 mm 3/6 (open-loop ≤1/6); a 9.3° θ-tilt levelled to <2°; round H9 bore; peak force ≤4.6 N (abort 9)	test_insertion.py · eval · data
M3 · Cartesian compliance	TCP admittance (yield-at-stiffness), a bimanual compliant carry, and a cockpit compliant contact mode	e = F/K across a 16× stiffness sweep; a real +8 N push yields the TCP ~21 mm and returns	test_admittance.py · test_carry.py · data
M4 · actuated gripper	parallel-jaw gripper, grasp-force control, a grasp-slip curve, a weld-free grasp-carry-place on the arm — and those jaws driving both hands of the live S0–S7 cycle (opt-in scene, no weld active anywhere in it)	grasp tracks target 2–5 N; slip payload grows with grasp force; part carried (~5 mm drift) and placed in a bin — no weld. In the cycle: six real grip/release operations with both welds inactive throughout, left picks the base off the table (12.00 N) and sets it down, 0.42 mm slip over the carry, insert to 22.1 mm at 2.98 N peak, jaws open to release, left re-grips the unit (0.43 mm drift) and bins it — 22.12 mm / 1.90° / 1.24 mm ACCEPT, 75.8 s (oracle-gated in the test that measures it, camera-gated at the assembly station in the cycle itself — see the QC note below)	test_gripper.py · test_gripper_arm.py · test_cell_gripper.py · data · QC-cost eval · data

Left — the weld-free cycle (excerpt): the left hand picks the base off the table and sets it at the assembly station, because jaws cannot hold it in mid-air the way the weld did. Right — what that cost, measured in the same 300 px inspection window: the weld path sees 1116 peg px and 7581 rim px and ACCEPTs; the weld-free path sees the left tool — 0 peg px, 89 % of the rim gone — and REJECTs, on a unit the pose oracle accepts on both paths (raw counts, re-derived in CI). HD video: cell_cycle_gripper.mp4

The honest QC boundary, and what closed it. The QC fixture's cameras cannot see the weld-free cell, and the reason is the conversion itself. The left tool has to approach the base top-down — the pocket faces up and the base's 60 mm length exceeds the 41.61 mm jaw gap, so the orientation is forced — which parks the left wrist between qc_top and the unit at exactly the pose QC inspects. Running the unchanged pipeline at its calibration resolution on both paths gives ACCEPT on the weld path and REJECT on the jaw path, so the flip is the cell and not the measurement. The last weld was the one holding the part in the overhead camera's line of sight. That reading stands, and the committed fixture-pair numbers stay on the weld path where they were earned. What has changed is the gate: the repair named here — in-situ inspection optics aimed at the assembly station, a second camera pair and not a re-calibration — is now in the tree. qc_station_side is the same lens at the same standoff as qc_side, aimed at the assembly station instead of the QC fixture, and the sequencer reads it in the one instant S5 leaves the unit standing free with both hands off it: the right wrist has retracted to park and the left has not yet re-gripped. A cell with a weld in it cannot have that instant, because it lets go of the part only onto the weld — so the second gate is a property of the conversion rather than a patch bolted on after it. There the station pair sees 956 peg px and ACCEPTs, the pose oracle agrees on the same frame, and Cell.qc_gate records which pair decided. "Weld-free" and "camera-verified" are true of the same cell now — through a second pair, which is the honest way to say it.

_{M4 grasp-force control — the jaws close to a commanded force and hold the part by friction alone (no weld), then open to release it.}

Only M5 — hardware bring-up remains, and it is genuinely gated on the physical Skate arriving: the sim controllers are identical on hardware, only the wrench source changes (a real wrist F/T sensor or a joint-torque estimator behind the M1 interface).

Sim foundations (Phase 0)

The converted official skt_v3 model ships with no actuators — sim/make_control_model.py adds 26 position servos — the twin's full joint set: two 8-DoF arms (the Skate's headline 16 DoF), an 8-DoF torso column and a 2-DoF head — and holds poses under physics with < 0.03 rad error; sim/make_collision_model.py replaces the jamming raw meshes with auto-fitted collision capsules (boxes via --boxes), so self-collision actually works. Joint/torque sensors and end-effector sites seed the telemetry schema (tracking plot). Honest limitations documented in sim/README.md.

Left: closed-loop control under physics. Right: hands meet and stop — orange boxes are the collision layer. HD video: control_demo.mp4 · collision_demo.mp4

Architecture

flowchart TB
    subgraph cell [SkateArm work-cell]
        direction TB
        SEQ[Sequencer\nGRAFCET / soft-PLC] --> MOT[Motion layer\nROS 2 + MoveIt, 2 arms]
        SEQ --> FEED[Feeder node\nAVR]
        MOT --> SKATE[Skate 16 DoF\nMuJoCo twin → real robot]
        POL[Manipulation policies\nACT / SmolVLA via LeRobot] --> MOT
        SKATE --> QC[QC station\nGD&T accept/reject]
        QC --> DASH[SCADA dashboard\nFlask + SQL]
        SEQ --> DASH
    end
    CAM[2x cameras] --> POL
    CAM --> QC

Loading

Demonstrator task: one arm holds/fixtures a part, the other inserts (peg-in-hole class), then in-cell measurement decides accept/reject and logs to the dashboard. The real Skate (16 DoF, span 1615 mm, RPi 5, UDP control) is en route — Phase 2 starts on arrival; skate_ros2 is already waiting for it.

Full architecture & mapping of all 12 prior portfolio projects onto subsystems: docs/ARCHITECTURE.md. Phased plan: docs/ROADMAP.md.

Quick start (simulation)

git clone https://github.com/dsl-robotics/skatearm.git   # this repo (the sim/ tools below live here)
git clone https://github.com/Rbotic/skate_teleop.git     # the official model (skt_v3)
cd skatearm
pip install -r sim/requirements.txt

# one-shot: build the control + collision models in a single step
python sim/make.py --skt-dir path/to/skate_teleop/skt_v3
#   (or  python sim/make.py --clone   — fetches the model and builds it for you)

# …or run the steps individually:
python sim/render_skate.py --model path/to/skate_teleop/skt_v3         # static renders
python sim/make_control_model.py path/to/skate_teleop/skt_v3           # + actuators & sensors
python sim/make_collision_model.py path/to/skate_teleop/skt_v3         # + collision capsules
python sim/demo_wave.py --model path/to/skate_teleop/skt_v3            # control demo (mp4/gif)
python sim/demo_selfcollision.py --model path/to/skate_teleop/skt_v3   # self-collision demo
python sim/telemetry_demo.py --model path/to/skate_teleop/skt_v3       # tracking/torque plot

Windows: use py instead of python/python3 (the bare names may open the Microsoft Store stub).

Each script is documented in sim/README.md. To drive the twin from a browser, follow the Commander quick start.

Community tools

Tools get built because SkateArm needs them — then released standalone:

Tool	What it is	Status
skate_ros2	ROS 2 bridge over Skate's native UDP + protocol-true MuJoCo sim endpoint	✅ shipped (sim-verified)
skate_moveit_config	MoveIt 2 config for the bimanual chains — SRDF from the URDF, OMPL planning, FollowJointTrajectory bridge to the driver	✅ built & verified on ROS 2 Jazzy
skate_ros2_control	ros2_control hardware interface — C++ SystemInterface + per-arm JointTrajectoryControllers, no Python bridge	✅ verified on ROS 2 Jazzy
skate_commander	Web cockpit — browser digital twin: drag-IK, mirror-mode motion, RRT-Connect routing, teach-in programs, optional mink / rerun.io / LeRobot backends, live telemetry (full feature catalogue · live preview)	✅ v0.8.5 (real-camera passthrough waits for hardware)
Control-ready MJCF	skt_v3 with actuators, ready for control work	✅ first version in sim/
Teleop dataset hub	Bimanual datasets in LeRobot format	planned
MuJoCo benchmark suite	Repeatable bimanual tasks — reach · carry · hand-off · peg-insert · force-regulated peg-insert — with quantitative metrics, headless & seeded, one disturbance stream per task	✅ first version in sim/
URDF/config validator	Sanity-check tool for Skate configs	planned
Getting-started handbook	From unboxing to first teleop	planned

Ideas and requests from other Skate owners are welcome — open an issue.

Why this project: leveling up from a single SO-101 arm to a bimanual humanoid — two-arm coordination and sim-to-real — while learning ROS 2, MuJoCo and policy learning end-to-end, and publishing open tools, datasets and guides for the Skate community.

Related projects

• SO-101 · ROS 2 + MoveIt real-hardware bring-up — a real SO-101 / SO-ARM101 arm pair on ROS 2 Jazzy + MoveIt + LeRobot; teleop, dataset record / replay and an ACT policy trained and run online on the physical arm (featured up top).
• Engineering Portfolio — 11 academic & applied projects: industrial robotics, PLC, embedded systems, metrology, CNC, mechanical design.

Author

Daniels Skots Lavs — mechatronics student (RTU), industrial electronics technician. open to junior robotics software roles CV (PDF) · GitHub profile · Engineering portfolio · porche121004@gmail.com

License

MIT — see LICENSE. The skt_v3 model and meshes belong to Rbotic/skate_teleop and are not redistributed here.