Rebuild the bed making demo IRL

[armwaheed]

Issue: Rebuild the bed making demo for a real Unitree G1 EDU working with a human partner

Fix:

• Review the work done so far in issue Make demo of two humanoids making a bed #2
• Run another reinforcement learning job for bed making, with a Unitree G1 EDU with only 23 DOF and Brainco 5 finger hands
• The G1 will need to ask and listen for help from a human partner
• Device Connect is still the orchestrator, communicating to the human through the G1's speaker and microphone
• The complete specification of the physical G1 to which I have access is in https://github.com/armwaheed/robotics-connect
• SSH access to the G1 is available upon request
• The goal is to enable the real, 23 DOF + Brainco hands G1 to make an actual bed, which will require the use of LiDAR, the RGB and depth cameras. NOTE that near-field LiDAR is more accurate than the depth camera, because the depth camera produces stereoscopic vision artifacts that make hand placement less accurate

[armwaheed]

Issue #3 — rebuilding the bed-making demo for the real 23-DOF G1 EDU + a human partner

Picking up from the issue #2 two-robot sim, here's the IRL rebuild. All of the sim/software side below is built and eye-verified; hardware bring-up on the physical EDU is starting now (the robot is powered on). Work is on branch issue-3-irl-bed-making (commit c74d406); the voice module is in armwaheed/robotics-connect g1-audio-module.

What changed from issue #2 → #3

Axis	Issue #2	Issue #3 (this)
Robot	2× 29-DOF, Inspire hands	1× 23-DOF EDU, Brainco hands
Partner	a second robot	a human, asked for help only when stuck
Channel	robot↔robot	robot↔human over the G1's speaker + mic
Goal	sheet drawn in sim	a real bed via real LiDAR + RGB + depth

1. A transfer-valid 23-DOF reach policy (the issue #2 policy was not transfer-valid)

The 29-DOF sim G1 has 6 joints the real EDU lacks — waist roll/pitch and both wrist pitch/yaw — and the issue #2 policy leans on waist pitch the EDU doesn't have. So I retrained with those 6 joints locked (held rigid, excluded from the action set and the observation), action = the 23 real EDU joints, EE = wrist_roll_link (the distal actuated arm link). Observation is 85-D (vs 151-D), restricted to what the hardware can actually report — keeping the obs/action contract explicit is the documented #1 sim-to-real lever for the G1 (the public deploy failures are obs-layout / joint-order mismatches, e.g. IsaacLab #4037, not the DOF-locking).

Trained from scratch (2048 envs × 1500 iters on the DGX Spark). Converged: reach_coarse 0.32→0.65, mean episode length ~393/400 (nearly no early terminations = it stays upright). Eye-verified across the rendered episode — the EDU balances on its own two feet at the bedside and reaches onto the bed, no topple, no kinematic cheats:

🎬 full 350-frame eval:
https://raw.githubusercontent.com/armwaheed/robots/issue-3-irl-bed-making/examples/isaac_bed_making/media/rl/g1edu/bed_reach_g1edu.mp4

Deployable policy committed at rl/policy_g1edu/ (policy.pt + policy.onnx). New rl/robot_cfg_g1edu.py + rl/bed_reach_env_cfg_g1edu.py; train.py/play.py take a --robot g1_edu selector.

2. "Robot solo, asks when stuck" — voice coordination over Device Connect

human_partner.py replaces the second robot with a human. The robot attempts the whole bed and only asks for help when it leaves its competence envelope — a CompetenceMonitor (inspired by execution-time failure prediction, BCVA / FAIL-Detect) that fires before a fall, with the trip-points tied to what the policy was actually trained for:

• out of reach (corner beyond the trained 0.55 m workspace),
• too heavy (grip load > the 35 N the policy trained to absorb),
• balance at risk (margin below a safe floor → let go and ask).

Each reason maps to a spoken question; the human's free-speech reply is grounded to a decision (KnowNo-style multiple-choice, "yeah go ahead" → yes). The dialogue + shared goal state are Device Connect @rpc/@emit so the whole human-assisted episode is visible in the dashboard. A full loopback episode passes (robot does 2 corners solo, asks for the 2 it can't, human helps → "the bed is made").

3. LiDAR-first perception (depth is coarse-only)

perception.py now takes the fine hand-placement target from the near-field LiDAR (Livox MID-360, ~±4 cm), not the depth camera — the head RealSense's IR stereo produces artifacts on textureless white bedding that hurt hand placement (your hardware note; corroborated by Seita et al. 2019, depth-over-RGB for bed-making). The depth camera is used only for the coarse coverage check. Same code runs on the sim RayCaster cloud and the real lidar_sight cloud (the real-to-sim loop).

4. Voice (the new hardware capability)

Built in robotics-connect as unitree/g1/voice/ — speak via the Unitree AudioClient over the DDS "voice" service (TtsMaker + raw-PCM PlayStream), listen via a PulseAudio mic capture + local ASR (the SDK doesn't expose the mic). Details in the robotics-connect#1 follow-up.

Honest status & what's next (on hardware, now)

Balance + reach (transfer-valid), the ask-when-stuck dialogue, and LiDAR-first perception are done in sim/loopback. The real bed is the remaining work, in order: (1) verify the voice loop on the EDU, (2) lidar_sight bed detection + near-field grasp live, (3) deploy the 23-DOF policy (mapping Isaac↔SDK joint order carefully for parity), (4) run the full ask-when-stuck bed-making loop and verify by recorded video. The sheet pull-under-load remains the hard manipulation edge (tracked via FALCON-style force curriculum and the Newton cloth path in #5).

Built with Claude Code (Opus 4.8). Physically valid for sim-to-real — no base pinning / teleporting / joint freezing.

[armwaheed]

IRL bring-up on the physical G1 EDU — progress, the microphone, and the human-as-Device-Connect-agent plan

Continuing on the real robot. Full writeup committed as examples/isaac_bed_making/BED_MAKING_IRL.md (branch issue-3-irl-bed-making). Highlights:

✅ Verified on the hardware

• The robot speaks. Built robotics-connect/unitree/g1/voice (Unitree AudioClient over the DDS "voice" service) and the G1 physically said "Hello. I am the bed-making robot." — TtsMaker works; on this firmware speaker_id=0 is Chinese, 1–4 are English (we default to 4).
• Real perception, live off the robot's own sensors — LiDAR (Livox MID-360, 180° upside-down mount corrected, cropped to the room to omit the open doorway), head RGB + depth (RealSense D435i, 51.29° down-tilt):

Head RGB (what it sees)	Head depth	Crown LiDAR (top-down, room-cropped)

Hand placement is LiDAR-first (~±4 cm); the depth cam's stereo artifacts on textureless bedding (the holes above) make it coarse-only — exactly the hardware note in this issue.

• The 23-DOF transfer-valid policy (eye-verified earlier in this thread) and the ask-when-stuck coordination are in place.

🎤 The microphone — an honest negative result (engineers engaged)

The "listen" half turned into a deep dive. Every native path to the G1's 4-mic array dead-ended — AudioClient doesn't expose it (ASR api 1002 → 3104), the one ALSA capture node is held exclusively by the factory, the wake-word capture route isn't on the XBAR mux, nothing republishes on rt/audiosender, and there's no on-board ASR process or voice log:

Root cause: the EDU has no on-board ASR — with Wake-up Conversation Mode on, a closed firmware wake-detector opens the mic and streams it off-robot to Unitree's app/cloud; there's no local userspace hook.

We've asked the Unitree engineers for the supported way to read the array, and filed the full reproduction in the writeup + robotics-connect#1.

🗣️ The plan that unblocks the human loop now

The robot doesn't need its own mic to work with a human, because Device Connect is the human↔robot bus. For the bed-making task, the human talks to Device Connect via a Bluetooth headset on the NVIDIA DGX Spark, with a DC sidecar that captures the headset (local Whisper ASR) and registers the human as a "human agent" in the fabric. The robot asks through its own speaker (verified) and hears through Device Connect — the issue's "Device Connect orchestrates the human interaction," sidestepping the closed on-board mic. If/when Unitree exposes the array, the same coordination layer swaps the headset source for the robot's mic with no logic change.

Next

Walk the EDU to the bed → deploy the 23-DOF policy → run the full ask-when-stuck loop with the human replying via the Spark headset/DC sidecar → Brainco grip on the real sheet. Mic: pending Unitree.

Built with Claude Code (Opus 4.8).

[armwaheed]

Device Connect human-in-the-loop — validated live on the hardware 🎧🤖

Continuing on the physical G1 EDU. The robot ("Rabia") now asks a human for help out loud through her own chest speaker and hears the answer over Device Connect, via a Bluetooth Headset Human Agent on the DGX Spark — both registered live on the Device Connect dashboard (beta tenant). Full writeup: BED_MAKING_IRL.md §7.1.

The loop, verified end-to-end

sequenceDiagram
    autonumber
    participant H as 🧑 Human
    participant R as Rabia · G1 EDU
    participant DC as Device Connect (beta)
    participant HA as Human Agent (headset)
    R-->>H: 🔊 "Can you hold the far corner?" (out loud)
    R->>DC: invoke_remote(human-agent, ask)
    DC->>HA: ask(question, yesno)
    HA-->>H: 🔔 earcon → listening
    H-->>HA: 🎤 "Yes, I'm holding the corner"
    HA->>HA: faster-whisper → ground → "yes"
    HA--)DC: human_replied{choice:"yes"}
    HA-->>R: { choice: "yes" }

Loading

A real run: Rabia asked "Can you hold the far corner of the sheet for me?"; I answered "Yes, I'm holding the corner." into the Jabra → transcribed → grounded to yes → human_replied event on the dashboard → returned to Rabia.

Both devices on the dashboard

Audio, validated on the real hardware

Bluetooth headset capture + energy VAD — close-talk HFP mSBC mic; the VAD cleanly separates the speech from the robot's own (loud) cooling fan, and Whisper transcribes → yes:

Pipeline — ~10.8 s end-to-end loop, and the robot speaker master gain restored to full (a prior session had dropped it to 60, softening even the factory announcements):

Design (the out-loud decision)

Robot→human = chest speaker (out loud, legible, co-present); human→robot = headset mic (always — close-talk SNR); earpiece is an optional quiet-mode toggle. Device Connect is the source of truth (the ask RPC + human_replied event), so the audio channel is a cheap, swappable presentation layer.

Engineering (in robotics-connect, merged to main)

• New robot-agnostic human_agent/ + the G1 unitree/g1/device_connect/ (Rabia) — armwaheed/robotics-connect#9.
• A two-env bridge: device-connect-edge needs Python ≥3.11 but the G1 SDK env is 3.10, so the sidecar runs in a clean 3.11 env and drives the chest speaker via subprocess. Generalized as the agent skill bootstrap-device-connect-env — armwaheed/robotics-connect#8.

Next

1. Walk Rabia to the bed (velocity-walk) and trigger the ask-when-stuck loop from the on-robot competence monitor (the help exchange itself is now verified live).
2. Deploy the 23-DOF policy for the bedside reach (Isaac ↔ SDK joint-order parity).
3. Brainco sensor-gated grip on the real sheet + sustained pull-load.

Follow-up for the next session: a deferred code-review cleanup — consolidate the duplicated Device Connect creds/URL resolution + DeviceDriver shim shared by human_agent.py / rabia_agent.py, and switch get_event_loop → get_running_loop. Built with Claude Code (Opus 4.8).

[armwaheed]

The microphone question is closed — Unitree engineering support confirmed it

Following up on the microphone rabbit-hole and the live human-in-the-loop validation: I took the full reproduction to Unitree engineering support, and they confirmed the negative result directly (support work order, June 2026).

What Unitree engineering support told us (paraphrased):

• There is no way to interface with or develop against the G1's microphone. The only supported route is the robot's built-in automatic speech recognition — along with the other voice-UI features documented under VuiClient_Service.
• To write your own speech-recognition / microphone code, you must attach an external microphone/speaker array over the USB-C ports rather than tap the built-in array.

So the closed-capture finding we filed — AudioClient ASR api 1002 → 3104, the only ALSA node held exclusively by the factory, the wake-word route off the XBAR mux, no DDS republish, no on-board ASR — isn't a gap to route around. It's by design.

Why this is exactly the Bluetooth-headset human agent over Device Connect

The confirmation validates the resolution already built and verified live. The robot doesn't need its own mic to work with a human, because Device Connect is the human↔robot bus: the human joins the fabric as their own agent — a Bluetooth headset on the NVIDIA DGX Spark plus a sidecar that runs local Whisper ASR — and the robot asks through its own (verified) chest speaker and hears the grounded reply over Device Connect. Rabia (the real G1 EDU) and the Bluetooth Headset Human Agent both run the full out-loud ask → answer → ground loop on the dashboard.

The supported on-robot alternative Unitree named — an external USB-C mic/speaker array — is a drop-in for the same coordination layer (just swap the sidecar's headset source), so nothing about the architecture changes if a fully on-robot listen path is ever wanted.

Docs updated to record the confirmation: BED_MAKING_IRL.md §6.1, and on robotics-connect the voice README, lib/README, and robotics-connect#1.

Built with Claude Code (Opus 4.8).

[armwaheed]

Continuation — ready for the live walk-to-bed (fresh session resumes here)

The robot is in the master bedroom in position. Everything up to the actual walk is built and verified on hardware; the live walk is the next step and is gated on safety. A fresh Claude Code session should pick up from here.

What's done + verified

• Generalized locomotion + navigation layer (robotics-connect#11, main@5ed7dd8): G1Locomotion over Unitree LocoClient.Move + measured rt/odommodestate odometry; closed-loop walk_to/turn_to/walk_forward; a sensor-agnostic Navigator (occupancy grid + A*); SimLocomotion for off-robot. The demo's bedside approach is wired to it (human_partner.approach_bed), loopback-verified.
• Controller abort — verified button-by-button on the real controller. unitree/g1/controller/g1_remote.py reads LowState.wireless_remote; any button latches an abort that the locomotion loops poll and that halts the routine (set_abort_source → stop + hold balance). Arms only after the buttons are released. Dry test: A/B/X/Y, full d-pad, L1/L2/R1/R2 all detected, ABORTED latched on the first press. ⚠️ This is a clean software halt, NOT the e-stop — the controller's firmware damping and the robot's physical power/e-stop are the real stop and must stay in reach.
• Bedroom perception (BED_MAKING_IRL.md §4.1, robots@821d452): from the fixed start pose the LiDAR locks the bed as a 0.68 m² plane at (1.44 m fwd, 0.44 m left), head RGB frames the near bed edge, head depth shows the bed+floor with the stereo artifacts on textureless bedding (→ hand placement stays LiDAR-first). All three checked against room photos.
• Depth dependency documented: pyrealsense2 is built from source (the PyPI wheel fails on the EDU's GLIBC) and needs depth_camera_sight/setup_env.sh sourced.

Environment (fixed start pose)

The robot always starts in front of the closed double doors (to the master bathroom), facing the bed ~1.5–2.5 m away across clear carpet (bed ahead and slightly left, headboard at the window). Clutter (step-stool, dresser, clothes) is off to the right, not in the straight-ahead path.

NEXT — the live walk (pass 1)

The operator manually drives the robot to within ~3 m of the bed on a clear path; the routine then does the short autonomous forward approach to the bedside under closed-loop odometry, and hands off to the reach. Before any leg motion (durable rule — WARN the operator first):

1. Read-only pose stream to confirm odometry tracks: g1_locomotion.py --iface eth0 (no motion).
2. Arm the controller abort, confirm clear area + operator on the hardware e-stop + healthy battery.
3. Stand + a short, confirmed walk_forward (the CLI's --forward needs a typed confirmation), then stop.
4. Then trigger the ask-when-stuck loop from the on-robot competence monitor.

In-bedroom human help loop — run the sidecar on a laptop

The Bluetooth headset is out of Bluetooth range of the DGX Spark from the master bedroom, so the existing Spark-hosted human-agent sidecar can't hear the human there. Fix: the human-agent sidecar does not have to run on the Spark — it only needs to be within Bluetooth range of the headset and able to reach the Device Connect NATS fabric over the network. So run the sidecar on a laptop brought into the bedroom (headset paired to that laptop); it joins the same fabric the robot is on, and the robot asks over its speaker / hears over Device Connect exactly as before. No logic change — just move the sidecar host.

Cross-refs: robotics-connect#11 · robotics-connect@5ed7dd8 · robots@821d452.

Built with Claude Code (Opus 4.8).

[armwaheed]

Continuation — on-hardware deploy ladder, an incident, and a transfer-robust retrain (resume here)

Picking up from the walk-to-bed handoff. A fresh Claude Code session should resume from this comment. Full writeup: BED_MAKING_IRL.md §10–11 (robots issue-3-irl-bed-making @ 0e840de).

What happened this session

• Walked the EDU to the bed under closed-loop measured odometry, controller-abort armed (eye- + telemetry-verified; light footboard contact, stayed balanced). Two vendor gotchas fixed in robotics-connect locomotion: BalanceStand needs a balance_mode arg; Move needs continous_move=True (else a 0.03 m/s shuffle that false-trips the stall guard).
• Whole-body RL deploy as a de-risk ladder — rung 0 (offline, joint-mapping verified on hardware; IsaacLab's action order is interleaved, not the SDK 0–28 order — map by name), rung 1 (arm-only via rt/arm_sdk, fall-safe, validated), rung 2 (whole-body via rt/lowcmd, vendor balance released).
• ⚠️ Incident + the durable safety fix: the whole-body transfer failed; the operator's controller-abort correctly damped it, but the deploy process was then kill -9'd — which latched the last high-gain command on the motors → the robot spin-kicked and broke an office window. Fix shipped to robotics-connect: never kill -9 a motor-control process; lib/safe_stop.py damps on every catchable exit; full rules in SAFETY.md. Robot is safely powered off + winched.
• Why the transfer failed → fix: dropping base_lin_vel from the obs (unobservable on the real G1) starved the value function and reach_coarse regressed 0.39→0.29. The research-backed fix is asymmetric actor-critic — base_lin_vel privileged to a critic-only obs group, the actor drops it (82-D, deployable). v2 retrain: reach_coarse 0.59 monotonic, ep_len 394/400, eye-verified robust balanced reach (mp4).
• DGX Spark slowdown solved: a retrain ran 3.2× slower — not config/thermal, but the GB10 GPU stuck in a low-power state (507 MHz/6 W under load). The only fix is a full AC power cycle (NVIDIA forum); after it, v2 trained in 67 min.

Resume from here

1. Hardware re-deploy of v2 (82-D actor, rl/deploy_contract_v2.json) on a graduated gantry — once the robot is powered back up and the window is repaired.
2. Lift the v2 policy + 82-D contract through robotics-connect lib/policy_deploy + the G1 RobotIO binding (DDS paths need an on-hardware re-check).
3. Then: ask-when-stuck loop from the on-robot competence monitor at the bedside; Brainco sensor-gated grip + sustained pull-load.
4. If hardware transfer is still marginal: add actuator latency / motor-strength DR (deferred — needs an actuator-model change).

Built with Claude Code (Opus 4.8).

[armwaheed]

Continuation — 2026-06-15: offline rung validated on hardware; whole-body aborted (mode/abort procedure error); corrected procedure + the hard open question

Resume anchor for the next (fresh) session is this comment. Robot stays powered off until the deploy code is reworked to the corrected procedure below and reviewed.

✅ Shipped / validated this session

• v2 (82-D, asymmetric-critic) reach policy lifted onto the productized robotics-connect harness (842daa9).
• --stage offline validated on the LIVE robot (read-only, zero motor commands):
  • DDS read path OK; 82-D obs built from the real IMU (projected_gravity ≈ [0.03, 0.005, -1.0], upright); v2 TorchScript runs in the deploy env (|a|max ≈ 2.77, finite); command-responsiveness confirmed (right target → right arm leads, left → left = the ambidextrous reach from training).
  • Harness + v2 trio synced to the robot (/home/unitree/robotics-connect-deploy + bedreach_deploy/policy_v2.pt). The offline rung is safe in any mode (no publish).

⛔ Whole-body attempt → aborted (near-miss, no new damage)

Root cause was a procedure/mode error, not the policy:

• Operated from Regular (AI-Sport) mode and tried to "verify the handheld abort" by pressing buttons — but in Regular mode A/B/X/Y combos are bound to vendor gesture routines, so the "abort test" commanded arm motions. Button activity on a loose tether destabilized the robot → it collapsed and ended up in Develop mode.
• The deploy process itself was inert (stuck at the Type 'whole' confirmation prompt — it never released the controller and never published a motor command).
• Resolution: robot powered off + battery ejected — the correct ultimate stop.

Corrected G1 mode + abort model (from Unitree docs)

Action	Buttons (FW ≥1.4)	Notes
Damping / e-stop	L2+B (old: L1+A)	compliant, settles slowly; the operator e-stop; a clean abort only in Develop mode
Locked standing	L2+UP	from damping; support the shoulders
Regular / AI-Sport	R1+X	buttons = vendor gestures here
Develop / low-level rt/lowcmd	L2+R2	precondition: SUSPENDED + DAMPING first; pauses AI-Sport; exit = reboot

Sequence: suspend → L2+B → L2+R2, operator-driven (NOT software MotionSwitcher.ReleaseMode). Verify the abort in Develop mode only. Develop mode executes queued rt/lowcmd → needs DDS hygiene + a damping-first start. Never kill -9 (latches the last command → runaway). The handheld abort works as an abort because it's read in Develop mode — that's what stopped the earlier spin-kick.
Sources: quadruped.de G1 controls FW1.4; Weston Robot G1 dev guide; unitreerobotics/unitree_sdk2_python#43.

The hard open question for next session — the suspension/activation paradox

A whole-body balance policy assumes feet-on-ground dynamics, which creates a real tension:

• Fully suspended (safe from falls) → the policy is off-distribution (no ground reaction) → its closed-loop balance corrections diverge → flailing/panic motion. You cannot meaningfully validate a balance policy while it is suspended — the panic is the guaranteed behavior, not a corner case.
• Feet-on-ground, gantry taut-but-slack as a fall-catch → policy in-distribution, with a real (but caught) fall possible.

Activation transient (the dangerous handoff): the shotgun comes from the policy seeing a pose far from its default and commanding a large action to return to default. Starting from a squat that isn't the default pose makes this worse. The fix:

1. Move to the EXACT default training pose first, under a scripted gain-ramped position move (not the policy).
2. Start the policy with hand_target = a no-op/neutral target (≈0 first action), then ramp the command.
3. First whole-body test = STAND (neutral command), never a reach. Add the reach only after stable standing is proven.
4. Feet-on-ground + slack gantry, NOT fully suspended.

Bigger reframe to revisit: do we need the legs in the loop at all for the bedside reach? --stage arms (vendor balance holds the legs; arms via rt/arm_sdk) delivers the reach with zero whole-body balance risk. Strongly consider doing the safe arms-over-vendor-balance reach first, and only taking on whole-body leg control if the reach demonstrably needs CoM shifting the vendor balancer can't provide.

Plan for next session (code rework first; robot stays OFF until reviewed)

1. Rework unitree/g1/deploy/g1_robot_io.py + the deploy script: drop the software ReleaseMode path, require operator-driven Develop mode (precheck/refuse otherwise), replace the input() gate with in-Develop-mode abort verification, add move-to-default + damping-first + gain-ramp startup, set explicit volatile QoS on the rt/lowcmd publisher.
2. Reorder the de-risk ladder to prefer --stage arms (fall-safe) for the first useful bed-reach.
3. Rewrite the operator runbook in BED_MAKING_IRL.md to the suspend → L2+B → L2+R2 sequence + the layered aborts (handheld L2+B / software latch / kill -TERM / physical e-stop — never kill -9).
4. No hardware until the above is in and reviewed.

[armwaheed]

Continuation — 2026-06-16: architecture decision (arm-overlay + LocoClient), mode-gate fixed, whole-body deferred

Resume anchor for the next (fresh) session is this comment.

The pivot (study of the prior locomotion work + Unitree/NVIDIA forums + live probing)

The whole-body rt/lowcmd deploy we'd been hardening is the wrong path for the 23-DOF G1 EDU, and we have the
right one already:

• CheckMode can't gate the takeover — it returns name='ai' in BOTH AI-Sport and a freshly-entered Develop
  mode (confirmed across a power-cycle); rt/sportmodestate is silent on this variant. The reliable signal is
  high-level service liveness: a read-only loco GET_FSM_ID RPC (id 7001) answers (code 0) when the balancer runs,
  errors/times out when it's off (Develop → code 3102, verified live). verify_whole_body_ready now gates on that.
• rt/lowcmd whole-body fights the live AI controller (jitter/oscillation) — it only works with the controller
  fully off (Develop), which is gantry-only and can't coexist with walking.
• Validated architecture = rt/arm_sdk overlay (blends with the balancer) + vendor LocoClient.Move — the
  eye-verified arms rung. So the bed-making deploy is arm-overlay + LocoClient, not whole-body.

Match the task to the robot (no new RL model needed)

• Bed high enough → upper-body reach within a balance-safe envelope: a deterministic arm trajectory
  (arm_fk IK → rt/arm_sdk). No RL training required.
• Bed too low + 23-DOF → decline and speak (voice): "too low to reach without a whole-body policy."
• 29-DOF → whole-body allowed (gantry/liveness-gated) — descriptor-gated stub (robotics-connect lib/task_gate.py),
  no 29-DOF unit to test.
• Whether 23-DOF whole-body is achievable at all → Unitree eng question, Unitree eng: is a whole-body rt/lowcmd motion policy supported on the 23-DOF G1 EDU? #6.

Shipped (pushed)

• robotics-connect main 2a23a85 — liveness-gated verify_whole_body_ready; operator checkpoint +
  balance-preserving return + gentle release in run_whole; arm-overlay blend-out; --stage arms refuses when the
  balancer is down; SAFETY.md/deploy README/SKILL updated; lib/task_gate.py stub. Tests 14/14.
• robots-issue3 issue-3-irl-bed-making c74124f — deploy-script checkpoint args + arms liveness guard;
  BED_MAKING_IRL.md §10.3 (mode findings + architecture decision) + §10.4 (match-task-to-robot). v2 lift 5/5.
• Studied prior locomotion work clean-room (IP-scrubbed; zero lineage in either repo).

Plan for the fresh session (priority: ship the 23-DOF demo)

1. Flesh out the upper-body reach path: implement lib/task_gate.py's envelope test (perceive-surfaces bed
   height + descriptor effector reach + a balance-safe joint-range model) and the deterministic arm_fk→rt/arm_sdk
   reach. Wire choose_reach_strategy into the bed-making behaviour (examples/isaac_bed_making/human_partner.py).
2. Walk → reach → decline glue: LocoClient.Move to the bed (have it), then upper-body reach if in envelope,
   else voice objection. Register the match-task-to-robot skill (add a skills/ SKILL.md + plugin.json entry).
3. Grip + draw (Brainco hands) on the real sheet — and the open risk below.
4. Whole-body stays gated/stub pending Unitree eng: is a whole-body rt/lowcmd motion policy supported on the 23-DOF G1 EDU? #6.

Open challenges / risks to carry forward

• The full task is reach + GRIP + DRAW, not just reach. Pulling a sheet applies a reaction load the vendor
  balancer must absorb — untested on the arm-overlay path. The 23-DOF demo's honest scope may be "reach + grip +
  arm-range draw on a high-enough bed," not a full bed-make. Validate pull-load on the balancer early.
• We have NOT directly tested whole-body rt/lowcmd in true Develop mode (AI off). The prior failure was while
  walking (AI active = contention). A stationary stand in true Develop mode is untested-but-maybe-possible — Unitree eng: is a whole-body rt/lowcmd motion policy supported on the 23-DOF G1 EDU? #6
  should settle it before we fully write off 23-DOF whole-body.
• Validate the liveness gate's Regular-mode branch directly (we saw code 3102 in Develop and inferred code 0 in
  Regular from LocoClient.Move working; confirm GET_FSM_ID → code 0 in AI-Sport).

[armwaheed]

Continuation — 2026-06-16: pull-load re-litigated (sim) + live 23-DOF whole-body stand tested → confirms the arm-overlay path

Resume anchor for the next (fresh) session is this comment.

Re-litigated — can the 23-DOF EDU survive a bedsheet pull? YES, with margin (sim)

The reach env already trains a ≤ 35 N random grip-slip force (randomize_ee_load); the untested case was a sustained directional draw — what pulling a sheet actually is. New harness rl/pull_stress.py applies a constant outward (worst-case for balance) hand force, magnitude-per-env, montage render (media/rl/pull_load_stress.mp4). Eye-verified: holds at 0 / 15 / 35 / 55 N, collapses at 75 N. A realistic draw is ~2–15 N → ~4–10× margin. Load is not the topple risk; reach posture / envelope is. Caveat: this is the sim RL policy, not the real deploy path (arm-overlay + vendor balancer) — the vendor side is a 5-minute real check (hang a known load on an arm-overlay reach and watch it hold), not a retrain.

Live whole-body stand test (4 operator-driven runs, slack gantry, no damage)

• ✅ move-to-default works — legs ramp to the commanded default stance (damping-first gain ramp), no leg-shotgun, holds at the checkpoint, 4×.
• ✅ safety gates clean — Develop-mode liveness, the abort handshake (window widened 20 → 120 s, abort_confirm_s), SafeStop damp on every abort.
• ⚠️ the default stance carries a forward CoM bias (leans gently forward when loaded; feet flat) — a sim↔real CoM/mass gap (sim asset is the 29-DOF Inspire model; real is 23-DOF + Brainco).
• ❓ policy balance — INCONCLUSIVE. First-ever real-leg engagement balanced ~300 ms then diverged into a limit cycle ("kicking/dancing"). But the feet were under-loaded (the tether bore most of the weight), and run_whole's own docstring warns a suspended policy flails → this is the documented off-distribution failure, not a proven transfer failure. The signature (brief stability → growing oscillation, at ~12 % blend) is a closed-loop instability (control latency / PD-gain / action-smoothing) compounded by under-load. A fair test needs full foot-loading with a trusted fall-arrest — the real blocker (only a standard Unitree gantry is available; it cannot do "fully loaded AND reliable catch" at once).

What this settles

Whole-body rt/lowcmd is the wrong path for the 23-DOF EDU — now empirically, not just on the mode/coexistence grounds of §10.3. Bed-making legs stay on the vendor balancer (arm-overlay rt/arm_sdk + LocoClient); the whole-body policy stays a sim result. Deprioritised (academic — whole-body cannot coexist with walking regardless), not chased — loosening the tether to "fairly" test a thrashing policy is unsafe + symptom-chasing.

Shipped (pushed)

• robotics-connect main e8e5603 — run_whole gained abort_confirm_s (default 120 s) for the remote-console abort handshake; the robot's deploy lib was reconciled to match.
• robots-issue3 issue-3-irl-bed-making 38d09c3 — new rl/pull_stress.py + BED_MAKING_IRL.md §10.5 + media/rl/pull_load_stress.mp4.
• Unitree eng: is a whole-body rt/lowcmd motion policy supported on the 23-DOF G1 EDU? #6 (Unitree) — the whole-body-on-23-DOF question posted with the concrete findings above.

NEXT (priority = the demo's critical path, NOT more whole-body)

1. On hardware: real arm-overlay reach + grip + draw on an actual sheet over the vendor balancer (the bed-making path; still unvalidated on hardware), including the 5-minute pull-load-on-the-balancer check. Much lower risk now — the legs run the Unitree factory walking/balance policy (LocoClient), so no leg-shotgun and no policy thrash; only the arms are overlaid.
2. The upper-body reach-envelope gate (lib/task_gate.py) + walk → reach → decline glue (LocoClient.Move + deterministic arm_fk → rt/arm_sdk).
3. If whole-body is ever revisited: model + randomise action latency, system-ID & soften the PD gains, add an output low-pass filter, filter the joint_vel obs, then retest — but only with a real fall-arrest rig (not available now).

The standing risk to keep in mind

No proper fall-arrest rig is available — only the standard Unitree gantry — so every hardware balance test will be under-loaded/inconclusive. Mitigation for the next run: it is an arm-overlay task with the vendor balancer holding the legs, so the gantry is a backstop, not the primary safety system — substantially less risk than the whole-body runs.

[armwaheed]

Continuation — 2026-06-17: IRL bedsheet pull — the arm-overlay reach MOVED on hardware at the bedside

Resume anchor for the next (fresh) session is this comment. The operator is moving to a laptop in the bedroom to be present during the tests.

Approach (confirmed on hardware)

Reach + grip + draw with the RL-policy arm overlay over the vendor balancer: the legs stay on the factory balance/walk controller, the arms come in via rt/arm_sdk (blended weight), and the same trained v2 policy (policy_v2.pt, transfer_v2_critic) drives the arm — for the reach and, by ramping the hand_target from the reach pose to a retracted pose, for the draw (the policy itself pulls the sheet in while balancing; it was trained with grip-slip load randomisation). The grip is a separate effector (the Brainco fingers were never in the RL action space), sensor-gated.

Shipped / validated this session (staged locally + on the robot; not yet committed)

• Reach-validation eye-verified (gantry): smooth right-arm reach and a clean blend-back to rest — the return-blend that jerked in an earlier session is smooth now. Fall-safe overlay end-to-end.
• Liveness-gate DDS-contention finding + fix + override. The arms-rung precondition (loco GET_FSM_ID liveness) false-negatives under DDS contention: a LocoClient created after the 500 Hz rt/lowstate subscriber (and the policy load) can't match its reply reader → code 3104 timeout even when the balancer is up (a clean-context probe returns code 0, FSM 500). Root-cause fix in g1_robot_io.py (prime the loco client in connect() before the subscribers, then poll) — holds on the gantry, but post-walk contention still beat the 12 s budget, so the pull script got an explicit --assume-balancer-up override, used only after an independent clean-context GET_FSM_ID code 0. It is false-negative-only (can refuse, never wrongly permit); the arms rung is fall-safe regardless.
• New orchestration (examples/isaac_bed_making/rl/deploy/): g1_bed_pull_v1.py — reach → hold-for-grip → policy-driven draw (target ramp) → release; mirrors run_partial's safety (arm_abort, SafeStop damp on any exit, blend in/out, per-tick rate-limit + clamp + abort) with a sentinel handshake (/tmp/arm_reached → /tmp/sheet_gripped → /tmp/draw_done). Plus bed_grip_v1.py — sensor-gated LEFT-hand grip via the Brainco TCP bridge.
• Brainco bridge up: left hand /dev/ttyUSB1 slave 0x7e, touch_ok=True; grip script reads live touch/proximity. Left hand chosen because the right Brainco hand is mechanically damaged.
• Robot walked to the left side of the bed (~0.45 m, square to the near edge). Tuned targets inside the trained command box (x 0.18–0.55, y ±0.40, z −0.16–0.10): reach [0.45, 0.12, −0.13], draw [0.20, 0.10, −0.03]. Dry-run bounded (|a|max 1.69 / 1.97).

Live result

With --assume-balancer-up, the LEFT arm moved on hardware — the RL-policy reach over the live vendor balancer executed (operator eye-confirmed). This validates the arm-overlay pull on hardware at the bedside. Stopped cleanly (SIGTERM → SafeStop DAMPING) before the grip step when a low-battery warning came up.

Open / NEXT

1. Battery SOC telemetry gap: rt/lowstate.power_v reads 0.0 on this firmware → no SOC from telemetry; rely on the robot's own low-battery warning. (Battery being swapped.) Find a BMS query if one exists.
2. After the swap: restart brainco_bridge (serial drops on power-cycle), re-stand the robot in Regular mode at the bedside, re-confirm clean-context GET_FSM_ID code 0.
3. Run the full pull with the eye-verify checkpoint: reach → confirm the hand is on the quilt → grip (sensor-gated) → policy-driven draw → release.
4. Proper gate fix (later): a clean-subprocess liveness check before connect() (deterministic) instead of the override.

Resume specifics

• Robot ssh unitree@192.168.1.119 (pw 123), DDS iface eth0, env unitree_deploy.
• On robot: /home/unitree/bedreach_deploy/{g1_bed_pull_v1.py, bed_grip_v1.py, policy_v2.pt, deploy_contract_v2.json}; RC harness at /home/unitree/robotics-connect-deploy (set ROBOTICS_CONNECT_ROOT to it — it has the fixed gate). Brainco at /home/unitree/robotics-connect/brainco_touch (bridge: g1brainco env).
• Sequence: clean-probe GET_FSM_ID (confirm code 0) → start g1_bed_pull_v1.py --assume-balancer-up (bg, reaches + holds) → eye-verify the reach → start bed_grip_v1.py (grips → draw → release), coordinated by the sentinels.

[armwaheed]

Continuation — 2026-06-17: arm-overlay reach + lift + grip + draw validated on hardware (deterministic-lift → RL hybrid)

Resume anchor for tomorrow's real-bed run.

Today (90% pass on a cot stand-in; real bed tomorrow)

Bed-making runs on the arm-overlay path: the factory balancer holds the legs, the arms run on the rt/arm_sdk overlay, and the v2 RL policy drives the reach. The full pipeline ran end-to-end on hardware — deterministic shoulder-wide lift → RL come-in from shoulder height → grip → draw → exact-reverse retraction, collision-free, with the factory balancer untroubled (torso twist within limits, no compensation during the draw — confirming the vendor carries the load in overlay mode, not the RL policy).

The key architecture call — why the hybrid

The RL reach alone can't do a high, collision-free top-down approach: its trained command box caps the hand below shoulder height, and a single hand-xyz target can't hold a "shoulders rolled wide" configuration — so it came in low and the forearm dragged the mattress (3×). Fix: a deterministic joint-space lift (stay shoulder-wide while extending so the arm clears the edge) hands off to the RL policy for the come-in + reach + draw. Collision-free approach + the trained reach/draw.

Shipped (pushed)

• robotics-connect main f994d2c — (1) smooth mode-aware overlay release: damp_once ramps the rt/arm_sdk weight down on every exit (an un-blended release jerked the arm across); legs/whole-body damp stays fast. (2) loco-liveness gate primed-before-subscribers + polled (a LocoClient created after the 500 Hz rt/lowstate sub times out — code 3104 — even when the balancer is up). (3) lib RobotIO.damp_once release contract + deploy-policy skill gotchas.
• robots-issue3 issue-3-irl-bed-making e8704a0 — rl/deploy/g1_bed_pull_v2.py (the hybrid + exact-reverse retract), bed_grip_v1.py (Brainco TCP grip), bed_deploy_common.py (de-duped ARM_JOINTS/ARM_LIMITS/loaders), the DRAW_TARGET fix, BED_MAKING_IRL.md §1 photos + §10.6. Removed the superseded top-down g1_bed_pull_v1.py.

Open for the real-bed run (tomorrow)

1. Draw direction — now slides laterally toward the headboard (+y at constant x), not inward toward the torso (the cot bug). Validate on the bed.
2. Grip must capture a fold — closing on a flat/under surface read ~0 force; aim at the cover's hanging edge so the fingers wrap it; keep the low touch threshold (soft fabric reads low).
3. Re-tune the come-in target for the real bed geometry (z ≈ mattress-top height; the cot was 8" higher → z=+0.05).
4. Liveness override (--assume-balancer-up) still gated on an independent clean GET_FSM_ID 0; the robust fix (clean-subprocess probe) is tracked.

Resume specifics

• Robot ssh unitree@192.168.1.119 (eth0), env unitree_deploy. Deploy files /home/unitree/bedreach_deploy/ (g1_bed_pull_v2.py, bed_grip_v1.py, bed_deploy_common.py, policy_v2.pt, deploy_contract_v2.json); RC harness /home/unitree/robotics-connect-deploy (set ROBOTICS_CONNECT_ROOT).
• Brainco bridge (for the grip): launch inside the g1brainco conda env (cd /home/unitree/robotics-connect/brainco_touch && python brainco_bridge.py, detached) — more stable in-conda; the serial drops on a power-cycle, so restart after a battery swap.
• Sequence: clean GET_FSM_ID 0 → g1_bed_pull_v2.py --lift-only (verify lift + retract clear) → full g1_bed_pull_v2.py + concurrent bed_grip_v1.py.

[armwaheed]

Continuation — 2026-06-18: real-bed grasp fixed, a human→robot handoff, a calibrated failure detector, and the ask-for-help orchestrator

Resume anchor for the next (fresh) session is this comment.

Today closed the loop on the grasp mechanics and produced the first graceful-degradation path — the robot that knows when it failed and asks a human. That escalation is the Device Connect value proposition, demonstrated end-to-end (dry) and built on validated hardware pieces.

Grasp mechanics (real bed, hardware-validated)

• Palm orientation is ours, not the policy's. The hybrid put the hand on the hanging edge but the RL come-in left the palm not facing it (thumb grazed, fingers closed on nothing). The EDU arm is 5-DOF (wrist roll only), so g1_bed_pull_v2.py --grasp-wrist-roll overrides the wrist roll deterministically (palm-to-edge, -1.5) while the policy keeps the shoulder+elbow reach.
• No pinching thin fabric with a flat hand. bed_grip_v1.py now opposes the thumb into a claw first, settles, then closes the digits into it — one smooth continuous flex (the old stepped set-points caused intermittent flexion).
• Grasp + balance proven; the task is not. With claw + palm right it caught the cover — but caught the fixed mattress cover with the sheet. Pulling the immovable cover twisted the torso ~15° and the factory balancer held it dead steady. A pass for the mechanics; the free-sheet end-to-end pull is still owed. (Smoothing the cover so the autonomous grasp takes only the sheet is the operator mitigation; longer-term it's a vision problem — see NEXT.)

Deterministic human→robot HANDOFF — validated end-to-end (g1_bed_handoff_v1.py)

No RL at all (so a draw from an intermediate present pose is in-distribution by construction): approach via the collision-clearing lift → present palm-up, open claw (wrist_roll -1.74) → human lays the sheet in & signals → claw closes → deterministic draw toward the head → exact-reverse retract. The record-forward / replay-reverse retract also frees the fabric without entangling it (nice bonus).

Pull-failure detector — hardware-calibrated (bed_fail_detect.py)

Free-vs-anchored, so the robot knows it grabbed the cover. Verdict on following error (torque proxy: ~0.18 free / ~0.35 anchored rad) + tau_est (~7 / ~14) — both separate ~2×. Demoted to logged-only after the data disagreed with intuition: base-IMU yaw (~0.1° either way — pelvis IMU misses the upper-torso twist you see) and joint velocity (~0 either way — the deterministic ramp saturates). Writes /tmp/pull_failed past a warm-up. Caveat: thresholds from single free/anchored samples — take a few more reps before trusting --abort-on-stall live.

The orchestrator — the demo state machine (g1_bed_orchestrator.py)

autonomous attempt(s) → detect /tmp/pull_failed → ASK THE HUMAN (banner + /tmp/robot_needs_help + pluggable --help-hook) → deterministic handoff fallback → done. Dry-run-validated both branches. --max-attempts tunable (default 2; one honest attempt may demo better). v1 limit: an empty grab reads as success (the detector catches anchored stall, not grabbed-nothing).

Shipped (pushed)

• armwaheed/robots issue-3-irl-bed-making 38daf4f — rl/deploy/{g1_bed_handoff_v1.py, bed_fail_detect.py, g1_bed_orchestrator.py} new; g1_bed_pull_v2.py + bed_grip_v1.py updated; BED_MAKING_IRL.md §10.7 + media/irl/handoff_draw_complete.jpg.
• Code-debt, deliberately deferred (not refactored blind): the deterministic-overlay scaffolding (blend / rate-limited ramp / reverse-retract / abort poll) is now duplicated and slightly drifted between g1_bed_pull_v2.py and g1_bed_handoff_v1.py → factor into a shared arm-overlay helper with a hardware re-validation.

Generalize to robotics-connect? (kept general — RC must NOT become a bed tool)

Three primitives genuinely graduate (humanoid-general, task-agnostic), as a proposal for sign-off — not done unilaterally:

1. ArmOverlay — safe weight-blended, rate-limited, abort-polled arm overlay with SafeStop damp and record-forward/replay-reverse retract. This is the duplicated scaffolding above; it belongs next to g1_robot_io/safe_stop, parameterized by a per-tick goal function (RL closure or deterministic waypoints).
2. EffortMonitor — the free-vs-stalled manipulation primitive (following-error + tau, warm-up + sustain). Bed-agnostic; any pull/insert/lift task wants it.
3. An agentic "attempt → verify → ask-human → fallback" skill/template — the escalation pattern, parameterized by task callbacks. The pattern is general; the bed orchestrator stays the example.
   Stays in the example (NOT RC): poses, claw/thumb_aux values, wrist-roll tunings, sentinel filenames, the bed narrative.

NEXT (priority)

1. Empty-grab detection — TOP PRIORITY, gates the live demo. Confirm fabric is actually in the hand (touch/vision) before the draw. This is the detector's blind spot and the real on-stage demo killer: the claw closes on nothing, the arm "draws" air, dq/tau look free, and the orchestrator reports success and never asks for help. A confident false-success is worse for the Device Connect story than an honest stall — so close this before the live orchestrator run.
2. Live orchestrator run at the bedside (only dry-run so far) — autonomous attempt → real /tmp/pull_failed → ask-human → handoff. Gated on Clean up Unitree G1 MuJoCo simulator scene and whole-body pickup behavior #1. Keep --abort-on-stall OFF (measure-only + human verify) until the detector has ~5 reps per side — single-sample thresholds with a 0.18-vs-0.35 follow margin will bite on a heavier sheet / stiffer-gain day.
3. Vision (not LiDAR) for autonomous self-positioning — LiDAR can't tell sheet from cover (same height/geometry).
4. Factor ArmOverlay/EffortMonitor into robotics-connect (with hardware re-validation) + wire --help-hook to a real Device Connect notify. (Graduate on a 2nd consumer task, not from this single one — premature abstraction is how RC becomes a bed tool.)
5. Walk the drawn sheet to the headboard (separate task).

Resume specifics

• Robot ssh unitree@192.168.1.119 (eth0), env unitree_deploy. Deploy files /home/unitree/bedreach_deploy/ (now incl. g1_bed_handoff_v1.py, bed_fail_detect.py, g1_bed_orchestrator.py); RC harness /home/unitree/robotics-connect-deploy (set ROBOTICS_CONNECT_ROOT). Brainco bridge in the g1brainco env (serial drops on a power-cycle → restart after a battery swap).
• Gate before any motion: clean-context GET_FSM_ID → code 0 (FSM 500 = standing/Regular), then pass --assume-balancer-up.
• Handoff flow: bed_grip_v1.py --present-claw --arm-reached-file /tmp/grab_go + g1_bed_handoff_v1.py; human places sheet then touch /tmp/grab_go ("close"). Orchestrator: launch g1_bed_orchestrator.py from the activated unitree_deploy env (--dry to rehearse the state machine).