Issue running the VQ-BeT policies

[hhsiao24]

Hi, I was trying to run the robot utility models for a research project but I was running into some issues getting the VQ-BeT model to work. I made the mount for the iPhone 14 Pro and the grippers, and followed the instructions on robotutilitymodels.com/docs/ for running the policy, but it isn't able to perform any of the tasks properly.

For context, we are using the Stretch RE2 with a DexWrist 2, Record3D app as specified, and set the angle to roughly 75º. The terminal commands to run the robot server and the policy both worked fine with no errors. However, when we take a policy step in the UI after updating the parameters and homing the robot, the Stretch RE2 robot consistently moves to the left instead of picking up the object. We used paper bags and a tissue box like the paper used, but the robot wasn't able to successfully grab the object in any of those cases.

Here's what we did:

• When we originally ran the script, the robot wrist was placed at a 90º angle to the arm and we mounted the phone where the two metal strips come together (see picture 1).
• Running the policy then resulted in the issue where the robot would consistently move to the left.
• After going through the paper again we noticed that the wrist in the paper was oriented away from the arm so it is in line with the object, so we went into the code and set the wrist yaw so it is oriented like the paper (see picture 2).
• We faced the same issue. We then decided to mount the phone further back on the wrist so it is above the motor powering the wrist (see picture 2) and faced the same issue.

We also added some additional photos and video for reference.

Picture 1: Position of the phone mount on the robot wrist

Picture 2: Side view of the new gripper orientation so it is in line with the wrist motor, and the phone mount is further back

Picture 3: Angle showing the phone running the Record3D app and the arm and wrist oriented in the same direction towards the target object (paper bag)

Picture 4: The angle of the phone is set to roughly 75º

Picture 5: Entire setup of the robot, phone, and paper bag

Picture 6: The mount we have and how it is positioned on the wrist

IMG_3933.mp4

We're not sure why this is happening, and would appreciate any guidance on how we might get this working.

Also, on the UI for running the policy, is there any way to make it so we don't have to keep pressing the "Take a policy step" button. In the demonstration videos, it seems like they run seamlessly in one press.

Thank you in advance! 🦾🤖✨

[haritheja-e]

Hi Hannah,

Thank you for providing such detailed documentation of things you have tried! The configuration you described generally looks correct, but there are a couple things that may be causing issues.

• I think the main problem is likely a distribution shift in the observation space. The “pick up” policies were unfortunately not designed for the RE2 gripper. Our “tissue pick up,” “bag pick up,” and “object reorientation” policies were trained using a gripper corresponding to the Stretch SE3, specifically the DexWrist3, which was wider and looks different from the first-person POV.
• In contrast, the “door opening” and “drawer opening” policies were trained with a gripper corresponding to the Stretch RE1, which I believe is nearly identical to the DexWrist2.
• You may be able to see the difference by comparing the POV in the sample data on our website: https://robotutilitymodels.com/#dataset.
• Unfortunately, we weren’t able to achieve reliable cross-embodiment policy transfer, so the “pick up” policies likely won’t work with earlier DexWrist versions.
• Another minor detail is that the iPhone 14 Pro has a slightly shifted camera compared to the iPhone 13 Pro (which we used for data collection and deployment). It should still function, but performance may degrade slightly unless the mount is adjusted to better center the camera.
• If possible, it would be helpful for us to look at the first-person POV from the UI to help confirm these assumptions.
• Regarding our higher frequency continuous control, we didn’t include it when cleaning up our codebase as it required significantly more supervision and was prone to damage to the wrist/robot in cases where the policy may fail. We do however have a variant in the gpt_retry branch controller where the policy would automatically execute the next action when the current one finishes, but this also requires some supervision and doesn’t provide much faster execution. In practice, during our evaluations, we had used the step-by-step controller.

Sorry for the trouble, and I hope this helps clarify things. Please feel free to follow up with any other questions or concerns!