Low action loss but failed real-world pick on Franka Panda with Pi05 LoRA: possible causes?

[HongyuanChen]

I am currently using a Franka Panda robotic arm with a wrist camera and a third-view camera fixed at a constant position. I want to perform a pick-and-place task, but the object to be picked may move within the plane and also have some rotation. Since the end-effector gripper needs to grasp the relatively flat two ends of the object, a certain degree of alignment is required. However, the translation range is actually not very large, likely within about 8 cm × 8 cm, and the rotation is limited to between -60° and 60°.

The robot collects real-time joint angles (q), gripper open/close states, and data from the two cameras through motion commands. All data are then synchronized to the 30 Hz frame rate of the wrist camera. The action is 8-dimensional: the first 7 dimensions correspond to the interpolated joint increment (dq), defined as the next frame minus the current frame, and the 8th dimension represents the absolute gripper open/close state (0/1).

A total of 138 episodes were collected, among which 120 were used as the training set and 18 as the validation set. The test set used the same norm_stats.json as the training set. I used Pi05 LoRA with use_quantile_norm = False, batch_size = 16, and step = 30,000. The recorded action loss is as follows:

step | train_loss | val_loss
500 | 0.1108 | 0.1088
10k | 0.0498 | 0.0596
20k | 0.0385 | 0.0370
30k | 0.0312 | 0.0270

However, when I deploy the policy, I first reconstruct the predicted action (dq) into (q), then send it to the Franka robot, and use forward kinematics together with a Cartesian impedance controller to drive the robot. I found that the robot cannot even accomplish the initial pick, and there is a very large discrepancy from the expected behavior. Why might this happen?

To verify that the Cartesian impedance controller itself is not introducing a large tracking error, I also evaluated the error between the commanded motion and the actually executed motion. Specifically, I used the recorded robot end-effector pose, and also estimated the gripper tip pose by assuming a 0.3 m distance from the gripper tip to the end flange. Then I computed the error between the commanded pose and the executed pose for both the robot end-effector and the gripper tip.

For the end-effector position error, the RMSE norm error is 0.0075 m.
For the gripper tip position error, the RMSE norm error is 0.0085 m.

These tracking errors are quite small and are on the same order as the positional error observed in the experiment, so it seems unlikely that the Cartesian impedance controller is the main cause of the failure. Therefore, I would like to ask why this might be happening.

failTrial.video-converter.com.mp4

[HongyuanChen]

I would like to add some additional details about the deployment pipeline, in case the issue is related to how the predicted actions are interpreted and executed on the real robot.

The deployment setting uses action_horizon = 20. A file of ros-policy-bridge connects the policy environment with ros to transfer data.

However, these predicted actions are not executed directly as raw dq. After receiving the current measured joint state (q_snap), the bridge accumulates the 20 predicted dq values into a sequence of 20 future absolute joint targets. In other words, the actual commands sent to the controller are reconstructed absolute joint positions rather than the original predicted joint increments:

Q_new[0] = q_snap + dq[0]
Q_new[1] = Q_new[0] + dq[1]
...
Q_new[i] = Q_new[i-1] + dq[i]

In addition, the ros-policy-bridge does not directly replace the old action chunk with the new one when asynchronous inference is used. Instead, if a new chunk arrives before the previous one has been fully executed, the bridge merges the old and new trajectories using a freeze + blend + replace strategy. Around the transition region, it further applies local minimum-jerk smoothing in order to reduce jerk and avoid abrupt switching. Finally, at each 30 Hz control tick, one absolute joint target is sent to the low-level controller.