Clarification on right-hand coordinate system and RGB alignment

[CYBER-QAQ]

Hi, thanks for releasing OpenTouch.

I am working with the released right-hand data and got confused about the coordinate definition of the hand-related fields. In particular, it is hard to tell how right_hand_landmarks, right_wrist_pos, right_palm_pos, camera_poses, and /calibration/rgb/* are supposed to relate to each other.

A few concrete things that would really help clarify:

• What coordinate frame are right_hand_landmarks in?
• Are they true 3D camera-space points, normalized image-space coordinates with relative depth, or something else?
• What is the intended way to project right_hand_landmarks back onto the RGB frames?

What made me unsure is that direct reprojection with the stored RGB calibration did not consistently align with the visible hand, while simple 2D axis swaps/rotations sometimes looked closer. Also, camera_poses appears to be identity in the released files, so I was not sure whether it is meant to be used.

Quick places to look:

• README.md
• preprocess/load_data.py

A short note on the exact right-hand coordinate convention would be super helpful for downstream use. Thanks!

[xiezhy6]

Hi, thanks for releasing OpenTouch.

I am working with the released right-hand data and got confused about the coordinate definition of the hand-related fields. In particular, it is hard to tell how right_hand_landmarks, right_wrist_pos, right_palm_pos, camera_poses, and /calibration/rgb/* are supposed to relate to each other.

A few concrete things that would really help clarify:

• What coordinate frame are right_hand_landmarks in?
• Are they true 3D camera-space points, normalized image-space coordinates with relative depth, or something else?
• What is the intended way to project right_hand_landmarks back onto the RGB frames?

What made me unsure is that direct reprojection with the stored RGB calibration did not consistently align with the visible hand, while simple 2D axis swaps/rotations sometimes looked closer. Also, camera_poses appears to be identity in the released files, so I was not sure whether it is meant to be used.

Quick places to look:

• README.md
• preprocess/load_data.py

A short note on the exact right-hand coordinate convention would be super helpful for downstream use. Thanks!

Hey, do you know how to project the provided 3d hand pose onto the RGB video?

[TLNA114514]

Hi, thanks for releasing OpenTouch.
I am working with the released right-hand data and got confused about the coordinate definition of the hand-related fields. In particular, it is hard to tell how right_hand_landmarks, right_wrist_pos, right_palm_pos, camera_poses, and /calibration/rgb/* are supposed to relate to each other.
A few concrete things that would really help clarify:

• What coordinate frame are right_hand_landmarks in?
• Are they true 3D camera-space points, normalized image-space coordinates with relative depth, or something else?
• What is the intended way to project right_hand_landmarks back onto the RGB frames?

What made me unsure is that direct reprojection with the stored RGB calibration did not consistently align with the visible hand, while simple 2D axis swaps/rotations sometimes looked closer. Also, camera_poses appears to be identity in the released files, so I was not sure whether it is meant to be used.
Quick places to look:

• README.md
• preprocess/load_data.py

A short note on the exact right-hand coordinate convention would be super helpful for downstream use. Thanks!

Hey, do you know how to project the provided 3d hand pose onto the RGB video?

Yes, I’m also puzzled by this; I’ve tried many different combinations, but none of them fit. Even an exhaustive search didn't work.