Have you ever thought about add fingerpose detect in webXR?

[Simon1059770342]

Hi Teacher Nik,

Recently, An issue came out of my mind, which is make object in WebXR change different behaviours according to my finger pose.

But

From immersive github, folks told us that “WebXR Device API does not provide a way to grant raw camera access to the sites.”.
I am not sure if feeding video into model.estimateHands is the only way to detect finger pose,

const predictions = await model.estimateHands(video);
just wondering if there is any other way to detect fingerpose in WebXR to you?

MANY THANKS FOR YOUR ASSISTANCE!

[NikLever]

Hi Simon https://threejs.org/examples/?q=webxr#webxr_vr_handinput_profiles <https://threejs.org/examples/?q=webxr#webxr_vr_handinput_profiles> Cheers Nik Nicholas Lever, 28 Platt Lane, Dobcross, Oldham, OL3 5QD t: 0797 370 2959 e: ***@***.***

…

On 16 Apr 2022, at 14:52, Simon1059770342 ***@***.***> wrote: Hi Teacher Nik, Recently, An issue came out of my mind, which is make object in WebXR change different behaviours according to my finger pose. But From immersive github, folks told us that “WebXR Device API does not provide a way to grant raw camera access to the sites.” <https://github.com/bialpio/webxr-raw-camera-access/blob/59b45bf56d72251493b10e4f86cd2a4bdb3966d0/explainer.md>. I am not sure if feeding video into model.estimateHands is the only way to detect finger pose, const predictions = await model.estimateHands(video); just wondering if there is any other way to detect fingerpose in WebXR to you? MANY THANKS FOR YOUR ASSISTANCE! — Reply to this email directly, view it on GitHub <#3>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALGSZ63DXX72ZVBVNWEVJULVFLA3JANCNFSM5TSNC5AQ>. You are receiving this because you are subscribed to this thread.

[Simon1059770342]

THX Nik!

I reckon that I didn't explain my purpose clear.

The vr sample you share with me is different from my purpose, which is detect finger pose in WebXR AR sample on Android device using Chrome instead of Headset.

And I also gave it a shot on VR hand input sample on my Android Chrome, it seems no work.

Then I tried getFrame() from WebXR API, fed the reslut as a video into model.estimateHands(video).

but still no work.

So just wondering if you have any suggestion on this.

THX for you patience and help!

[NikLever]

Hi Simon WebXR hand input docs https://www.w3.org/TR/webxr-hand-input-1/ <https://www.w3.org/TR/webxr-hand-input-1/> https://github.com/immersive-web/webxr-hand-input/blob/main/explainer.md <https://github.com/immersive-web/webxr-hand-input/blob/main/explainer.md> I haven’t tried it in an AR example. Best resource I can find is an A-Frame example https://aframe.io/examples/showcase/handtracking/ <https://aframe.io/examples/showcase/handtracking/> A-Frame uses ThreeJS as the rendering library. Cheers Nik Nicholas Lever, 28 Platt Lane, Dobcross, Oldham, OL3 5QD t: 0797 370 2959 e: ***@***.***

…

On 17 Apr 2022, at 07:43, Simon1059770342 ***@***.***> wrote: THX Nik! I reckon that I didn't explain my purpose clear. The vr sample you share with me is different from my purpose, which is detect finger pose in WebXR AR sample on Android device using Chrome instead of Headset. And I also gave it a shot on VR hand input sample on my Android Chrome, it seems no work. Then I tried getFrame() from WebXR API, fed the reslut as a video into model.estimateHands(video). but still no work. So just wondering if you have any suggestion on this. THX for you patience and help! — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALGSZ67RULCREYWAGKY5IJDVFOXK3ANCNFSM5TSNC5AQ>. You are receiving this because you commented.

[Simon1059770342]

THX Nik!
I gave the sample a shot, found when I click AR button, screen showed VR scene after stay in AR scene for a couple of sec.

FYI my purpose is to make a virtual object in AR scene chang different behaviours by recognizing my finger pose, so this could be slightly different from hand input in WebXR hand input docs.

[Simon1059770342]

WebXR hand input vs. Fingerpose detection input

For the best of my knowledge,

1. Hand input comes from controller, like HTC Vive handheld device, or phone screen.

2. Fingerpose detection input comes out of camera frame.