Individual Eye Tracking using Unity OpenXR Plugin

[seattleandrew]

We have a customer that is unable to get individual eye tracking data using XR_ANDROID_eye_tracking xrGetFineTrackingEyesInfoANDROID through the Unity Open XR plugin on a Samsung Galaxy XR device.

In specific we are looking for API support to get:

1.        Individual eye position (instead of combined)
   

2.        Eye openness / blink detection
   

3.        Relative timestamp for each eye-tracking reading
   

Qualcomm has stated that individual eye tracking is supported, but the Open XR library does not seem to allow it. Additional details from Qualcomm:
Data Availability
• Gaze Direction: Supported via Tobii’s eye tracking algorithm.
• Confidence Level / Validity Flag: Available through Tobii API.
• Eye Openness / Blink Indicator: Supported.
• Pupil Dilation: Internally tracked but not currently exposed. This would require a new API extension. Please let us know if you’d prefer this via Unity or OpenXR.
• Relative Timestamp: Provided for each eye-tracking data sample.
• Headset Position Feedback: Accessible globally or per eye through OpenXR APIs.

Technical Notes
• For fine-grained eye pose info, the XR_ANDROID_eye_tracking extension provides xrGetFineTrackingEyesInfoANDROID at the OpenXR layer.
• Display refresh rates can be adjusted using XR_FB_display_refresh_rate.
• Eye tracking performance depends on camera frame rates and synchronization:
o Camera Modes:
• Quarter resolution: 200×200 @ 60fps or 90fps
• Full resolution: 400×400 @ 60fps or 90fps
o Synchronization:
• Hardware sync tolerance: <1µsec
• Software sync tolerance: <10µsec
• For optimal results and smooth gaze-driven interactions, camera frame rates near 90Hz, synchronized with display refresh cycles, are strongly recommended. Lower rates (e.g., 60Hz) can introduce latency and affect user experience.

[SaadTahir7]

Hi @seattleandrew , I am also trying to access the individual eye tracking data but am unable to fetch it. I tried it using the FaceTracking sample to get the eye movements but that are just 0-1 confidence values rather than raw eye tracking gaze data.

[LevanaChen]

Hi, eye tracking integration has been done by Unity OpenXR: Android XR, through Android XR: AR Face feature. Public document can be found by https://docs.unity3d.com/Packages/com.unity.xr.androidxr-openxr@1.2/manual/features/faces.html.

• Eye pose is available via ARFace.leftEye, ARFace.rightEye
• Openness/blink status is retrieved by TryGetAndroidOpenXREyeTrackingStates(ARFace, out AndroidOpenXREyeTrackingStates)

Also noted, here is how Unity expose coarse poses.

For precise eye poses, you can use Eye Gaze Interaction instead. We are also working on a new feature to expose fine eye poses through Google Extensions package with similar API approach.

[SaadTahir7]

Hi @LevanaChen. I tried both solutions and following are my thoughts.

Note: Make sure to use the AndroidXRPermissionUtil.cs class and get Fine + Coarse + Face Tracking permissions

EyeGazeInteraction Solution

I tried the solution with EyeGazeInteraction and it worked when I set the OpenXR > Eye Gaze Interaction Profile and created an Input Action with <EyeGaze>/pose. I was able to get the binocular eye data. Following is the script that moves a Sphere around the screen.

void Update()
{
    if (eyePoseAction != default)
    {
        var gazePose = eyePoseAction.action.ReadValue<UnityEngine.InputSystem.XR.PoseState>();

        Vector3 gazeDirection = gazePose.rotation * Vector3.forward;
        Vector3 indicatorPosition = mainCamera.transform.position + gazeDirection * indicatorDistance;

        gazeIndicator.transform.position = indicatorPosition;
        gazeIndicator.transform.rotation = gazePose.rotation;
    }
}

ARFace Solution

I also tried the solution using the AR Face feature from AndroidXR and it also worked. Following is the script

    [SerializeField] ARFaceManager faceManager;
    [SerializeField] TextMesh debugText;

 void Update()
{
    foreach (var face in faceManager.trackables)
    {
        // Get eye poses
        var leftEyePose = face.leftEye;
        var rightEyePose = face.rightEye;

        debugText.text = $"Left Eye: {leftEyePose.position}, \n {leftEyePose.rotation}";
        debugText.text += "\n" + $"Right Eye: {rightEyePose.position}, \n {rightEyePose.rotation}";

        // Get eye tracking states (flags enum)
        if (face.TryGetAndroidOpenXREyeTrackingStates(out var eyeStates))
        {
            // Check left eye validity
            bool leftEyeValid = eyeStates.HasFlag(AndroidOpenXREyeTrackingStates.LeftEyePoseValid);
            bool leftEyeShut = eyeStates.HasFlag(AndroidOpenXREyeTrackingStates.LeftEyeShut);
            bool leftEyeGazing = eyeStates.HasFlag(AndroidOpenXREyeTrackingStates.LeftEyeGazing);

            // Check right eye validity
            bool rightEyeValid = eyeStates.HasFlag(AndroidOpenXREyeTrackingStates.RightEyePoseValid);
            bool rightEyeShut = eyeStates.HasFlag(AndroidOpenXREyeTrackingStates.RightEyeShut);
            bool rightEyeGazing = eyeStates.HasFlag(AndroidOpenXREyeTrackingStates.RightEyeGazing);

            debugText.text += "\n" + ($"Left Eye - Valid: {leftEyeValid}, Shut: {leftEyeShut}, Gazing: {leftEyeGazing}");
            debugText.text += "\n" + ($"Right Eye - Valid: {rightEyeValid}, Shut: {rightEyeShut}, Gazing: {rightEyeGazing}");
        }
    }
}

Conclusion

It would be awesome to get Eye Gaze Interaction for individual eyes as well throough Android XR Extensions package. That will simplify things a lot. And Thanks A lot 😸

[LevanaChen]

Glad it work on your end and solve the original problem.

I will follow up here once the new feature is available.

[SaadTahir7]

Do we have a way to retrieve Eye Images textures from the Android XR API? Or is it also a new feature to be added in the near future?

[jimmy-hello-world]

@LevanaChen Can you share expected timeline when the original issue might be resolved. I'm unable to retrieve the following.

• Individual Eye Gaze Direction
• Confidence Level / Validity Flag
• Eye Openness / Blink Indicator
• Pupil Dilation
• Relative Timestamp: Provided for each eye-tracking data sample

[aarelovich]

@LevanaChen Hi. I'm working on an application that requires fine eye tracking as the data gathered needs to be non-filtered and high quality. Usint the suggested ARFrace I was able to at least get two separate data streams (one for each eye) however it's quality is low. I'm thinking that this might be because of using COARSE permission and not the FINE permision? I tried to gather a single data point each frame. I did this at 72 Hz as measured by my unity app. I.E. 72 frames per second. However, when I printed out the information (raw data as retrieved) it looked like this:

The image shows that we get a different data points that are the same, with some noise, between 3 to 4 times in a row and then we jump to another point. In our experience, eye trackers don't usuailly provide the raw data like this. They are ususually softer transitions. Maybe this could be addressed? It would also seem like the EyeGazeInteraction would be the place to put something like this. But I'm not sure.

[LevanaChen]

@LevanaChen Can you share expected timeline when the original issue might be resolved. I'm unable to retrieve the following.

• Individual Eye Gaze Direction • Confidence Level / Validity Flag • Eye Openness / Blink Indicator • Pupil Dilation • Relative Timestamp: Provided for each eye-tracking data sample

The original issue is solved by #15 (comment), the new feature is about the script API to access fine eye poses.

[LevanaChen]

@LevanaChen Hi. I'm working on an application that requires fine eye tracking as the data gathered needs to be non-filtered and high quality. Usint the suggested ARFrace I was able to at least get two separate data streams (one for each eye) however it's quality is low. I'm thinking that this might be because of using COARSE permission and not the FINE permision? I tried to gather a single data point each frame. I did this at 72 Hz as measured by my unity app. I.E. 72 frames per second. However, when I printed out the information (raw data as retrieved) it looked like this:

The image shows that we get a different data points that are the same, with some noise, between 3 to 4 times in a row and then we jump to another point. In our experience, eye trackers don't usuailly provide the raw data like this. They are ususually softer transitions. Maybe this could be addressed? It would also seem like the EyeGazeInteraction would be the place to put something like this. But I'm not sure.

The suggestion of Android XR: AR Face feature is for coarse eye poses, precise eye poses currently retrieved by Eye Gaze Interaction.

[aarelovich]

@LevanaChen Than you fr your response, but the original issue was, in a way, raised on our behalf. The problem with the eye gaze interaction is that it is currently not possible (at least to my understanding) to get the separa eye data. You can ony get Combined Gaze. If I'm mistaken about this, I would like to see an example where this is being done. I'm also not sure that you can get a timestamp on the data or any of the other information such as eye openess or pupil dilation.