RGB image quality

[wetoo-cando]

Hi @t-34400

The RGB images saved by the yuv-to-rgb converter are "highres" (1280x960), but low quality. See example below.

I checked the conversion scripts. The cv2.cvtColor step is the only one that is not strictly lossless, but at the same time usually considered quite high-quality.

The YUV420_888 itself has chroma subsampling. If that is the source of this issue, and is the best we can get from the Android camera2 API, I guess we can't do much about it?

Any thoughts on this?

[wetoo-cando]

// The only pixel format supported atm
private const int YUV_420_888 = 0x00000023;

https://github.com/xrdevrob/QuestCameraKit/blob/08b7c940c8916c81e65efb557c19f010b3bf6d6c/Unity-QuestVisionKit/Assets/PassthroughCameraApiSamples/PassthroughCamera/Scripts/PassthroughCameraUtils.cs#L18

[t-34400]

Thanks for the report, and sorry for the late reply.

I think your assessment regarding YUV_420_888 and chroma subsampling is spot on. That format does inherently reduce color detail, but in this case, I suspect the main limitation might actually be the camera hardware itself.

Even though the resolution is reported as 1280x960, the sensor and lens may not be capable of delivering the level of detail we would expect from that size. The image you shared shows fairly soft edges and a lack of fine structure, which makes me think the camera’s optical or sensor quality is simply not very high — especially for color accuracy and sharpness.

So in my view, the RGB image quality issue isn’t just about the YUV-to-RGB conversion, but also about the baseline quality of the captured image.

Of course, this is just a guess — if you’ve observed otherwise or have test cases to compare, I’d be interested to hear more.

Thanks again for the insight.

[wetoo-cando]

Thanks @t-34400

Not sure if the sensor itself is the issue.

I captured another dataset (in a lab env), and this time, while recording with the headset, paid special attention to the visual quality of the passthrough shown in the QRC app. The image is pretty sharp; the issues with the soft edges we see in the picture above are not present.

Why should it be better in the passthrough, and not so in the recording?

I'll try to go to the same place as in the above picture and check the passthrough quality just to be sure.

[t-34400]

Hi @wetoo-cando,

Thanks for the follow-up. I think your observation is very likely correct.

After digging into related reports, it seems that the quality degradation is not due to hardware (e.g. sensor or lens limitations), but rather a problem in the preprocessing pipeline that takes place between the raw sensor feed and what gets exposed via the Camera2 API as a virtual camera.

For example, issues like the following support this:

• Image corruption meta-quest/Meta-Passthrough-Camera-API-Samples#1
• Image quality... oculus-samples/Unity-PassthroughCameraApiSamples#13

Both point out that jagged edges and loss of clarity are visible in PCA-acquired frames but not in the passthrough as shown by the OS. That strongly suggests that the system-level passthrough goes through a higher-quality undistortion and enhancement stage, which is not used (or not applied at the same quality) for the PCA virtual camera stream. So even if we're capturing at 1280x960 resolution, the actual input to PCA might already be distorted or resampled and not in a particularly high-quality way.

Thanks again for raising this! it's very helpful to see this discussed.

[t-34400]

Hi @wetoo-cando,

Just FYI: I also confirmed that the jagged edges and overall image distortion seem to have been improved after updating to the latest firmware (v78.1030).

This comment in a related issue mentions that the problem has likely been fixed by a driver/firmware update. From my side, the RGB image quality looks much better now.

Might be worth checking on your end as well!

[wetoo-cando]

Thanks @t-34400
I'll take a look soon.