Need advice on creating a vp9 suite specific to Apple VideoToolbox in GStreamer

[DominiqueLrx]

I've recently added vp9 support to GStreamer's vtdec element, which uses Apple's VideoToolbox framework.

To help validating my changes were working, I ran the fluster vp9 suites by adding a new decoder: GStreamerVTVP9Gst10Decoder.patch

My findings are described in the MR overview, but essentially VideoToolbox's vp9 implementation doesn't support the whole vp9 feature set. Therefore I had to create a vp9 suite that omits test vectors:

• for resolution below 64x64
• for svc when more than one layer is used
• where the resolution changes over time
• using other than 4:2:0 chroma sampling

Also, VideoToolbox vp9 supports 10-bits, but not with bit-perfect results so the one 10-bits test that would be applicable fails because of dissimilar results.

So: what is the right way to have a suite tailored to vtdec/vp9 so we can monitor its correctness over time? Here are possibilities, I'd like to know what's desirable (happy to do it, possibly with some help as I'm new to fluster):

• Introduce a new suite: VP9-TEST-VECTORS-BASIC.json.
• Do as above (new suite with just basic tests) but since it's 50% redundant with VP9-TEST-VECTORS.json, I could remove redundant tests from VP9-TEST-VECTORS and maybe rename it VP9-TEST-VECTORS-ADVANCED.
• Introduce a mechanism to avoid running test vectors known to be broken for the running decoder. Essentially a json list of test vectors known to not be supported. Results would now show something like Success/Fail/Error/Ignored

I can imagine the above could break a lot of existing validation tooling, which is why I first wanted a discussion before naively opening a MR.