How do we want to specify formats?

[gfxstrand]

There are a few possibilites here. (Thanks to @robclark for the write-up. I'm mostly just moving it to github.):

• fourcc:
  • Simple and already used in many places (drm/kms, egl dmabuf extensions, v4l, etc)
  • Should be sufficient to describe anything that can be shared between multiple different gbm2_backends (ie. between scanout, gl/vk, camera, vidc, etc)
• Khronos dataformat spec:
  • is more expressive than fourcc.. fourcc is not sufficient for describing “advanced” textures (mipmap levels + array and/or 3d, for example)..
  • This case is less likely to be involved in cross-device sharing, but could be used in cross-api sharing (ie. interop between $vendor’s gl and vk driver)
  • note that the interop case includes sharing between different versions of $vendor’s gl/vk driver (such as different version in a container/chroot) where layout of layers/levels might have changed)?

Straw man suggestions:

1. Use fourcc plus additional capabilities/constraints to fill in the gaps for gl/vk interop?
2. Use dataformat plus helper lib which can be used by backend to map from
3. Use fourcc together with (possibly device-specific) metadata. Types of metadata supported would be handled by constraints

[gfxstrand]

From @cubanismo:

Good summary Rob. I personally think where dataformat will shine is in advanced use cases, probably many of which we aren’t seeing or haven’t thought of yet because the ability to share complicated surface formats isn’t exposed anywhere yet. The dataformat spec was designed to fill in the gaps of fourcc. I feel safer going with the dataformat spec as the basis of API because of that. I agree some helper libs to actually make dataformat usable as an API interface is highly desirable, and I’ve discussed that a bit with Andrew Garrard, the spec author. I believe we’re all in agreement that the data format spec by itself is not palatable for direct use in an API without some helper libs or headers defining some conversions and further data structures.

[gfxstrand]

From @robclark:

Well, it looks like dataformat is serializable which is my big concern (ie. keep in mind that not all proc’s can open all devices) so if we get some helper libs to make dataformat easier to deal with for the common cases, then sounds good to me.

[gfxstrand]

From Albert Freeman:

What about HDR and beyond (e.g. full cie xyz/luv/[or similar])?

[gfxstrand]

Personally, I'd rather avoid the data format specification as it's way more verbose than we are likely to need. However, if we're going to have a centralized allocator, we need some way to tell it the format so that it can perform the allocation. For most things that can be shared between vendors, fourcc is sufficient. Where it starts to run out is if I have some crazy non-scanout format that I want to share between Vulkan and CL for instance. If we just had a vkGiveMeWhatINeedToShoveIntoGBM2 call, then we could allow some of these things to be vendor-specific.

[cubanismo]

What's wrong with verbosity if you have a function call like:

khrDataFormat = GetFormatFromVkFormat(VK_FORMAT_R8G8B8A8_UNORM);

and similar for other APIs? That seems much better to me than falling back to opaque formats when we go beyond what fourcc can do. Most apps won't care about the fancy stuff the data format spec can do, and those apps shouldn't need to think about it. They can use helper functions and tables, which we could probably include in the same package with little effort. But for the 1% that do care, it would be great to have it available.

Driver vendors could use inverse helpers and ignore formats with bits they don't support. Maybe some driver vendors don't want to worry about anything advanced, but with HDR coming, perhaps some will soon care about things like defining an arbitrary white point for their YUV surface via its format.

[robclark]

I guess my biggest concern is dataformat seems like way overkill for something like a v4l camera or video decoder. If we had some helpers that can map generic fourcc to dataformat and back (and return an error if passed a dataformat that doesn't map to a normal fourcc, perhaps) maybe it could work.

Maybe it could work the other way by using vendor capabilities to pass the extra information, since anything "fancy" would have to be supported by both sides of the sharing.

Unrelated, our header format is almost the same but not exactly as the dataformat descriptor block header.

[gfxstrand]

@cubanismo For one, thing the line of code you cited doesn't work because Khronos data format specifiers are variable-length. This makes them way more painful to work with than an emum. Also, at least when I read it, it looked like VkFormat may not be enough information to actually construct a full data format (I may not understand it quite right though). For YUV, at least, the data format includes a bunch of information that seems to more-or-less equate to tiling and/or plane locations. That seems like something we may want decoupled.

[cubanismo]

@jekstrand Yes, in general the blocks are variable-sized, but for a fixed set of formats, the function could return pointers to static const data built at compile time. Andrew at one point had a header that generated such a table for GL or Vulkan as an example I believe.

When converting from a less-detailed format description to a more-detailed one, yes there's not exact 1-1 mapping. However, generally when going from less->more, there are some reasonable defaults that can be chosen. Ideally the Vulkan working group would create a definitive mapping and publish it. Going from more->less (data format -> fourcc) would be lossy, but that's inevitable. I don't view the fact that dataformat is more expressive as a negative. I view it as future-proofing.

I believe the additional YUV data you're referring to is sample positions and which plane a given component lives in? It's unclear whether we want to allocate planar surfaces as one allocation or two in allocator library terms, but I don't think there's anything in the data format descriptors that don't belong in a format, and I think fourcc also describes some of the same properties you're concerned about.

[gfxstrand]

@cubanismo If it's just a pointer to a const table, that's probably not substantially worse than the rest of the variable-sized blobs we've been talking about. I'm still a bit conflicted on whether or not the extra information is future-proofing or just adding confusion and pain. But my mind isn't 100% made-up yet.

[fluppeteer]

@jekstrand Yes, the intent is that most APIs will effectively have an enum list of formats that they use, which can be used as an index into an array of pointers to data format descriptors. What you'd normally pass around is the enum - data format descriptors are intended to be short enough for fast processing, but they can't be sufficiently general without being substantially less efficient than a simple enum. I do have a version of this mapping for Vulkan and partly GL; I'm revising it for correctness since some formats changed after it appeared. It will also definitely be only an example mapping: Vulkan and GL make no statements about whether their R,G and B channels contain red, green and blue data or something entirely different. To talk to an external API and make sense, someone probably needs to have said something about this interpretation; to be useful, mappings probably need to describe the minimum common case. Some APIs might like to fill in their standard formats and allow the table to be extended by users importing external content or specifying their own more detailed approaches, but it was never the intent that hardware has to directly interpret the data format descriptor, and ideally that software should be doing so only infrequently.

The idea of having these mappings available is that you can then, for example, use them to determine how you want to do inter-API mapping. There are legitimate reasons that the user might consider all 32-bit (total) formats to be equivalent, or to believe that all RGB formats are equivalent irrespective of size. There are certainly APIs which will want to ignore the colour space aspect of the format. Rather than saying two things have to be the same, I think it's better to define what we mean by "the same".

The data format specification is intended to be FourCC on steroids. Two images may be the same "format" but store planes in different places, have different strides or be different sizes - as such that's assumed to be "outside the format". Interpreting a texel block (repeating pixel pattern), however, is part of the format - otherwise block compression wouldn't be considered part of the format. The data format spec therefore does specifically include sample position information, which can be used both for YUV chroma channel sample placement and for things like Bayer samples.

My experience is that image metadata tends to go missing when formats are transferred between APIs, especially if there's a helper library in between that doesn't know about the metadata. There are limits to what can sensibly be embedded in a general-purpose data format descriptor (the layout is extensible, so you could include a full ICC profile, but that's unlikely to be the general case) - but a best effort has been made to pack in as much as possible. There's also an effort to pack in things that you probably should have specified but might not have done - because there are a lot of home-rolled attempts to define this kind of thing in a way that requires incompatible extension. FourCC's belated addition of BT.709 and lack of discussion of full and narrow range YUV content is an example.

The data format spec currently needs updating to support the newest HDR TV standards. I'm on it. :-) It's likely that some additional metadata will always be appropriate in some environments - particularly some HDR video formats have frame-specific metadata which is probably not best encoded in the "format", although it could be held in an extension block if it was really decided that was the best approach. In most environments I wouldn't expect it to be sensible for the "format" to change this frequently, though.

[gfxstrand]

@fluppeteer It's entirely possible that my desire for simplification comes from not having had to deal with the details of TV standards. I'm not going to wish I had :-) but I'll grant that my knowledge is incomplete.

[robclark]

@jekstrand / @cubanismo / @fluppeteer,

So some random thought, and something I mentioned in d3c96b6 but it seems like capability blocks are kinda the same thing as dataformat descriptors.. ie. if I understand dataformat properly it should describe all the combinations of things we are talking about with capability blocks/descriptors with respect to possible tiled/compressed/etc layout..

So maybe the question about fourcc vs dataformat is looking at things the wrong way around. Maybe user asks for a fourcc in assertion_t and gets back a capability set consisting of dataformat descriptors plus corresponding constraint set?

I'm still not sure how user would ask for a particular format + 3d-array-mipmap.. maybe fourcc isn't the right input in assertion_t (or at least not without some additional parameters) but it does seem a bit like dataformat is the output of what the user asks for and not the input.. otherwise I'm not really sure how the user would know what dataformat to ask for.

[gfxstrand]

@robclark, Yes, they do look similar but I'm not convinced they're the same. We could, in theory, base our final generated metadata on the data format spec, but I don't think capabilities is a good match. Note: I'm not suggesting that we do. I would rather have something Intel-specific with an Intel format enum when it comes to the final serializable image metadata blob.

[robclark]

@jekstrand fwiw, I'm assuming we don't have to exchange this over the wire every frame so the conversion between dataformat and intel format enum is inconsequential. Admittedly I'm still at the "open mind" phase with this so just exploring other possibilities but I don't think "what enum we plug into the sampler state" is too important for how the "liballoc" (or whatever it ends up being called) API works..

My rough thoughts leading to that idea was mostly "dataformat looks a lot like constraint block/descriptor thing" and "dataformat seems too specific for a client to ask for" (ie. more appropriate as a driver response to what client asks for, or as a way to communicate description of some existing buffer between different APIs that client is using)

(And PS, I've only started digging more into dataformat today so could be reading more into how it is intended to be used than I should.. but my impression so far is that it is too precise for what a client would want to ask for to create a new buffer)

[gfxstrand]

@robclark, just to be clear, I didn't mean that we shouldn't use data format as part of the "what to send to the allocator" package. I just meant that we may not want so shove everything in as data format extensions. In the end, that distinction may be mute as capability and constraint descriptions are looking a lot like data formats.