How [de]centralized do we want to be?

[gfxstrand]

One of the resounding things that came out of the discussions at XDC was the mobile guys saying that they don't want to have to do contiguous memory allocation in their GPU driver. Instead, they wanted a central allocator (such as ION) that would understand all the constraints and be able to allocate an image. Both NVIDIA and Intel badly wanted to do the allocation inside their driver stack in the case of single-vendor interactions so that they could handle all their "magic".

Two options were discussed for resolving this:

• Require the client to go around to each of the drivers until it finds one that can allocate its image
• Have a central allocator (similar to gralloc) that understands everything

In the end, we settled on the second option (central allocator) with the understanding that it would have a bunch of pluggable back-ends and would internally walk around to the different back-ends until it found one that could allocate.

This answers the question for the final allocation step, but there are several more steps involved:

• Convert GL/Vulkan/CL/v4l image specification to allocator spec (VkFormat -> allocator format, etc.)
• Get allocation capabilities
• Intersect capabilities and union constraints
• Allocate memory (central)
• Lay out the image (highly vendor-specific except for the 2D case)

[gfxstrand]

My preference would be to keep the "get allocation capabilities" portion decentralized so each driver has an entrypoint for it. Why? Because it means that in the common case of an app and a compositor, the only thing that absolutely has to monkey with the allocator is the compositor. The client can just send capabilities and recieve ready-to-import images.

[robclark]

@jekstrand fyi, have a look at https://github.com/robclark/allocator/blob/master/USAGE.md .. I still need to update it for the capability_set_t stuff (ie. keep constraints coupled to capabilities), but it does at least show what I had in mind for the "go around and ask each device to allocate" idea. Where one of the devices could possibly be ION for the weird mobile use-cases. But I think this approach would let us keep allocation in the GPU driver on more desktop(ish) systems.

Basically it is the "pluggable backends" idea, where the pluggable backends are in userspace.

(fyi, also some comments at #3)

[Benjamin-Gaignard]

How do you know which devices are involved in the capabilities negotiations ?
Can you describe how this could work for example in GStreamer ?

[robclark]

@Benjamin-Gaignard I think that is a question for GStreamer dev's.. I was planning to ask Wim to have a look, but probably not until the concept is a bit more concrete. At least in the gst case, everything is in a single process. Maybe a more interesting question is how this ends up working across processes (for example wayland, plus pinos to separate app in flatpack/snap/etc container from helper process with access to camera).

Short version: some collection of userspace processes are doing something.. so they must know what they are doing. The extra APIs and protocols that need to be developed are left as an exercise for the reader for now ;-)

[Benjamin-Gaignard]

With dma-buf we already have a cross processes and cross devices (at least in v4l2, drm) mechanism. We could also know which devices are attached to a buffer.
@robclark I don't learn you anything since you have actively participated to dma-buf creation :-)

dma-buf support in gstreamer and wayland is progressing and widely used in Android.
With defered allocation (map_attachment is called after attach) we can find the best way to allocate a buffer without adding to new protocols.

[robclark]

@Benjamin-Gaignard well, the complication is when two devices can exchange vendor compressed/tiled formats, possibly with associated pitch constraints, etc.. that sort of information is not available in the kernel.

[Benjamin-Gaignard]

I think we need to add a list of memory constraint in device structure with get/set functions.

Those constraints could be implemented like metadata are in GStreamer:
https://cgit.freedesktop.org/gstreamer/gstreamer/tree/gst/gstmeta.h#n180

struct constraints {
struct list_head list;
int type; / identifier of the constraint _/
int (_init)(struct constraints _c);
int (_free)(struct constraints c);
struct constraints (merge)(struct constraints *a, struct constraints *b); / merge constraints a and b into one new structure. This function depend of the data type */
}

We can to get the constraints of all devices attached to dma-buf and merge them before trying to find a compatible allocator.

An example of constraint could be the number of possible entries in sg_table for a buffer.

struct segment {
struct constraint;
int nb_segment;
}
with a merge function like this one (pseudo C):
struct constaints * segment_merge(a,b)
{
struct constraint *c = new constraint;

c->nb_segment = min(a->nb_segments, b->nbsegments);
return c;
}

On my platform, in camera preview use case, I have a USB camera with IOMMU so let says nb_segment is huge (0xFFFF) but my display need contiguous memory so nb_segments = 1.
After merging those two constraint I will get a constraint where nb_segment = 1 and be able to select an allocator that fit with that.

We can do the same with memory range, titling, etc...

[robclark]

@Benjamin-Gaignard I did suggest a fairly simplified version of that quite some time back to deal with the simple "do I need contiguous or not" case. But there are more complex cases for constraints about which the knowledge does not exist in the kernel. Like pitch/stride constraints for various formats.

[Benjamin-Gaignard]

@robclark in GStreamer or Android pitch/stride/alignment are already handled per formats.
GStreamer do that in caps negotiation while Android/Gralloc hard code it for each platform, I don't think we need to reinvent that, right ?
Constraints in kernel should to help to solve where (which memory bank, memory region, contiguous or not) allocation should be done.

[lynxeye-dev]

@Benjamin-Gaignard At least if you are dealing with dynamic GStreamer pipelines you don't have all the devices that may take part in the buffer sharing attached up front to the dma-buf. Which means if you can't afford to drop the frame (if the source isn't going to produce a new one for a while) you need to actively resolve the buffer to something the newly attached sink can digest. This might be a complex operation (GPU blit or whatever) that really shouldn't be the business of the kernel, but the userspace driving this hardware.

[lynxeye-dev]

@jekstrand I think allocation should really be decentral as well. After caps/constraints have been intersected you go around asking the backends who is able to allocate with this intersection. This means we might need to smart up some of the underlying device drivers to understand foreign constraints, but that's really more an implementation detail than anything more serious.

From a kernel driver writers PoV it would be nice to have some central kernel internal thing to extend your drivers abilities to allocate memory with foreign constraints. But that part should certainly not be exposed to userspace, other than the GPU driver claiming the ability to allocate contig memory for example.

[gfxstrand]

@lynxeye-dev, Your comment just made me realize that I've been making assumptions about what I mean by "decentralized". I think we have 4 different levels of "centralization":

1. Kernel (yes, I know, there are multiple levels inside the kernel but I'm ignoring that for now)
2. Userspace liballoc core
3. Userspace liballoc driver back-ends
4. Userspace GL/Vulkan/CL/etc. driver

I think what I'd prefer is to have at least caps/constraints querying all the way down (or up?) in the GL/Vulkan/etc. driver. Initially I wanted to put everything there, but the SoC use-case has convinced me that we need to have at least the final allocation in something more centralized. Obviously we can't 100% centralize it (because we need vendor plugins) but we don't want clients to have to walk around looking for someone to allocate.

[lynxeye-dev]

@jekstrand Agree on the various levels of centralization. I would argue that your list above should probably swap entry 2 and 3 to get it sorted from the farthest away from the normal application usage (the app arguably should not talk to the kernel directly) to what an application should use for the common usage.

I understand your desire to have caps/constraints querying at the accel API level, but I don't have a good idea how this would work for other devices we might want to integrate with that don't even expose this API level, like V4L. The highest API level exposed by V4L itself is the kernel level, we could have a common driver backend sitting on top of that, but even then the highest API level exposed would be the liballoc core level.

I'm leaning toward the notion that the application (even compositor) may not want to talk to liballoc core if there is a higher API level available. So allocating a Vulkan buffer (or EGL back buffer or whatever) should stay inside the respective API. For a sharable buffer you probably would just pass in the intersected caps/constraints to your Vulkan buffer alloc call with the Vulkan driver talking to liballoc to do the allocation (possibly lopping back into your own driver through your liballoc driver backend).

[robclark]

@jekstrand I think I'd go for your entry 3 (although I think of them more of alloc device instances, ie alloc_dev_tin USAGE.md. I don't think "liballoc core" should actually be a thing. Because in some cases (unless we had a central allocator daemon that had permissions to open all of the devices) you are not going to have all the alloc_dev_ts in the same process. And we want to leave that up to someone above us to know how to serialize (wayland/binder/whatever)..

Also, fwiw, current android builds using freedreno/virgl/vc4 are not using ION.

@lynxeye-dev re: v4l or other APIs which are purely kernel level, idea was to have a generic/common backend for them (ie. liballoc-v4l.so backend which knows how to use v4l uapi to query supported formats and whatever else). Have a look at USAGE.md. (Although I should probably extend that a bit to show getting the alloc_dev_t via some hypothetical EGL extension for case of a gl driver..)