f_autoconvert: keep hw frames on the GPU when possible - #18300
Conversation
|
You should be able to sequence it using the format filter to first implicitly convert by specifying format=nv12 or whatever the right one is, and then format=d3d11va (or whatever we call it). You could also ask Claude to implement the cuda d3d interop path. CUDA supports it but no one has been motivated to do it so far. |
|
I'm not a big fan of round-tripping transfer through system memory silently. Depending of image resolution and framerate this can be a lot of bandwidth that can silently kill the performance on lower end gpus. You should be about to export frame from CUDA, Vulkan, D3D12 as win32 handle and import that directly to d3d11, it will still do copy, but hopefully the GPU->GPU one. Also you can already use nvdec with d3d11vpp, like so |
|
Good idea, and you were both right that the round trip should not be the default. It turns out it can be done like this instead: the first version routed through system memory whenever the source was a hw format, which also meant a pair that could be mapped would never get the chance to. This version only falls back to system memory when the pair has no mapping available. It now decides per candidate target: bool download_first = !imgfmt_is_sw &&
!mp_hwupload_can_map(fmts[i], img->imgfmt);with Before, with After: For reference, on master with Based on Claude investigation, it found that It also confirmed the paths that were already zero copy are untouched and stay silent: d3d11 + d3d11va still goes Testing it ran: 2x and 3x @kasper93 on @philipl on the cuda/d3d interop suggestion: it did not get there. What it did get is the prerequisite, device derivation for D3D11VA, which lavu does not implement at all: That is device level only. It shares no frames and removes no copies, it just puts both devices on the same adapter. I plan to send it to ffmpeg-devel, but would rather hear any objections here first. It found the interop itself blocked in two places: lavu's (As I understand it: D3D11 hands out a frame as one slice of a stack of pictures, saying "here is the stack, take slice 3". The Vulkan side has nowhere to write down "slice 3", it can only point at a whole picture. So the only D3D11 texture it can accept is one holding exactly a single frame. A fix I have not attempted: lavu's dynamic d3d11va pools already allocate one texture per frame through It would also not remove the need for this branch regardless: D3D11 interop is one directional. Vulkan can import a D3D11 texture, D3D11 cannot import Vulkan memory. So vulkan to d3d11 has no mapping in that direction at all, and something has to carry it. |
|
You'll have to spend some time thinking about what design actually makes sense here. I don't know if Claude if capable of doing a good job on its own; my experience suggests it probably would not, and the comments reflect that. The cuda <-> vulkan interop is an example of one where there is a one-way import restriction. Cuda cannot export, it can only import - so the model is that vulkan always exports an image and then cuda imports it, and if the data needs to flow from cuda to vulkan, cuda does a GPU-side memcpy. And, in fact, even when data is flowing from vulkan to cuda, we do a copy anyway for consistency. GPU-side memcpys are cheap. So if d3d11 requires a one-way relationship, you'd probably want to handle it in the same way. You should look at/point Claude at the cuda interop API docs. Between that and the existing ffmpeg code, there should be enough general guidance. The majority (all?) of the work required here is on the ffmpeg side. As for whether any of the resulting code would be accepted, I don't know. ffmpeg doesn't have an official policy, but in practice this means "If the code meets the project standards and you can communicate about it showing an actual understanding of what you're doing, then who cares what tools you use". I would not consider what we have right now as meeting that bar, but I wouldn't be the one reviewing or approving any resulting ffmpeg PR. The fundamental d3d11 (or 12) interop with Vulkan and Cuda is interesting, and on some level long sought after, but it's a messy exercise that clearly hasn't motivated anyone sufficiently over the years. |
I agree, it may not be able to contextualize everything that's going on, let alone see the sane solution, except surface level glue/workarounds. Our frame copy/mapping/conversion code is already quite complex in what it does, and it's easy to get lost in it, with all the corner cases.
This is more complex interop, and basically needs not only "convert" code work, but also pre-existing knowledge in frame allocator. Though, it's not really nothing special. Most of interop need to have frames/memory allocated in certain way that is compatible with "exporting" or "sharing".
I always prefer to have same device pipeline, without sharing. I rectal trying to evaluate d3d12 hwdec use directly to our d3d11 renderer, but it was slow. I don't recall now if this was only GPU-GPU copy, or something more, but was disappointed in the result, where native d3d11 was faster in this case and better.
Yeah, most would be on ffmpeg side, maybe with an API to request sharing between certain apis up front. Though, I have not though about what is available now. I don't want to be this guy, but since we have multiple APIs, which all could be interoperable, this could be done in generic way, with some abstraction, maybe exiting ffmpeg API is enough. But the point being, that we should think about extensibility, not saying that every interop needs to be implemented, but most of the logic is in fact common between those APIs, they often use same underlying primitive to share memory. Anyway, I know nothing, I have not did into just now. |
c0e65d2 to
e0bf0bc
Compare
|
Sorry for the pause, I kept running out of Claude tokens implementing your suggestions. Pushed a redesign. The previous revision still round tripped every hw to hw transfer through system memory. This one tries the direct libavutil transfer at runtime and falls back only when that fails, remembering the result per source format and warning once. The FFmpeg side is two commits on https://github.com/flowreen/FFmpeg/tree/d3d11-vulkan-transfer : device derivation (what #18301 uses) and D3D11 to Vulkan frame transfers (what this PR uses). The transfer commit is conformant now. Single plane textures import with the dedicated allocation the spec asks for, capability queried, no vendor checks. Multiplanar formats stay on the system memory route, so NV12 passthrough behaves as today, while single plane output such as x2bgr10 (the case RTX HDR needs) gets the GPU path. Synchronization is a shared D3D11 fence imported into Vulkan as a timeline semaphore, so the copy is ordered against D3D11 work on the GPU instead of by spinning on the CPU. Both commits are at the point where I would defend them on ffmpeg-devel. @kasper93 on performance, you are right and it is not close: the vulkan VO costs about 1.4 ms per frame over the d3d11 VO before any filter exists, an order of magnitude above the filter cost on either route. The direct transfer halves the sharing overhead and native d3d11 stays the better choice. 
@philipl on cuda to d3d11: checked, it hits the same wall. CUDA imports a D3D11 resource only as a dedicated allocation, which rejects the nonzero offset the chroma plane needs, so nvdec still round trips on the way in. Verified on an RTX 5090: byte exact both directions for BGRA, X2BGR10 and RGBAF16 up to 4K including odd sizes, screenshots hash equal to the native d3d11 path, all 12 hwdec modes pass with nvidia-true-hdr riding the direct path, and the validation layers report nothing against the transfer path. Neither mpv PR needs the FFmpeg side: both build and run against stock 8.1.2, though the vulkan case needs the own device commit in #18301 before the filter can be created at all. If you would rather wait for the FFmpeg side before merging either, say so. Either way, I hope the FFmpeg side proves useful beyond mpv. |
|
Unfortunately, you will need the $100 Claude subscription if you want to be able to work on a project of this scale without interruption. A few points:
|
e0bf0bc to
8c4fc06
Compare
|
Dropped the implicit host memory paths and updated this PR's branch. The FFmpeg branch and #18301 are unchanged. hwupload no longer copies hw to hw transfers through system memory, and autoconvert no longer downloads to convert between two hw formats. How it works now: when both hw and sw targets are available, a real test with the current frame (map or transfer, against the same device and pool parameters the uploader would use) decides between the direct GPU route and the plain download the consumer already accepts. When only hw targets exist there is nothing to decide, so the uploader is inserted and a failing transfer surfaces on the first frame like today. A target that would need a sw format conversion on the way is rejected at setup. Net effect: the direct x2bgr10 path is unchanged, nvdec into a vulkan filter chain keeps working wherever libavutil supports it, and nvdec into d3d11vpp now needs nvdec-copy or d3d11va decoding, with plain nvdec the filter drops out on the first frame.
Claude's explanation here was unexpected to me, in the sense that nothing in FFmpeg's existing semaphore code had to change for it: D3D simply names its timeline semaphore "fence". ID3D11Device5::CreateFence creates the same object D3D12 uses, a shareable 64 bit counter that only ever counts up and can be waited on for any value, so it is reusable, which is exactly what a Vulkan timeline semaphore is. Vulkan defines VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_D3D12_FENCE_BIT for exactly this import, so the D3D11 fence comes in as a plain timeline semaphore and the wait goes through the existing code in vulkan.c (ff_vk_exec_add_dep_wait_sem) unchanged. For in tree code that already treats it this way, libavutil/hwcontext_d3d12va.c does all of its sync with a single ID3D12Fence that counts up forever, each wait just targets the value it needs. In the transfer commit the pairing sits in one struct, D3D11SyncState in hwcontext_vulkan.c holds the ID3D11Fence and the VkSemaphore it is imported as side by side, with one value counting for both.
We went looking for them and there is nothing to find: d3d12.h declares no semaphore object at all, the only synchronization object ID3D12Device can create is the fence, and everything else (queue waits, cross API sharing, D3D11on12) is built on that one type. On the multiplanar case: the blocker is not alignment. Importing a shared NV12 texture into Vulkan requires a dedicated allocation (VkMemoryDedicatedAllocateInfo), and a dedicated import always maps the whole resource starting at offset 0, so every legal binding lands on the luma plane and the chroma plane is unreachable. We measured this on NVIDIA: the only arrangement that reads chroma correctly is binding two single plane VkImages (R8 and R8G8) over a non dedicated import at a driver specific offset, which violates the dedicated allocation rule (VUID-VkBindImageMemoryInfo-image-01445) and returns VK_ERROR_DEVICE_LOST on Intel, so it cannot be shipped. Splitting the planes on the D3D11 side before export does not work either, ID3D11DeviceContext::CopySubresourceRegion silently refuses NV12 into R8/R8G8 textures (the destination stays zero), so a real split would need a shader or ID3D11VideoProcessor pass that hwcontext_d3d11va has no code for. So multiplanar stays off this route until drivers expose the planes, and the CUDA to Vulkan multiplane import you mention landing would be the clean fix for the nvdec side. The same dedicated wall exists in CUDA's own API today: external memory imports of a D3D11 resource must set CUDA_EXTERNAL_MEMORY_DEDICATED, and the dedicated import then rejects the nonzero offset the chroma plane sits at. A fix on NVIDIA's side would land in one of two places: cuExternalMemoryGetMappedMipmappedArray accepting per plane mappings on dedicated imports (or the multiplane compatibility you said they are adding), or the Vulkan driver binding the chroma plane of a dedicated NV12 import at its real offset instead of aliasing luma at offset 0. The CUDA side of that would first surface in the CUDA release notes and the external memory API linked above, and once it ships a direct nvdec route becomes possible to implement, nothing in this PR provides it by itself. On D3D11 vs D3D12: d3d11 is what mpv on Windows actually produces, it decodes with d3d11va, filters with d3d11vpp (ID3D11VideoProcessor) and has no d3d12 code at all. On the FFmpeg side, hwcontext_d3d12va exists with its own decoders, but that context only implements transfers to and from system memory, and neither hwcontext_vulkan nor hwcontext_d3d11va has a d3d12 case, so no d3d12 frame can reach this path today. Nothing in the design is d3d11 specific though, the sync object already crosses as a D3D12 fence, Vulkan has the matching D3D12_RESOURCE memory handle type, and supporting d3d12 frames would be a small follow up if mpv ever wires up FFmpeg's d3d12va decoders. This discussion became too technical 馃槅 |
|
I'm not sure it updated the PR but I'm sure you'll sort that out. If you cannot import multiplane nv12 into Vulkan from d3d, then you need to import separate single plane images and then copy the contents of the Vulkan image planes into those imported images. If you're lucky you can use the separate images on the d3d side without another copy but if not, you may have to copy them into a multiplane image to enable whatever filter. |
Shareable textures are cheap and allow libavutil to transfer frames directly to another device, for example d3d11 to vulkan. Every D3D11 pool mpv creates gets the flag.
mapper_init() required ra_get_imgfmt_desc() to succeed but never used the result. A Vulkan hwdec device is only registered on a libplacebo RA (vulkan_init requires one), and libplacebo maps planes by their Vulkan format, so the check only blocked formats the description cannot express, like the bit packed X2BGR10, which map fine. vo_gpu on vulkan runs on such an RA too. Its renderer builds the same description itself, does not check the result, and renders black for these formats. That is its behavior with or without this check, since nothing could produce them for a VO before.
When a hw source has to be converted to a different hw format, build the chain around the uploader instead of rejecting the format. Whether libavutil can map or transfer between the two devices cannot be queried, so when a sw target also exists and the result decides the route, mp_hwupload_probe_hw_to_hw() tests it with the current frame, mirroring the device, pool parameters and map or transfer choice the uploader will use, and the plain download is taken when the frame cannot stay on the GPU. With only hw targets there is no alternative route, so the uploader is inserted untested and a failing transfer surfaces on the first frame. Rejecting a target previously left the just created uploader in filters[2] and clobbered fmts and num_fmts, ending the scan after one candidate. Rejections now free the uploader and try the next target, and when no target survives with only hw targets, the build fails rather than passing frames no consumer could read.
8c4fc06 to
631cb67
Compare
Re-pushed with cleanups.
Implemented, pushed as a third commit on the FFmpeg branch: https://github.com/flowreen/FFmpeg/commits/d3d11-vulkan-transfer Compared to stock, nv12 and p010 frames now cross d3d11 <-> vulkan directly, both directions byte exact, and the 1080p hop costs about 200 us against about 940 us through system memory. 
聽
nvdec is unchanged, cuda still cannot export its frames, so that side is blocked on NVIDIA shipping their part. |
flowreen
changed the title
|
Good to see that there are performance benefits. As for nvdec and cuda - cuda can import from D3D11 - check the CUDA driver API's interop documentation. This is already how the cuda <-> vulkan interop works - you have to export from the vulkan side, and then you copy into the buffer from cuda to get the nvdec frame out. If you want to export the frame to d3d11, you have to do the same thing - import some d3d11 buffer/image and then copy the frame into it. |
The RA has no per component description for bit packed formats, but the interop only ever moves whole texels, and the texture format carrying the layout is known by name, the same way the d3d11va format table names it. This lets x2bgr10 frames from a CUDA source reach the VO.
You are right, it works. Implemented, pushed as a fourth commit on the FFmpeg branch: https://github.com/flowreen/FFmpeg/commits/d3d11-vulkan-transfer It rides the graphics interop API, whose map/unmap ordering replaces all fence machinery, and nv12/p010 reuse the plane bridge, which moved into hwcontext_d3d11va for both interops to share, so the two plane commit was re-pushed too. Compared to stock, cuda frames now cross to and from d3d11 directly, both directions byte exact including decoder texture arrays, and the 1080p nv12 hop costs about 400 us against about 2.3 ms through system memory. 
聽
In mpv the runtime probe picks the route on its own: |
|
Good milestone. However, looking at the ffmpeg code, you're using cuGraphicsMapResources which is basically a deprecated API. you should use cuImportExternalMemory instead, and you should be importing the D3D11 fence with cuImportExternalSemaphore - so doing the same semaphore basic sync that you do on the Vulkan side. This should remove the need for the pthread mutex - which you should not want to be using. |
3 participants
Converting between two hw formats in the filter chain either fails on the first frame (only hw targets, for example nvdec feeding d3d11vpp) or always leaves the GPU (mixed targets, autoconvert unconditionally downloads), because libavutil transfers between two hardware frame contexts exist only for a few device pairs and support cannot be queried, only tried.
With this change autoconvert builds the chain around the uploader for hw sources and keeps the frame on the GPU when that is actually possible:
With the libavutil counterpart (https://github.com/flowreen/FFmpeg/tree/d3d11-vulkan-transfer) D3D11 textures transfer directly to Vulkan in both directions, single plane formats as one image and nv12/p010 through per plane bridging, so both the default nv12 passthrough and --vf=d3d11vpp=format=x2bgr10 (RTX Video HDR and VSR) stay on the GPU under --gpu-api=vulkan. Without it, behavior matches master for every route that exists today.
Under a Vulkan VO the d3d11vpp filter itself needs #18301 before it can be created at all; the two PRs are independent otherwise.
AI disclosure
Disclosure: this patch was written with AI assistance (Claude Code), which diagnosed the bug, wrote the change, and ran the verification on my machine. I have reviewed it and understand what it changes and why, I take responsibility for it, and review responses will be written by me. filters/f_autoconvert.c carries no licence header and so falls under the repository default of LGPLv2.1+; I submit the change under that licence.