Revisit command-buffer simultaneous-use

[EwanC]

During the course of writing CTS test for cl_khr_command_buffer, in particular KhronosGroup/OpenCL-CTS#1584, questions have arisen about the behaviour of CL_COMMAND_BUFFER_CAPABILITY_SIMULTANEOUS_USE_KHR and how usable it is for developers.

E.g - Do the following examples exhibit simultaneous-use behaviour?

// Case 1: queue is an out-of-order queue.
// Note: No event dependencies!
clEnqueueCommandBufferKHR(1, &queue, cmdbuf, 0, NULL, NULL);
clEnqueueCommandBufferKHR(1, &queue, cmdbuf, 0, NULL, NULL);
clFinish(queue);

// Case 2: queue0 and queue1 are in-order queues.
clEnqueueCommandBufferKHR(1, &queue0, cmdbuf, 0, NULL, NULL);
clEnqueueCommandBufferKHR(1, &queue1, cmdbuf, 0, NULL, NULL);
clFlush(queue0); // these flushes may not be required
clFlush(queue1);
clFinish(queue0);
clFinish(queue1);

In my mind, a lot of the behaviour of simultaneous-use actually depends on whether a vendor implements queues eagerly or lazily. In lazy execution (what happens in Codeplay' implementation for example) we wouldn't start submitting anything until the clFinish()/cFlush(), so the pending count keeps on increasing until then, even for in-order queues. In an eager queue implementation, the first enqueue would be run straight away, and may indeed have finished by the time the host code calls the second command buffer.

Depending on an unreported implementation detail is hard for a user developing code to reason about
though, so maybe we can do better here. Before the cl_khr_command_buffer extension becomes final from provisional we should confirm if the currently specified behaviour is what we want, or whether we should revisit this property.

[bashbaug]

I think this is another case to consider, especially for implementations that implement queues lazily (AKA "batched"). I think this would be considered simultaneous use?

// Case A: queue is an in-order queue.
clEnqueueCommandBufferKHR(1, &queue, cmdbuf, 0, NULL, NULL);
clEnqueueCommandBufferKHR(1, &queue, cmdbuf, 0, NULL, NULL);
clFinish(queue);

The in-order queue semantics guarantee that the first command buffer execution will "complete" before the second command buffer execution "starts", so they will never be executing "simultaneously", but the second command buffer execution may certainly be "queued" or "submitted" before the first command buffer execution is "complete".

[EwanC]

I've been mulling this over in the background, and these are my thoughts so far on ways forward. Nothing concrete but posting to keep the discussion ticking over and maybe help flush out more ideas:

1. Change functionality from optional to mandatory. Would be the most beneficial choice from the user perspective, but increases implementation burden. Possible implementations for simultaneous-use that don't have native support could be to A) deep-copy the command-buffer and enqueue the copy, B) runtime schedules the simulatenous submission after the first submission by creating an implicit dependency, C) runtime blocks on simultaneous submission in command-buffer enqueue entry-point until the first submission is complete and flushes the queue (effectively the runtime does the clFinish(queue) rather than application code)

2. Change the definition of simultaneous execution from a course "if pending count > 1" to something more granular that takes into account dependencies. For example, "if pending count > 1 and the command-buffer being enqueued has a no dependencies on pending executions of the command-buffer". The ordering dependency means no simulatneos execution of command-buffer could occur, so I think Ben's in-order snippet would be valid on a runtime without simultaneous-use support.
   Cross queue & out-of-order queue submissions of the same command-buffer would still need user invocations of blocking waits if there is no explicit event dependency. If we wanted to avoid that, I think we're back at 1B) above.

What i'm wary of with suggestion 2) is that the implementation effort reqired for vendors that don't support simultaneous enqueue, we'd be as well specifying 1) to cover more use-cases.

[EwanC]

From the 16/05/23 WG call, I said I'd look again at what Vulkan do for this:

• The VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT flag passed on command-buffer creation specifies that a command buffer can be resubmitted to a queue while it is in the pending state, and recorded into multiple primary command buffers.

This is analogous to our command-buffer creation simultaneous use flag.

• vkQueueSubmit backs this up under valid usage:

If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state

This corresponds to us not being able to call clEnqueueCommandBufferKHR with a command-buffer in the pending state, if the simultaneous use flag was not set.

• Pending state in the command buffer lifecycle is defined as

Queue submission of a command buffer changes the state of a
command buffer from the executable state to the pending state.
Whilst in the pending state, applications must not attempt to
modify the command buffer in any way - as the device may be
processing the commands recorded to it. Once execution of a
command buffer completes, the command buffer either reverts
back to the executable state, or if it was recorded with
VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT, it moves to the
invalid state. A synchronization command should be used to
detect when this occurs.

I think that this aligns with what we have specified now, that it's not just about the simultaneous execution of a command-buffer but also the simultaneous processing of the command-buffer by the runtime.

• vkUpdateDescriptorSets also has some wording relating to pending state, which could be analogous to clUpdateMutableCommandsKHR updating commands in a command-buffer.

Descriptor bindings updated by this command which were created without the VK_DESCRIPTOR_BINDING_UPDATE_AFTER_BIND_BIT or VK_DESCRIPTOR_BINDING_UPDATE_UNUSED_WHILE_PENDING_BIT bits set must not be used by any command that was recorded to a command buffer which is in the pending state

We currently say for clUpdateMutableCommandsKHR that "Using this function when command_buffer is in the pending state and not created with the CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR flag causes undefined behaviour."

[bashbaug]

Interesting... I think this implies that all three of the cases above should be considered "simultaneous use", agreed?

Basically, to use a command-buffer "non-simultaneously" you need to observe that the command-buffer is no longer in the pending state, which could be done directly (by querying the command-buffer state) or perhaps done indirectly (say, by calling clFinish on the command-queue executing the command-buffer?).

IMHO it's an interesting thought experiment to consider if we could write some sort of validation layer to check if an application is properly using command buffers that do not support simultaneous use. For example, could we:

1. Track all command-buffer executions that are possibly pending. When a command-buffer is enqueued for execution, check if it is on the "possibly pending" list. If it is, this is an error. If it is not, add it to the list.
2. If an application calls clWaitForEvents to wait on the event associated with the command buffer execution, remove the command-buffer from the possibly pending list.
3. Likewise, if an application calls clFinish on the command-queue where the command-buffer is enqueued for execution, or otherwise performs some sort of blocking call on the command-queue, remove the command-buffer from the possibly pending list.
4. This is the tricky one: if an application waits on an event using clWaitForEvents that itself has a dependency on the command buffer execution, remove the command-buffer from the possibly pending list.

This might be doable?

For fun, do we agree that this would technically be considered simultaneous use?

// Case B: queue is an in-order queue.
clEnqueueCommandBufferKHR(1, &queue, cmdbuf, 0, NULL, NULL);

// This will cause the command-buffer to eventually begin executing:
clFlush(queue);

// The command-buffer will probably finish while sleeping...
sleep(a_very_long_time);

// Uh oh, we never observed that the command buffer was complete!
clEnqueueCommandBufferKHR(1, &queue, cmdbuf, 0, NULL, NULL); 

[EwanC]

Interesting... I think this implies that all three of the cases above should be considered "simultaneous use", agreed?

Basically, to use a command-buffer "non-simultaneously" you need to observe that the command-buffer is no longer in the pending state, which could be done directly (by querying the command-buffer state) or perhaps done indirectly (say, by calling clFinish on the command-queue executing the command-buffer?)

Yup, I think we're aligned on the interpretation. So I don't think we need any functional changes to the spec, and the slightly awkward code in the CTS that motivated this discussion is a result of the need to observe the completion of the previous enqueue.

We could make improvements in the description of simultaneous-use to better clarify these findings though, that 1) simultaneous-use isn't just about when the command-buffer is executed, but when it's enqueued. 2) The onus is on the user to provide explicit synchronization in the non-simultaneous case to guarantee that the previous enqueue has completed.

IMHO it's an interesting thought experiment to consider if we could write some sort of validation layer to check if an application is properly using command buffers that do not support simultaneous use.

I think that should be possible, there might be other transitive cases like a blocking command, e.g clEnqueueReadBuffer, on the same queue with a previous pending command-buffer enqueue dependency. But it does sound possible, and useful!

For fun, do we agree that this would technically be considered simultaneous use?

Yup, I don't think it would be guaranteed by the spec that the case isn't simultaneous use.

[SunSerega]

If an application calls clWaitForEvents to wait on the event associated with the command buffer execution, remove the command-buffer from the possibly pending list.

What about clSetEventCallback? For instance, it could be said:

• While the callback is executing, a new enqueue of the command buffer is allowed on the callback thread. But after the callback finishes, a call to clWaitForEvents is once again needed to ensure no simultaneous use. Even if the callback was triggered by clSetUserEventStatus and immediately executed on the same thread.

This might be overly limiting. But I think the alternative is to always consider new enqueue a non-simultaneous use if clSetEventCallback was ever called on any related event.

And anyway, while the limitation to blocking calls in the callback may make it harder to work around, I believe it is better to enqueue everything new in the callback and only send blocking calls like clFinish (which are not checked for simultaneous use like the enqueue) to be executed in a separate thread.

[bashbaug]

What about clSetEventCallback? [...]

Yes, I think this could work to avoid "simultaneous use" too, because of this line in the spec?

"If a callback procedure P is registered on an event E, then E global-synchronizes-with all operations of P."

If we wanted to be really thorough we would go through each line in the execution model part of the spec to be sure its covered.

[EwanC]

We've been comparing the simultaneous use semantics to Vulkan, but all the other APIs I've looked at don't provide this functionality (CUDA-Graph, HIP-Graph, Level-Zero, DX12) and limit a single instance of a command-buffer to be executing at once.
However, the majority of these API provide a clone API (couldn't see on in DX12) which could be used to support the use-case of submit->update->submit without blocking on the host. As a user could create multiple clones in a warm up stage of the application, then in the hot path: submit the original command-buffer, update a clone, submit that clone while the original command-buffer is still executing.

Although creating these clones might be more work for the user, semantics across implementations would be clearer. One concern i'm not sure about is if cloning would be extra overhead for hardware which driver's do natively support simultaneous use. However we could maybe define the clone semantics in such a way that the clone isn't a complete deep copy. For example, in
the Level Zero clone API there's the wording "The cloned command list must be destroyed prior to the source command list." which means that some resources wouldn't need to be copied over to the clone.

In summary, I'm floating the idea that we could introduce a command-buffer clone API to replace simultaneous-use.

[bashbaug]

Noting for completeness: there was also some discussion of a clone API in internal issue 251, in two places actually.

I agree this is something we should consider.

[FreddieWitherden]

We've been comparing the simultaneous use semantics to Vulkan, but all the other APIs I've looked at don't provide this functionality (CUDA-Graph, HIP-Graph, Level-Zero, DX12) and limit a single instance of a command-buffer to be executing at once. However, the majority of these API provide a clone API (couldn't see on in DX12) which could be used to support the use-case of submit->update->submit without blocking on the host. As a user could create multiple clones in a warm up stage of the application, then in the hot path: submit the original command-buffer, update a clone, submit that clone while the original command-buffer is still executing.

In summary, I'm floating the idea that we could introduce a command-buffer clone API to replace simultaneous-use.

CUDA and HIP go further by providing implicit cloning. That is to say I can submit a graph, then update a node (say by cuGraphExecKernelNodeSetParams), submit it again, and everything is fine in the sense that the first queuing will use the old parameters and the second queuing will use the new parameters. This is because "The modifications only affect future launches of hGraphExec. Already enqueued or running launches of hGraphExec are not affected by this call. hNode is also not modified by this call." Broadly, this is what developers want. Managing what is queued, what is executing, and what has executed is the runtimes problem and it is in a better place to know how to handle such modifications (as in knowing if to make a clone or not).

[SunSerega]

In my wrapper, I started with explicit cloning by the user, then made it implicit through a deep but lazy copy (only copy deeper when needed). But in the end, I've pivoted to the opposite idea:

The created graph is completely immutable, however, inputs like kernel parameter or buffer offset for clEnqueueWriteBuffer are sub-graphs that, when created, are specified to return either:

• A constant, or
• A parameter, that can be set per graph invocation, or
• A dynamic value, like a result of an RNG call on the CPU during graph execution

And they can also have a payload that needs to execute before the return value becomes valid.
Like executing a slow function on the CPU to check/prepare something or executing another command on the returned cl_mem handle.
But as long as the return value is a constant or graph parameter, this payload only adds an event to the clEnqueue* functions, while the returned value can be passed to clEnqueue* immediately.

In other words, the key idea here is that changeable command inputs are created beforehand.
So the user can specify if they will need to modify the value (and when), and the implementation can optimize the invocations of clEnqueue* based on that info.

P.S. The point of separating the constants and graph parameters is to avoid the mutable state right before the graph invocation. clSetKernelArg is a thorn in the thread-safe code. But passing such parameters into my equivalent of clEnqueueCommandBuffer has some overhead. Besides, most values are just not meant to be changed. So, I made it easiest and most intuitive to use constants everywhere, while using parameters or dynamic values requires actual thinking about what you want.

[EwanC]

Thanks for the input @FreddieWitherden We've got wording in clUpdateMutableCommandsKHR already that it only affects future enqueues of a command-buffer, which I hoped would imply "already enqueued or running launches of the command-buffer are not affected by this call" but maybe should say that explicitly too.

I think an interesting difference to CUDA is the wording in cudaGraphLaunch about ordering with previous executable graph launches.

Only one instance of graphExec may be executing at a time. Each launch is ordered behind both any previous work in stream and any previous launches of graphExec. To execute a graph concurrently, it must be instantiated multiple times into multiple executable graphs.

Creating implicit dependencies between executions of the same command-buffer isn't something we've talked about before, but if I'm reading that correctly is what CUDA is doing for executable graphs. So I think one difference with the CUDA model v explicit clone, is that with the CUDA model a submit->update->submit flow would prevent submit 1 and submit 2 from overlapping execution. Whereas if the user did an explicit clone and submitted that for the second commit, it could run with the first submit.

---

Do you have a link to your wrapper @SunSerega? It sounds interesting, and i'm not quite following yet how the subgraph/payload concepts could be applied to command-buffers other than the idea of creating some handle to the graph inputs/outputs in advance.

[FreddieWitherden]

Creating implicit dependencies between executions of the same command-buffer isn't something we've talked about before, but if I'm reading that correctly is what CUDA is doing for executable graphs. So I think one difference with the CUDA model v explicit clone, is that with the CUDA model a submit->update->submit flow would prevent submit 1 and submit 2 from overlapping execution. Whereas if the user did an explicit clone and submitted that for the second commit, it could run with the first submit.

So it is important to remember in CUDA/HIP there is a graph and then there are instances of the graph. The instances are created with cuGraphInstantiateWithFlags and can then have their arguments tweaked (to an extent). Only instances of a graph can be launched, and a single instance will only ever run once. So if you want parallel execution you would instantiate multiple instances. (As for the why: I suspect instantiating a graph creates a bunch of CUDA streams under the hood; submitting a graph to a stream actually results in the kernels being submitted to these graph-private streams and only a synchronization marker is actually added to the stream you submitted the graph too.)

[SunSerega]

Do you have a link to your wrapper @SunSerega? It sounds interesting, and i'm not quite following yet how the subgraph/payload concepts could be applied to command-buffers other than the idea of creating some handle to the graph inputs/outputs in advance.

About my wrapper

https://github.com/SunSerega/POCGL
In this case, I was talking about this in particular:
https://github.com/SunSerega/POCGL/blob/unstable/Modules.Packed/OpenCLABC.pas

## uses OpenCLABC;

// "A" is an array, backed by GPU memory
// "ind" is a sub-graph, returning an integer value
// "Result" is a CommandQueue returning nil aka. nothing
function MakeQueue(A: CLArray<byte>; ind: CommandQueue<integer>): CommandQueueNil;
begin
  // HPQ = HostProcedureQueue. A command wrapper for a CPU code returning nothing
  // In this case the first action is to create a new thread and make it sleep for a second
  // Pretty ugly way to add a delay, but just an example
  Result := HPQ(()->Sleep(1000), need_own_thread := true);
  // Addition means sequential execution
  // Create CommandContainerQueue with a single buffer write
  Result += A.MakeCCQ.ThenWriteValue(1, ind).DiscardResult;
  // Basically the result is a write command, that first waits for a second before executing
end;

var A := new CLArray<byte>(4);
var Q := CQNil; // an instance of a ConstQueueNil singleton
var P := new ParameterQueue<integer>('par name for debug');

// Multiplication means parallel execution
// (doesn't make much sense to write to the same buffer in parallel tho...)

// "ind" is a ConstQueue
Q *= MakeQueue(A, CQ(0));
// Can also convert to ConstQueue implicitly
Q *= MakeQueue(A, 1);

// "ind" is a ParameterQueue
Q *= MakeQueue(A, P);

// HFQ is like HPQ but with a return value
// "ind" is of a special internal type,
// neither const nor parameter. basically dynamic
Q *= MakeQueue(A, HFQ(()->3));

CLContext.Default.SyncInvoke(
  Q + A.MakeCCQ.ThenGetArray.ThenUse(a->Println(a)),
  P.NewSetter(2)
); // Prints [1,1,1,1]

---

As for how it might relate to command buffers here... Well, not directly.
I mainly set it as an example of how fully immutable command graphs can exist. I'm not sure if command buffers can change enough at this point.

The issue header still says command buffers are provisional, but even the simplest implementation of this idea would still change a lot to work... For instance:

cl_command_input inp_const = clCreateCommandInputConst(
	0, nullptr, // command buffers to execute as payload before this input becomes valid
	sizeof(int), 123, // the const return value
	ec // error code return
);

cl_command_input inp_par = clCreateCommandInputParameter(
	0, nullptr,
	sizeof(int), // the size of parameter value
	ec
);

void make_dynamic_result(void* ptr) {
	int* res = ptr;
	*res = 3;
}
...
cl_command_input inp_dyn = clCreateCommandInputFunction(
	0, nullptr,
	sizeof(int), make_dynamic_result, // the callback to query value after all the command buffers
	ec
);

// (use these three as inputs for special commands in the command buffer)

int par_val = 2;
cl_command_buffer_parameter_value par_values[] = {
	{ inp_par, sizeof(par_val), &par_val }, // specify info about parameter size again, as a sanity check
};

ec = clEnqueueCommandBufferParameterisedKHR(
	//TODO All the normal parameters
	1, par_values
);