At some point, we may want to (add back?) support for a PriorityScheduler along the lines of what can be found in
The idea was that when a DataFlowTask is spawned, it is added to the DAG but the actual scheduling of the underlying task is delegated to a worker task which takes nodes from a runnable priority queue and executes them. After a task is finished, it inserts itself in a finished queue for removal (by another worker).
Pushing tasks into the runnable queue happens in one of two ways:
- A task inserts itself into
runnable if it is dependency-free when it is created.
- If a task has a dependency when created, it is the responsibility of its last unfinished dependency to put it in the runnable queue. This means that the cleanup phase of finishing a task involves
- Removing itself as a dependency on all of its outgoing edges
- If it was the last dependency of one of those outgoing nodes, add that node to
runnable
- Remove itself from the
DAG
This was dropped for simplicity until the package matures a bit more, and because I was not very happy with the way tasks were executed under this scheduler, but perhaps it is worth bringing it back to live at some point since priority scheduling can help guide the scheduler through the critical path more easily.
At some point, we may want to (add back?) support for a
PriorityScheduleralong the lines of what can be found inDataFlowTasks.jl/src/otherschedulers.jl
Line 108 in cf15cc7
The idea was that when a
DataFlowTaskis spawned, it is added to theDAGbut the actual scheduling of the underlying task is delegated to a worker task which takes nodes from arunnablepriority queue and executes them. After a task is finished, it inserts itself in afinishedqueue for removal (by another worker).Pushing tasks into the
runnablequeue happens in one of two ways:runnableif it is dependency-free when it is created.runnableDAGThis was dropped for simplicity until the package matures a bit more, and because I was not very happy with the way tasks were executed under this scheduler, but perhaps it is worth bringing it back to live at some point since priority scheduling can help guide the scheduler through the critical path more easily.