Skip to content

DESIGN.md

Latest commit

 

History

History
237 lines (149 loc) · 6.63 KB

DESIGN.md

File metadata and controls

237 lines (149 loc) · 6.63 KB
Raw

Design

This document is the detailed design doc for the task runtime.

Main pieces

Runtime

Runtime is the main API the caller uses.

Its job is to:

  • accept a callable and its arguments
  • create the state shared between the task and the future
  • create the Future
  • wrap the work into a Task
  • hand that Task off to the ThreadPool

So Runtime is basically the coordinator. It does not run tasks itself. It prepares them and sends them to the pool.

Runtime::submit() is thread-safe and may be called concurrently from multiple threads while the Runtime is alive.

ThreadPool

The ThreadPool owns:

  • the worker threads
  • the queue of pending tasks
  • the mutex and condition variable protecting that queue
  • the drain_ flag used during shutdown

The pool is responsible for waking workers when work arrives and making sure shutdown drains already submitted work instead of cancelling it.

Task

Task is a small move-only type-erased wrapper.

It stores one callable and later exposes task.run().

The important design choice here is that Task itself only stores a nullary void callable. That keeps the thread pool simple. It does not need to know about return types, exceptions, or user arguments.

Runtime::submit() adapts user work into that shape before the task ever reaches the queue.

Future<T>

Future<T> is a simple handle returned to the caller.

  • waits for the task to finish
  • hands back either the result or the captured exception

Important properties:

  • it is move-only
  • await() is single-use
  • a given Future has one logical consumer
  • dropping the future does not cancel the task

If await() is called twice, it throws FutureAlreadyAwaited.

If await() is called on a moved-from future, it throws FutureMissingState.

SharedState<T>

SharedState<T> is the shared source of truth for task completion.

Both the queued task and the returned future hold a shared_ptr to it.

It stores:

  • the task result as std::expected<T, std::exception_ptr>
  • whether the task is done
  • whether the result has already been consumed
  • the mutex and condition variable used for waiting

This is the piece that lets the task and the future live on different threads while still agreeing on task state.

Submission flow

When the caller does:

auto future = runtime.submit(func, arg1, arg2);

the runtime does a few things in order:

  1. figure out the return type
  2. allocate a SharedState<ReturnType>
  3. create a Future<ReturnType> pointing at that shared state
  4. build a Task that owns the callable, the arguments, and the shared state
  5. submit that task to the thread pool

The caller only sees the Future.

How callables and arguments are stored

Tasks run later, possibly after the submission site has already returned. Because of that, the runtime needs to own the callable and its arguments.

The storage rules are:

  • lvalue callables and arguments are copied into task-owned storage
  • rvalue callables and arguments are moved into task-owned storage

That gives the runtime ownership of everything it needs when the worker thread eventually runs the task.

More specifically, submission copies copyable lvalue callables and arguments once into task-owned storage and moves rvalue callables and arguments into task-owned storage without copies.

When the task later runs, the stored callable is invoked with the value category it requires: &-qualified callables run as lvalues, and &&-qualified callables run as rvalues.

One subtle but important detail: storage uses decay types. So plain references are not preserved unless the caller explicitly wraps them.

What actually runs in the worker

The worker thread does not run the original callable directly.

Instead, Runtime::submit() creates a lambda that captures:

  • the callable
  • the arguments
  • the shared state

That lambda is what gets stored inside Task.

When it runs, it:

  1. calls std::invoke(...) on the stored callable and arguments
  2. if the task returns a value, moves that value into SharedState::set_done(...)
  3. if the task returns void, just marks the shared state done
  4. if user code throws, catches the exception and stores std::current_exception()

So the task wrapper is doing two jobs:

  • run user work
  • publish the outcome back to the future

Worker thread loop

Each worker thread sits in worker_thread_loop().

The loop is:

  1. take the queue lock
  2. wait until either there is work in the queue or shutdown has started
  3. if shutdown has started and the queue is empty, exit
  4. otherwise move one task out of the queue
  5. release the lock
  6. run the task

The important part is that the lock is released before running user code.

That matters because holding the queue lock during task execution would be bad:

  • submissions would block behind running tasks
  • workers would block each other from taking more work
  • one slow task could effectively serialize the whole pool

So the lock only protects the queue. It is not held while user code runs.

How await() works

Future::await() just forwards to SharedState::await().

Inside SharedState::await():

  1. the future takes the shared-state lock
  2. it checks whether the result was already consumed
  3. if it was, it throws FutureAlreadyAwaited
  4. otherwise it marks the result as consumed
  5. then it waits until the task is done
  6. finally it moves the stored std::expected back to the caller

The result is marked consumed before waiting, not after.

That is intentional. It keeps await() single-use even if two threads race to call it on the same future object.

Error handling

User exceptions do not escape out of the worker thread.

Instead:

  • the task catches them
  • stores std::current_exception()
  • and marks the task complete with an error result

So the caller sees failure as:

std::expected<T, std::exception_ptr>

instead of as an exception thrown directly from the worker thread.

That keeps the runtime simpler and makes task failure part of the returned result channel.

Shutdown behavior

Destroying Runtime destroys its ThreadPool.

When the pool starts shutting down, it:

  1. takes the queue lock
  2. sets drain_ = true
  3. releases the lock
  4. notifies all workers

After that:

  • new submissions are rejected
  • already queued tasks still run
  • workers keep going until the queue is empty
  • only then do workers exit

So shutdown is draining, not cancelling.

That is why runtime destruction waits for submitted work to finish.

If someone tries to submit after draining has started, ThreadPool::submit() throws SubmittedToDrainingThreadpool.

Deadlocks

This is currently treated as user error. The runtime does not detect it or work around it.