Skip to content

Add parallel game workers behind -j/--jobs - #10

Open
sav wants to merge 21 commits into
masterfrom
sav/parallel-jobs
Open

Add parallel game workers behind -j/--jobs#10
sav wants to merge 21 commits into
masterfrom
sav/parallel-jobs

Conversation

@sav

@sav sav commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Adds -j/--jobs, spreading games across a pool of worker threads. The PGN
reader stays a single sequential producer and dispatches each game as it
completes; submit() blocks while every worker is busy, so memory stays bounded
by the worker count no matter how large the database is.

src/game.cpp is untouched: Builder::endPgn already moved each game out and
let the next startPgn allocate a fresh one, so the handoff was already safe to
transfer across threads. The dispatch point is the visitor lambda in
Runtime::run.

Design

A sol::state cannot be entered by two threads, so a Pipeline owns one Lua
state plus its own copy of every plugin, its own engines and its own
ctx.scope. Runtime keeps the options, the shared output writer, the writer
registry and N pipelines. -j1 is a strict specialization of the general path,
not a separate branch: in-flight work is bounded by the worker count rather than
a queue length, so at -j1 submit(k+1) cannot return until game k has
finished and a stop from k is always seen before k+1 is dispatched.

src/runtime.cpp (969 lines) is split into options / writer / pipeline /
pool / runtime along the dependency order.

Behavior changes

  • ctx.shared is removed. No bundled plugin used it, and a table cannot be
    shared between Lua states, so it could not have kept its documented meaning
    under -j.
  • Above -j1 games are written in completion order, not input order.
  • ctx.open returns the same writer for a path already open instead of
    truncating it, so stages and workers can share a file. Reopening with the
    other mode is now an error.
  • init/finish run once per worker, so histogram.lua and dedup.lua
    only aggregate correctly at -j1. ctx.worker/ctx.workers are exposed for
    the simple cases (csv.lua uses them to write its header once); a general
    reduce channel is recorded in TODO.md.

Incidental fix

Engine called std::fprintf/std::strerror between fork() and _exit().
Neither is async-signal-safe: with worker threads, one holding the stdio lock at
fork time would deadlock the child while it still held the pipe open, hanging
the parent's read. The message is now rendered before the fork and the child
issues a single ::write.

Testing

93 ctest cases, up from 83. -j1 is byte-compared against the sequential
output; -j4/-j8 are compared as a multiset of lines since completion order
is not fixed. Separate cases cover the bounded stop, an abort raised on a worker
travelling back to the main thread intact, and two stages sharing one
ctx.open file across every worker.

Measured on 4800 games through a compute-bound plugin (32 cores): 8.47s at
-j1, 4.33s at -j2, 2.20s at -j4, 1.24s at -j8.

ThreadSanitizer has not been run. A TICTAC_SANITIZER build option is
wired up and the flags reach both the compile and link lines, but no clang
sanitizer runtime is installed on the machine this was developed on
(libclang-rt-19-dev), and g++-13 there lacks <print>. The suite was run
repeatedly instead, which is not equivalent. Worth a TSan pass before merge.

sav added 8 commits July 26, 2026 21:11
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
@sav

This comment was marked as resolved.

Comment thread CMakeLists.txt
Comment thread CMakeLists.txt
Comment thread LUA.md
Comment thread LUA.md Outdated
Comment thread LUA.md Outdated
Comment thread LUA.md Outdated
Comment thread README.md
Comment thread TODO.md Outdated
Comment thread plugins/csv.lua Outdated
@sav

This comment was marked as resolved.

sav added 9 commits July 27, 2026 09:15
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
@sav
sav force-pushed the sav/parallel-jobs branch from 938eca6 to 47fc4f9 Compare July 27, 2026 12:15
Assisted-By: Claude Opus 5
Comment thread plugins/csv.lua Outdated
Comment thread src/app.cpp Outdated
if (on_error == "pass") opts.onError = OnError::Pass;
else if (on_error == "drop") opts.onError = OnError::Drop;
else opts.onError = OnError::Abort;
// Resolved here so the runtime always gets a real worker count.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove this comment. the code and the function documentation is enough in this case.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, dropped in 1d909a6. The >= 1 invariant is already stated where it matters -- on RunOptions::jobs ("always >= 1 by the time run() sees it") -- and --jobs' own help text carries the "0 = one per CPU" half, so the std::max line reads fine on its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant