Add parallel game workers behind -j/--jobs - #10
Open
sav wants to merge 21 commits into
Open
Conversation
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
This comment was marked as resolved.
This comment was marked as resolved.
sav
commented
Jul 27, 2026
This comment was marked as resolved.
This comment was marked as resolved.
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
Assisted-By: Claude Opus 5
sav
commented
Jul 27, 2026
sav
commented
Jul 27, 2026
| if (on_error == "pass") opts.onError = OnError::Pass; | ||
| else if (on_error == "drop") opts.onError = OnError::Drop; | ||
| else opts.onError = OnError::Abort; | ||
| // Resolved here so the runtime always gets a real worker count. |
Owner
Author
There was a problem hiding this comment.
you can remove this comment. the code and the function documentation is enough in this case.
Owner
Author
There was a problem hiding this comment.
Agreed, dropped in 1d909a6. The >= 1 invariant is already stated where it matters -- on RunOptions::jobs ("always >= 1 by the time run() sees it") -- and --jobs' own help text carries the "0 = one per CPU" half, so the std::max line reads fine on its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
-j/--jobs, spreading games across a pool of worker threads. The PGNreader stays a single sequential producer and dispatches each game as it
completes;
submit()blocks while every worker is busy, so memory stays boundedby the worker count no matter how large the database is.
src/game.cppis untouched:Builder::endPgnalready moved each game out andlet the next
startPgnallocate a fresh one, so the handoff was already safe totransfer across threads. The dispatch point is the visitor lambda in
Runtime::run.Design
A
sol::statecannot be entered by two threads, so aPipelineowns one Luastate plus its own copy of every plugin, its own engines and its own
ctx.scope.Runtimekeeps the options, the shared output writer, the writerregistry and N pipelines.
-j1is a strict specialization of the general path,not a separate branch: in-flight work is bounded by the worker count rather than
a queue length, so at
-j1submit(k+1)cannot return until gamekhasfinished and a
stopfromkis always seen beforek+1is dispatched.src/runtime.cpp(969 lines) is split intooptions/writer/pipeline/pool/runtimealong the dependency order.Behavior changes
ctx.sharedis removed. No bundled plugin used it, and a table cannot beshared between Lua states, so it could not have kept its documented meaning
under
-j.-j1games are written in completion order, not input order.ctx.openreturns the same writer for a path already open instead oftruncating it, so stages and workers can share a file. Reopening with the
other mode is now an error.
init/finishrun once per worker, sohistogram.luaanddedup.luaonly aggregate correctly at
-j1.ctx.worker/ctx.workersare exposed forthe simple cases (
csv.luauses them to write its header once); a generalreduce channel is recorded in TODO.md.
Incidental fix
Enginecalledstd::fprintf/std::strerrorbetweenfork()and_exit().Neither is async-signal-safe: with worker threads, one holding the stdio lock at
fork time would deadlock the child while it still held the pipe open, hanging
the parent's read. The message is now rendered before the fork and the child
issues a single
::write.Testing
93 ctest cases, up from 83.
-j1is byte-compared against the sequentialoutput;
-j4/-j8are compared as a multiset of lines since completion orderis not fixed. Separate cases cover the bounded stop, an abort raised on a worker
travelling back to the main thread intact, and two stages sharing one
ctx.openfile across every worker.Measured on 4800 games through a compute-bound plugin (32 cores): 8.47s at
-j1, 4.33s at-j2, 2.20s at-j4, 1.24s at-j8.ThreadSanitizer has not been run. A
TICTAC_SANITIZERbuild option iswired up and the flags reach both the compile and link lines, but no clang
sanitizer runtime is installed on the machine this was developed on
(
libclang-rt-19-dev), and g++-13 there lacks<print>. The suite was runrepeatedly instead, which is not equivalent. Worth a TSan pass before merge.