moderngekko-port: add one-command PGO workflow - #26
Open
dougchansan wants to merge 5 commits into
Open
Conversation
`pgo-run` builds an instrumented module, runs it so a developer can play a representative workload, merges the profiles that produces, validates them, and builds the final module against them. The profiled program is the generated per-game module, not moderngekko-port. The port is a build driver that runs for minutes; the module is the recompiled game, and it is where the frames go. Both backends are covered, and each needs its instrumentation in a different place. The C backend's chunks are C, so clang's -fprofile-generate and -fprofile-use= reach them through the module's compile flags. The LLVM backend emits and codegens IR in-process, so no compile flag ever reaches those chunks; instrumentation comes from DolRecomp's own IR-level PGO passes. In both cases -fprofile-generate also has to reach the module *link* line, which is where the profiling runtime comes from -- an instrumented module that never linked it writes no profile at all. PGO is modelled as a semantic build input rather than an ambient CFLAGS mutation around two Build() calls, and it is part of the module cache identity: the mode (off/generate/use) and, for use builds, the SHA-256 of the merged profile's *contents*. A plain module can no longer answer a PGO build, and two different profiles written to the same scratch filename no longer collide. The same profile moved elsewhere still reuses its module, because the path is not part of the identity. The same key gap existed for DolRecomp's codegen environment variables, which were not in the key at all, so those are folded in when set. A failed run must never leave an instrumented module selected. The generation build writes to a private cache root, and the PGO module becomes active only after every stage has succeeded. Any failure exits nonzero, names the stage, and leaves the previously active module untouched. Incidental to the above: - The command line moved out of main() into tools/port_command_line.hpp so the one path every command goes through is testable. - SHA-256 moved from src/runtime/game.cpp to a header, so a profile can be hashed without linking the runtime. - Process launching on Windows moved to the wide-character APIs. Quoting went through path::string(), which narrows via the active code page, so a cache under a path that code page cannot represent reached the compiler as question marks.
A real GM4E01 build died at 41/192 with ninja: error: WriteFile(...cpu_interpreter_integer.c.obj.rsp): Unable to create file. No such file or directory which says nothing about what was wrong: the module build's object and response files had gone past Windows' 260-character limit, and ninja does not use the \?\ extended-length prefix. A plain build already sits close to that limit, and pgo-run nests a second module cache under the workspace, which was enough to push an ordinary cache location over. Measure the longest path the module build would need before building anything and stop with a message naming --profile-dir, rather than twenty minutes later inside ninja. The workspace's generation cache is also renamed from generate-modules to gen, since it prefixes every one of those paths.
`llvm-profdata show` came back as 'C:\Program' is not recognized as an internal or external command so a merged profile with 1138 functions and a maximum count of 2.2 billion was reported as "no profile summary" and the run failed at validation. ReadCommand goes through _wpopen, which runs `cmd /c <command>`, and cmd drops the quotes around a program path containing spaces as soon as a second quoted argument follows it. One quoted argument is fine, which is why probing `llvm-profdata --version` worked and reading the summary did not. Wrapping the whole command in one more pair of quotes makes cmd strip exactly that pair and pass the rest through, which is what `cmd /c "..."` is documented to do. RunCommand was never affected: it builds the argument vector for CreateProcessW itself and no shell is involved, which is why the merge that produced the profile succeeded in the same run that failed to read it back. Also report the training run's exit status rather than only that it was nonzero. A game closed normally exits 0, and every other value is a different problem; RunCommand collapsed all of them to false.
llvm-profdata indents its version under the "LLVM (http://llvm.org/):" heading, so the manifest recorded llvm_profdata_version= LLVM version 22.1.5
dougchansan
force-pushed
the
feature/pgo-run
branch
from
August 11, 2026 03:58
782c463 to
8644750
Compare
dougchansan
marked this pull request as ready for review
August 11, 2026 04:20
…build Reported from a TimeSplitters 2 run: stages 1 to 5 all succeeded -- the module was instrumented, the game trained, and llvm-profdata merged and validated a 10,836-function profile -- and then the PGO build failed with error: .../merged.profdata: unsupported instrumentation profile format version repeated once per chunk worker, with nothing said about versions. There are three LLVMs in an LLVM-backend PGO build, not two. clang compiles the module's C sources and supplies the profiling runtime, llvm-profdata merges what that runtime writes, and the LLVM DolRecomp is linked against reads the result back inside PGOInstrumentationUse. The existing check compares the first two, which is why clang 22 and llvm-profdata 22 passed it, and the recompiler -- built against LLVM 19 or 20, as its CMakeLists requires -- rejected the profile they agreed on. The check cannot be extended to cover the third: DolRecomp does not report its own LLVM version, so there is nothing to compare against. It also fails late, because DolRecomp opens the profile eagerly but only parses it once emission has started. So say what happened instead. On a failed LLVM-backend PGO build, name the versions actually used, state that the pinned recompiler wants LLVM 19 or 20, and give the ldd invocation that shows which one it has. Note specifically that overriding --llvm-profdata alone does not fix it, since the C sources are compiled by whichever clang is on PATH and that clang provides the runtime -- the obvious first thing to try, and it does not work. README documents the same constraint up front. Thank you to @mooman for flagging this.
dougchansan
force-pushed
the
feature/pgo-run
branch
from
August 11, 2026 21:44
3a6f5a4 to
845c6d0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
moderngekko-port pgo-run, which does the whole instrumentation-PGO cycle in one command: build an instrumented module, run it so a developer can play a representative workload, merge and validate the profiles that produces, and build the final module against them.--backend cand--backend llvm.The profiled program is the generated per-game module, not
moderngekko-port. The port is a build driver that runs for a few minutes; the module is the recompiled game, and it is where every frame is spent.Usage
Full option list:
inspect,buildandrunare unchanged.Implementation
Workspace isolation. Everything a run produces lives under
--profile-dir, or under<output>/<disc-id>/pgowhen that is not given:raw/for the.profraw,gen/for the instrumented module's own private module cache,merged.profdata, andpgo-manifest.txt. The generation build is pointed at that private cache root specifically so it cannot write the realactive-module.txt.raw/is emptied at the start of every run, so a profile left by an earlier build cannot be merged into this one.Compile and link instrumentation. The module template has no hook of its own for extra flags, so the generation build passes
-DCMAKE_C_FLAGS=-fprofile-generateand-DCMAKE_SHARED_LINKER_FLAGS=-fprofile-generate. Both are needed: the profiling runtime that actually writes the.profrawcomes from the link line, and an instrumented module that never linked it produces no profile at all. This was verified against real GM4E01 module builds on both backends rather than assumed — see Testing.The use build passes
-fprofile-use=<merged.profdata>plus-Werror=profile-instr-out-of-date.-Wprofile-instr-unprofiledis deliberately not promoted: a training run that never entered a function is normal, and there are thousands of them.LLVM DolRecomp environment. The LLVM backend emits and code-generates IR in-process, so no compile flag ever reaches its chunks. Instrumentation comes from DolRecomp's own IR-level PGO passes, driven by
DOLRECOMP_LLVM_PGO=gen/=use,DOLRECOMP_LLVM_PROFILE=<merged>andDOLRECOMP_LLVM_PGO_STALE=error.-fprofile-generatestill reaches the link for the same reason as above. These are set through a scoped environment guard that restores in its destructor, so a failed stage cannot leakDOLRECOMP_LLVM_PGO=geninto later builds in the same shell.llvm-profdata discovery and validation. Tried in order:
--llvm-profdata,$LLVM_PROFDATA, beside the resolvedclang,clang --print-prog-name=llvm-profdata,xcrun --findon macOS, thenPATH. A candidate is only accepted if it prints an LLVM version banner — a nonexistent path still produces output, so nonempty is not proof. A known Clang/llvm-profdata major-version mismatch is a hard error; an unparseable banner skips the check rather than guessing.There is a third LLVM this cannot check. With
--backend llvmthe merged profile is written byllvm-profdataand read back by the LLVM that DolRecomp is linked against, and those are separate installations. The pinned recompiler builds against LLVM 19 or 20, so a distribution whose defaultclangis newer merges a profile the recompiler cannot parse. DolRecomp reports no version of its own, so there is nothing to compare against and the check cannot be extended to cover it. It also fails late: DolRecomp opens the profile eagerly, so a wrong path is caught immediately, but only parses it insidePGOInstrumentationUseonce emission has started. A failed LLVM-backend PGO build therefore prints the versions actually used, the LLVM 19/20 requirement, and thelddinvocation that shows which one the recompiler has — including the specific note that overriding--llvm-profdataalone does not fix it, since the C sources are compiled by whicheverclangis onPATHand that clang supplies the profiling runtime.After merging,
llvm-profdata showis parsed and the profile is rejected if it has no function records or a maximum function count of zero. IR instrumentation creates a record for every function whether or not it ran, so records-without-counts is exactly what an early flush looks like — a successful merge of a useless profile. No title-specific minimum is imposed; a small program with one hot function is a legitimate profile.Cache identity. The module cache key gains the PGO mode (
off/generate/use) and, for use builds, the SHA-256 of the merged profile's contents plus the stale-profile policy. The profile path is deliberately excluded: the same profile copied elsewhere is the same build, and two different profiles written to the same scratch filename are not.The same gap existed for DolRecomp's code-generation environment (
DOLRECOMP_LLVM_CPU,DOLRECOMP_LLVM_FEATURES,DOLRECOMP_LLVM_TARGET,DOLRECOMP_LLVM_CHUNK_INSTRUCTIONS,DOLRECOMP_C_CHUNK_INSTRUCTIONS,DOLRECOMP_DISPATCH_LOOKUP,DOLRECOMP_UNSAFE_DIRECT_CALLS), none of which were in the key at all, so a module built withDOLRECOMP_LLVM_CPU=znver3answered a later build with it unset. Those are folded in when set, and contribute nothing when unset so a default environment keeps the cache entries it already has.Active-module transaction safety. The PGO module becomes active only after the instrumented build, a clean training exit, raw-profile discovery, the merge, validation, the PGO build and final module validation have all succeeded. Any failure exits nonzero, names the stage, leaves the previously active module untouched, and keeps the work files for diagnosis. A forced kill of the training process is treated as a failure, not a successful run — the profiling runtime flushes on normal shutdown and skips it on a kill.
Manifest. The module's
manifest.txtand the workspace'spgo-manifest.txtrecordpgo_mode,pgo_backend,pgo_profile_sha256,pgo_profile_path,pgo_raw_profile_count,pgo_total_function_count,pgo_max_function_count,pgo_stale_policy,clang_versionandllvm_profdata_version.Incidental changes
Three things were in the way and are separated into their own commits or clearly scoped:
tools/port_command_line.hpp— the command line moved out ofmain(). It is the one path every command goes through and no test could reach it;runforwards unknown arguments whilebuildrejects them, andpgo-runadds three options that must not leak into the others.include/moderngekko/sha256.hpp— SHA-256 moved out ofsrc/runtime/game.cppso a profile can be hashed from a translation unit that does not link the runtime. A second copy of a hash function stays correct right up until one copy is fixed.path::string(), which narrows via the active code page, so a module cache under a path that code page cannot represent reached the compiler as question marks.Testing
Automated tests
Two new standalone test targets, neither of which needs a game or Dolphin linkage, so both run in the existing fast cross-platform CI job:
cmake --build build --target moderngekko_pgo_support_test cmake --build build --target moderngekko_port_command_line_test ctest --test-dir build -R "moderngekko\.(pgo_support|port_command_line)" --output-on-failuremoderngekko.pgo_supportcovers cache identity (off vs generate vs use all differ; two profile digests at one path differ; one digest at two paths does not; the stale policy is included),llvm-profdata showparsing (valid summary accepted; zero maximum count, zero records, malformed numbers, an error message and a missing field all rejected), raw-profile discovery (finds nested.profraw, ignores.profdataand.txt, handles a directory namedraw profiles プレイヤー, returns empty for a missing directory), workspace derivation, manifest fields, llvm-profdata candidate ordering, path formatting, and the Windows path-length estimate.moderngekko.port_command_linecovers the parsing that every command goes through:buildstill rejects a stray--load-state,runstill forwards it,--still stops option parsing, the three pgo-run options do not exist forbuildand are forwarded byrun, and backend/opt-level validation is unchanged.Full project build and tests
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release cmake --build build ctest --test-dir build --output-on-failure --timeout 300 -E "fullbench|fuzzer|zstreamtest"Result: build exit 0, 45/45 tests passed, 0 failed. (
codegen_compilefails if ctest is run from a shell without the MSVC environment, because the nested CMake it spawns findscl.exewith noINCLUDE/LIB; inside a developer prompt it passes. This is unrelated to the change.)End-to-end, C backend, real game
Run against a locally extracted GM4E01 (Mario Kart: Double Dash). No game data is committed and none was downloaded.
Command exit status 0. Reported:
Checked against the artefacts on disk rather than only the summary:
-fprofile-generatereaches compilationFLAGS = -fprofile-generate -O3 ... -flto=thin -O2 -ffp-contract=offon all 191 translation units in the generatedbuild.ninjaLINK_FLAGS = -shared -fprofile-generate -fuse-ld=lld-link.profrawis writtenTotal functions: 1138,Maximum function count: 2233771529,Total count: 93602657813-Werror=profile-instr-out-of-dateand produced one unrelated-Wcommentwarning from GXRuntime and nothing elsee60ab604...(140 MB) vs4639573a...(79 MB)...-7e2807c1130bafbevs...-d2fdc6f1b4437e35active-module.txtnames the PGO module-d2fdc6f1b4437e35moduleactive-module.txtwas ever created; work files were kept each timeThe training run was closed with
WM_CLOSE— a normal window close, not a kill — because that distinction is the point: the profiling runtime flushes on the normal shutdown path.Three defects were found by this end-to-end run and fixed in their own commits; none of them could have been caught by unit tests:
ninja: error: ... Unable to create file. No such file or directory.llvm-profdata showreturned'C:\Program' is not recognized as an internal or external command, so the 1138-function profile above was rejected as empty._wpopenrunscmd /c, which drops the quotes around a spaced program path once a second quoted argument follows.End-to-end, LLVM backend, real game
Same game,
--backend llvm, withmoderngekko-portbuilt-DDOLRECOMP_ENABLE_LLVM=ONagainst LLVM 20.1.8, and Clang andllvm-profdata20.1.8 onPATHso they match the LLVM DolRecomp links.Command exit status 0.
dolllvm: PGO use, profile C:\mgpgo\llvm\pgo\merged.profdata (7b5595c7b89c0384/23640128)dolllvm: PGO profile match: 11606/11606 functions matched, 0 unmatched across 0 stale chunks, underDOLRECOMP_LLVM_PGO_STALE=error. This is DolRecomp's positive check — every function that matched its record carries entry-count metadata — so a full match is a direct statement that the profile and the emitted IR agree-fprofile-generatereaches compilation and linkingFLAGS = -fprofile-generate -O3 ...andLINK_FLAGS = -shared -fprofile-generate -fuse-ld=lld-linkin the module build, alongside the 5,803 objects DolRecomp had already instrumented.profrawis writtenef42ccdc...(413 MB) vsb296f53f...(327 MB)...-9b8e223a7de7c7ffvs...-bdbf0415b1d74eb9, and both differ from the C backend's keys for the same gameactive-module.txtnames the PGO module-bdbf0415b1d74eb9moduleThe resulting
pgo-manifest.txt:Reported from the field
Thank you to @mooman for flagging this.
After the runs above, a TimeSplitters 2 (GTSE4F)
--backend llvmrun on Linux failed at stage 6 withunsupported instrumentation profile format version, repeated once per chunk worker. Stages 1-5 had all succeeded, including a merged, validated 10,836-function profile: Clang andllvm-profdatawere both 22.1.8 and agreed with each other, and the LLVM inside DolRecomp — 19 or 20 — rejected what they produced. That is the third-LLVM case described under Implementation, and it is what the added diagnostic and README section are for.Worth noting that the transaction safety behaved correctly under a real failure that nobody engineered: the run exited nonzero, named the stage, left the previously active module untouched, and kept the work files.
Platforms
Limitations
llvm-profdataare required. GCC and MSVC fail immediately with an explanation rather than quietly producing an unprofiled module.--backend llvmadditionally requires Clang andllvm-profdatato match the LLVM DolRecomp was built against (LLVM 19 or 20 for the pinned recompiler). This one is not checked, because DolRecomp does not report its LLVM version; it is documented in the README and explained on failure. The proper fix is a--versionin DolRecomp that printsLLVM_PACKAGE_VERSION, which would let this check run in a second instead of failing ten minutes into a build — that belongs in a DolRecomp change, not here.