Add feasibility study for agent-driven PR backlog processing - #8
Open
dsarno wants to merge 9 commits into
Open
Conversation
Measures what it would take to evaluate, triage, revise and resolve the open pull request backlog using agents, grounded in one pull request carried end to end rather than in estimates. Includes the raw measurements (diff-size distribution over 119 real pull request heads, backlog age profile, build timings, per-stage token costs) and an interactive model for varying the assumptions.
Bounded audit of the string subsystem as a second workstream alongside pull request processing: 29 findings, two of them memory-safety or correctness bugs reachable from script. Includes a reproduction of the integer-overflow guard defect run against a locally built engine.
Records the hexagonal grid cell leg (6,875 commits of drift, 331k tokens, two high-severity defects found) and the string subsystem audit, and notes the cross-leg observation that in both completed legs the rebase was routine while the value was in the review.
Records the shared hash-table leg: 224k tokens, four defects fixed, three pre-existing ones correctly scoped out, and a benchmark that honestly reports the refactor as performance-neutral rather than as a speedup. Also records the leg's most transferable finding: the change's test file was named .h while the build globs .cpp, so its tests had never been compiled.
Adds the shared hash-table leg and the two findings from it that generalise: a change whose tests were never compiled because the file extension did not match the build glob, and a benchmark that contradicted the change's own premise and was reported regardless. Expands the recommended quality gates from five to seven accordingly.
Actions cannot be enabled from this environment (the proxy blocks the endpoint), so Godot's static-check scripts were run locally against every changed file on all three branches instead; all pass. Also records a hazard found while doing it: several of those scripts rewrite files in place and are scoped only by a regex in the pre-commit config, so invoking them directly without that filter corrupts source while reporting success.
An earlier draft counted unique content processed rather than billed tokens, understating volume by about 100x and cost by about 12x. Context is resent on every call, so a pull request whose unique material is ~300k tokens bills 25-44M. Replaces the estimates with measurements taken from the API usage fields over 1,224 calls: 96.2 percent of input is cache reads, mean context per call is 243k, and output is 0.24 percent of spend. Cache-read price, not the headline input rate, is what governs cost, which reorders the model comparison. Adds the full per-component log and updates the calculator.
The feasibility study evaluated this as a cost question. The actual purpose is to produce reproducible evidence that agent-driven work can help an open-source engine get through a stalled queue and improve the engine along the way, for an audience of maintainers who have good reason to be wary of AI contributions. Adds the project brief: ground rules, what 'better' means and how to measure it against stock, milestones with stated-in-advance stop criteria, and the three artefacts that keep findings and decisions out of commit logs. Records the correction that the fork is not yet better than upstream, and that the audit workstream rather than the backlog work is what generates that claim.
Two attempts at a local ASan+UBSan build were both OOM-killed silently, stopping mid-compile with no error line. Records the configurations tried, why 15 GB is not enough, and the generalisable point: a log tail cannot distinguish a working build from a dead one, so a pipeline running builds at scale needs process-count and artifact-mtime liveness checks with a stall timeout. Notes that sanitizer coverage for the hash-table work already exists from a targeted differential fuzzer, and that whole-engine sanitizer coverage belongs in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A feasibility study, under
pr-backlog-study/, for using agents to evaluate, triage, revise and resolve the open pull request backlog. Analysis only — no engine code changes on this branch.It is grounded in one pull request carried end to end on this fork rather than in estimates. That branch is a separate PR.
The headline
Compute is not the constraint. Processing the whole backlog is roughly 880M tokens — about $90 on the cheapest capable open-weight model, $6,000 on Claude Opus 5. Agent wall-clock is under three days at 50-way concurrency; builds are ~640 machine-hours.
The binding constraint is the human audit gate: ~425 person-hours, about ten months at ten hours a week, roughly 110× the agent time. Model choice should be driven by defect-catch rate, not price — the price spread is 130× but the absolute numbers are small next to the human time.
What was measured, not assumed
What the pilot found
The one leg carried to completion surfaced five real defects in code that had been in review for three years, including a state-corruption bug where a validation-ordering mistake could leave the physics server permanently in its "flushing" state, silently disabling monitoring guards engine-wide.
A second candidate was researched and the correct answer turned out to be not to rebase it — it needs reimplementation, and several of its files no longer exist at those paths. That verdict is the product, not a failure, and any cost model assuming every PR gets rebased is wrong.
Two findings worth acting on
workflow_dispatchreturns403via API. All pilot verification was local. Enabling Actions is the highest-value setup step for scaling — it swaps a 4-core box for the full 7-platform matrix with the sanitizer builds.Contents
pr-backlog-study/README.mdpr-backlog-study/measurements.mdpr-backlog-study/calculator.htmlGenerated by Claude Code