bench-incremental: add nix-dynamic tools (experimental builder) by aldoborrero · Pull Request #124 · numtide/go2nix

aldoborrero · 2026-04-16T09:46:27Z

Summary

Re-roll of #120 against the post-#121 buildTool interface.

Adds nix-dynamic and nix-dynamic-nocgo tools that build via buildGoApplicationExperimental (recursive-nix + dynamic-derivations + ca-derivations). Both share the existing nixTool plumbing — only the expression template and extraOpts differ.

The harness now probes each tool once at startup; if a tool's probe build fails and it opts in via SkipOnFail(), it's dropped with a SKIP notice and the rest of the run continues. So nix-dynamic* degrades gracefully on stores without the recursive-nix system feature; other tools still fail fast.

writeNixExpr now takes the template as an argument so the dag and experimental templates share the same code path.

Test plan

go build && go vet
nix build .#checks.x86_64-linux.bench-incremental-help
nix build .#checks.x86_64-linux.formatting
Smoke: --fixture light --tools nix-dynamic --runs 1 --scenario no_change → probe OK nix-dynamic, no_change 1.01s (eval 0.88s + build 0.13s)
--tools bogus → lists nix-dynamic, nix-dynamic-nocgo in available

github-actions · 2026-04-16T09:52:12Z

Benchmark Regression Check

Scenario	Tool	Base (s)	Current (s)	Change	Drvs (base)	Drvs (curr)	Status
no_change	nix-ca-nocgo	0.66	0.66	-0.1%	0	0	ok
no_change	nix-nocgo	0.67	0.66	-0.1%	0	0	ok
leaf-private	nix-ca-nocgo	1.86	1.95	+4.8%	2	2	ok
leaf-private	nix-nocgo	9.45	1.47	-84.5%	2	2	ok
mid-private	nix-ca-nocgo	2.01	2.00	-0.2%	2	2	ok
mid-private	nix-nocgo	1.50	1.56	+3.9%	4	4	ok
deep-private	nix-ca-nocgo	2.32	2.18	-6.4%	3	3	ok
deep-private	nix-nocgo	1.80	1.75	-2.7%	8	8	ok

Baseline: main | Current: 8514b4b1850fc6b00c9de2dc5d5a3693c6292c63

aldoborrero · 2026-04-16T10:28:30Z

❯ nix run .\#bench-incremental -- --tools nix-dynamic,nix-dynamic-nocgo,bazel --fixture torture --runs 5 --json comparison.json
Resolving Nix dependencies...
Probing tools...
  SKIP nix-dynamic: nix-store --realise failed: exit status 1
y/app-full: exit status 1"
       For full logs, run:
         nix log /nix/store/6jld0wfd5g0yk10m6w6wm26lwafjlh62-bench.drv.drv
error: build of resolved derivation '/nix/store/6jld0wfd5g0yk10m6w6wm26lwafjlh62-bench.drv.drv' failed
error: failed to obtain derivation of '/nix/store/dxs22a4cpf89imj28p9grmvzhynrfngk-bench.drv.drv^out'
error: Cannot build '/nix/store/6l2nx1kz40f0qasy9xj1l5il3nrxyiz2-bench-dynamic.drv'.
       Reason: 1 dependency failed.
error: Build failed due to failed dependency

  SKIP nix-dynamic-nocgo: nix-store --realise failed: exit status 1
y/app-full: exit status 1"
       For full logs, run:
         nix log /nix/store/s9win1f4pmk0gkgz7nz9r6pa58fvaqnw-bench.drv.drv
error: build of resolved derivation '/nix/store/s9win1f4pmk0gkgz7nz9r6pa58fvaqnw-bench.drv.drv' failed
error: failed to obtain derivation of '/nix/store/xabsz8sj6x8b2w5679600q0b3q05vvvy-bench.drv.drv^out'
error: Cannot build '/nix/store/lxdfja6bbkiidyk5xm7cwrscs81by86q-bench-dynamic.drv'.
       Reason: 1 dependency failed.
error: Build failed due to failed dependency

aldoborrero · 2026-04-16T10:29:26Z

That truncated error is discovering packages: go list in …/app-full: exit status 1 — the inner go2nix resolve reached the go list phase (so the host-store fix in f6b9767 worked), but go list -mod=readonly fails because:

app-full/go.mod has replace github.com/gin-gonic/gin => github.com/gin-gonic/gin v1.9.1
the local-replace target internal/web/go.mod requires gin v1.12.0
go list needs v1.12.0's .mod file in GOMODCACHE to walk the module graph, even though MVS+replace selects v1.9.1
app-full/go2nix.toml only has the FOD for v1.9.1, so v1.12.0 isn't staged → module lookup disabled by GOPROXY=off

This is a pre-existing dynamic-mode limitation (the torture fixture's conflict-a/conflict-b modules are designed to exercise exactly this version-conflict-via-replace pattern; dag mode doesn't hit it because it runs go list at eval time with the host's full GOMODCACHE). The fix is in go2nix generate: emit [mod-info] lockfile entries (go.mod-only hashes) for every version mentioned in the module graph, not just MVS-selected ones, and have resolve.go's setupGOMODCACHE stage those as cache/download/<path>/@v/<ver>.mod.

So this PR is correct as far as it goes (probe SKIPs cleanly, bazel still runs); the torture×dynamic comparison stays blocked until that lockfile-format fix lands. --fixture light --tools nix-dynamic works and shows dag beating dynamic ~2× on every scenario (no iface cutoff in dynamic mode).

aldoborrero · 2026-04-16T10:31:41Z

Correction: the lockfile-format change isn't needed. It's a GOWORK mismatch — the dag plugin sets GOWORK=off (resolve.rs:382), dynamic-mode resolve.go didn't, so workspace MVS ignored app-full's replace and picked v1.12.0. One-line fix in #125. With that, torture×dynamic should run.

Adds nix-dynamic and nix-dynamic-nocgo as buildTool entries that build via buildGoApplicationExperimental (recursive-nix + dynamic-derivations + ca-derivations). The harness probes each tool once at startup and drops a tool with a SKIP notice if its probe build fails — so the dynamic tools degrade gracefully on stores without recursive-nix while the rest of the run continues. writeNixExpr now takes the template as an argument so the dag and experimental templates share the same path. :house: Remote-Dev: homespace

The recursive-nix inner daemon serves the same store the outer build runs against. With NIX_REMOTE=local?root=..., the inner go2nix resolve registers FOD drvs whose inputDrvs aren't all present in the rooted store, so AddToStore fails with "path '...' is not valid". The host store has them (it's where the bench resolved nixpkgs/go2nix-cli from). Dynamic builds are still per-fixture-path so concurrent runs don't interfere; the dag tools keep using the rooted store for isolation. :house: Remote-Dev: homespace

aldoborrero · 2026-04-16T12:51:42Z

❯ nix run .\#bench-incremental -- --tools nix-dynamic,nix-dynamic-nocgo --fixture torture --runs 5 --json comparison.json
Resolving Nix dependencies...                                                                          Probing tools...
  SKIP nix-dynamic: nix-store --realise failed: exit status 1
g lines:
       > error: executing '/nix/store/sfvyavxai6qvzmv9p9x6mp4wwdz4v41m-bash-interactive-5.3p9/bin/bash': Argument list too long
       For full logs, run:
         nix log /nix/store/n6cck0zyzx51dykw98g57f9ryr2xl7w1-bench.drv
error: build of resolved derivation '/nix/store/n6cck0zyzx51dykw98g57f9ryr2xl7w1-bench.drv' failed
error: Cannot build '/nix/store/55z51nl5zkkfh4ws4927h4cx1jbkv0zv-bench-dynamic.drv'.
       Reason: 1 dependency failed.
error: Build failed due to failed dependency

  SKIP nix-dynamic-nocgo: nix-store --realise failed: exit status 1
g lines:
       > error: executing '/nix/store/sfvyavxai6qvzmv9p9x6mp4wwdz4v41m-bash-interactive-5.3p9/bin/bash': Argument list too long
       For full logs, run:
         nix log /nix/store/6bz6s1k1di8qc8gywhgnzg07zyskvd6x-bench.drv
error: build of resolved derivation '/nix/store/6bz6s1k1di8qc8gywhgnzg07zyskvd6x-bench.drv' failed
error: Cannot build '/nix/store/14czp5272wjdljd608lj8a6iv91j50dz-bench-dynamic.drv'.
       Reason: 1 dependency failed.
error: Build failed due to failed dependency

no runnable tools after probe

aldoborrero · 2026-04-16T12:54:12Z

That's the link-drv importcfg_entries env var (~400KB on torture — one packagefile line per transitive dep × 3270+) exceeding Linux's MAX_ARG_STRLEN (128KB). Fixed in #125 (865896b): passAsFile importcfg_entries so Nix writes it to a temp file instead of passing via env. Dag mode already does this for compileManifestJSON.

So torture×dynamic needs both #125 (the two resolve.go fixes) and this PR. With both applied I get:

scenario	nix-ca	nix-dynamic	winner
no_change	1.64s / 0 drvs	1.11s / 0	dynamic 1.5×
leaf	12.10s / 2	22.04s / 4	dag 1.8×
mid	11.80s / 2	22.24s / 5	dag 1.9×
deep	13.46s / 9*	24.00s / 18	dag 1.8×

* deep run-2=2 drvs (iface cutoff after first touch)

Dynamic wins the no-edit case (saves the ~0.5s derivationStrict eval floor); dag wins every touch (no iface cutoff in dynamic — deep is 18 drvs vs 2 — plus the wrapper drv reruns go list+drv-gen ~10s/touch).

aldoborrero · 2026-04-16T13:03:13Z

Clarification: #125 merged with only the GOWORK=off commit; the passAsFile fix raced the merge. Opened #127 for it. Once that lands and this PR is rebased, torture×dynamic runs to completion.

aldoborrero · 2026-04-16T14:04:26Z

Superseded by #128 (rebased onto main with #125/#127; opened fresh to avoid force-push).

aldoborrero requested a review from brianmcgee April 16, 2026 09:49

aldoborrero mentioned this pull request Apr 16, 2026

resolve: GOWORK=off + passAsFile importcfg_entries (dynamic-mode torture fixes) #125

Merged

aldoborrero added 2 commits April 16, 2026 12:08

brianmcgee force-pushed the aldo/bench-incremental-dynamic-v2 branch from f6b9767 to 4c0f5a1 Compare April 16, 2026 11:52

aldoborrero mentioned this pull request Apr 16, 2026

resolve: passAsFile importcfg_entries (dynamic-mode link drv exceeds MAX_ARG_STRLEN on torture) #127

Merged

aldoborrero mentioned this pull request Apr 16, 2026

bench-incremental: add nix-dynamic tools (experimental builder) #128

Open

aldoborrero closed this Apr 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bench-incremental: add nix-dynamic tools (experimental builder)#124

bench-incremental: add nix-dynamic tools (experimental builder)#124
aldoborrero wants to merge 2 commits intomainfrom
aldo/bench-incremental-dynamic-v2

aldoborrero commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026 •

edited

Loading

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

aldoborrero commented Apr 16, 2026

Summary

Test plan

Uh oh!

github-actions Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Regression Check

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

aldoborrero commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Apr 16, 2026 •

edited

Loading