build: prototype ThinLTO deadcode planning and size tuning

[luoliwoshang]

Summary

Prototype a complete ThinLTO-compatible deadcode pipeline for LLGo's Go
method-table pruning, then tune the Darwin LLVM 19 ThinLTO pipeline until the
result is smaller than the existing non-LTO -deadcodedrop path in the tested
programs.

This PR is self-contained and based on main. It includes:

1. package Meta production and one global deadcode planner;
2. package-owned method-table rewriting before ThinLTO summary emission;
3. regeneration of package ThinLTO bitcode and temporary linker archives;
4. a size-oriented ThinLTO import budget for the deadcode mode;
5. Darwin LLVM 19 SLP recovery for optimized ThinLTO pipelines;
6. linker optimization-level compatibility for Os/Oz.

The resulting model is:

package LLVM modules + package Meta
    -> global Meta summary
    -> global deadcode.Plan
    -> rewrite each owning package module
    -> emit rewritten ThinLTO bitcode + summary
    -> build temporary package archives
    -> normal LLVM ThinLTO index/import/internalize/backend pipeline

LLGo remains responsible for Go-specific reachability. LLVM receives the
already-rewritten package modules and remains responsible for ThinLTO and
subsequent cross-module optimization.

Motivation

The existing non-ThinLTO -deadcodedrop path emits same-name strong globals in
the entry module to override package-owned weak method tables. That mechanism
does not compose with ThinLTO's module summaries and symbol resolution.

With:

-lto=thin -deadcodedrop

the strong-override experiment previously crashed LLVM 19.1.7 in:

FunctionImportGlobalProcessing::processGlobalForThinLTO

The replacement global also lives outside the package module that owns the
original weak_odr definition, COMDAT, and ThinLTO summary identity. Teaching
that override mechanism more ThinLTO special cases would preserve the wrong
ownership boundary.

This PR instead computes one global Go reachability plan, then applies the plan
inside each package module before that module's ThinLTO summary is written.

Design

Global planner

internal/deadcode.BuildPlan consumes the merged package Meta summary and root
set and returns an explicit plan:

type Plan struct {
    LiveSlots map[string][]int
}

The current Meta analysis remains the source of Go-specific reachability facts.
The pipeline boundary does not require the current algorithm to remain fixed:
future work can add reflection facts, string-flow information, or other planner
inputs without restoring link-time strong overrides.

Package-level rewrite

internal/dcepass.RewriteTypeMethodTables applies the global plan to the LLVM
module that owns each method table.

For dead method slots, it replaces IFn/TFn targets with
runtime.unreachableMethod. The original global stays in the original package
module and preserves its:

• ABI-compatible initializer layout;
• weak_odr linkage;
• COMDAT membership;
• package/module identity used by ThinLTO.

No same-name strong duplicate is emitted in the entry module for this mode.

Build integration and bitcode regeneration

The ThinLTO deadcode path is enabled only for:

-lto=thin -deadcodedrop

Package LLVM modules are kept alive until linkMainPkg has collected all Meta
and built the link-specific plan. materializeThinLTODeadcode then:

1. rewrites every package-owned method table;
2. emits fresh ThinLTO bitcode from the rewritten module;
3. emits a fresh ThinLTO summary describing the rewritten references;
4. normalizes the rewritten object into a temporary package archive;
5. rebuilds the final package input list before invoking the linker.

This ordering matters. Rewriting after summary emission would leave LLVM
analyzing stale edges: the summary could retain a method target that the IR had
already replaced with runtime.unreachableMethod.

The first prototype deliberately disables package-cache hits in this combined
mode. Cache overlays and immutable source bitcode are follow-up work.

ThinLTO import budget

LLVM's default ThinLTO import budget is performance-oriented. Imported bodies
also duplicate LLGo funcinfo entry sites. The combined ThinLTO deadcode mode
uses:

-Wl,-mllvm,-import-instr-limit=5

This retains very small cross-package imports while avoiding the text and
funcinfo growth observed with the default import budget.

Size optimization levels

ld64.lld accepts numeric --lto-O0..3 flags and rejects --lto-Os/Oz.
LLGo now passes a linker optimization flag only for numeric levels. Os and
Oz still select the corresponding LLGo pre-link pipeline, while the linker
uses its supported default backend level.

Darwin ThinLTO SLP recovery

The K8s experiment exposed a separate LLVM 19 Mach-O LLD pipeline problem.

LLVM 19 PipelineTuningOptions default to:

LoopVectorization = true;
SLPVectorization = false;

ELF LLD explicitly enables both from the LTO optimization level, but LLVM 19
Mach-O LLD does not set PTO.SLPVectorization. LLGo's
thinlto-pre-link<O2> pipeline intentionally defers SLP to the backend, so
Darwin ThinLTO never runs the pass.

LLVM main now contains the missing Mach-O assignments:

• LLVM 19 ELF configuration
• LLVM 19 Mach-O configuration
• current Mach-O configuration

For LLVM 19 compatibility, Darwin ThinLTO O2, O3, and Os package
pipelines now append:

function(slp-vectorizer)

Linux, FullLTO, non-LTO, O1, and Oz pipelines are unchanged. Once LLGo
moves to an LLVM version containing the upstream fix, post-link SLP is
preferable because it can also see imported code.

SLP root-cause evidence

The dominant K8s regression was:

crypto/internal/fips140/nistec.init

The Go standard library embeds an 88,064-byte P-256 precomputed table. The
retained ThinLTO pre-link module contained:

88,064 x store i8
0 x llvm.memcpy

Individual pass probes against the exact package bitcode produced:

Pass/pipeline	nistec.init result
instcombine	88,064 scalar stores
memcpyopt	88,064 scalar stores
vector-combine	88,064 scalar stores
slp-vectorizer	5,504 <16 x i8> vector stores
default<O2>	5,504 vector stores
default<Os>	5,504 vector stores
default<O1>	88,064 scalar stores
default<Oz>	88,064 scalar stores

The real linker command contained -flto=thin and --lto-O2. A single-job
--lto-debug-pass-manager trace showed the complete O2 backend pipeline,
including LoopVectorizePass, but zero SLPVectorizerPass executions.
nistec.init stayed at 88,079 IR instructions through the backend.

Without SLP, code generation emitted repeated mov plus strb/strh/str
instructions. With SLP it emitted constant-pool ldr q and stp q sequences.
The function's estimated machine-code range fell from 767,116 bytes to 77,244
bytes.

Import budgets 0 and 5 produced the same 767,116-byte function before the SLP
fix, proving that cross-module importing was not the primary cause.

Size results

Environment for the final measurements:

macOS arm64
LLVM 19.1.7
O2
LLGO_BUILD_CACHE=off
-a (forced package rebuild)
PCLN/site information enabled

Four demos

The baseline is non-ThinLTO without deadcode pruning. Existing DCE is the
current non-ThinLTO strong-override implementation. New is the complete pipeline
in this PR.

Demo	No-DCE baseline	Existing DCE	New ThinLTO+DCE	New vs existing DCE
goimporter-1389	5342.0 KiB	3881.6 KiB	3716.4 KiB	-165.2 KiB / -4.26%
embedunexport-1598	3904.5 KiB	2514.4 KiB	2429.8 KiB	-84.6 KiB / -3.37%
mimeheader	2201.0 KiB	1613.3 KiB	1372.3 KiB	-241.0 KiB / -14.94%
gotypes	3938.2 KiB	3269.1 KiB	3174.5 KiB	-94.6 KiB / -2.90%

All four final binaries exited with status 0. mimeheader printed the expected
host value and the complete gotypes demo finished successfully.

Single forced-build wall-time samples for the final binaries were 30.55 s,
24.46 s, 20.99 s, and 21.78 s respectively. These are diagnostic samples, not
reported as benchmark medians.

K8s workqueue

Benchmark source:

k8s.io/client-go/util/workqueue@v0.22.2
xgo-dev/benchmarks commit 94c0229de770b3b58dcb133c3dea60c4435c4a00

Mode	Total bytes	__text	nistec.init	__llgo_fie	__LINKEDIT
Existing non-LTO DCE	7,601,824	1,629,080	77,244	59,040	2,244,608
ThinLTO+DCE before size fixes	8,036,320	2,575,756	767,116	147,328	1,851,392
Complete pipeline in this PR	7,259,760	1,629,720	77,244	147,744	1,851,392

Compared with ThinLTO+DCE before the import/SLP tuning:

• total file size: -776,560 bytes / -9.66%;
• __text: -946,036 bytes / -36.73%;
• nistec.init: -689,872 bytes / -89.93%.

The final binary is 342,064 bytes (4.50%) smaller than the existing non-LTO
DCE binary. ThinLTO's __llgo_fie remains larger, but its smaller
__LINKEDIT and restored text optimization more than compensate in this case.

The K8s test binary still exits during startup with the existing:

fatal error: unreachable method called. linker bug?

The existing non-LTO DCE binary fails the same way. K8s is therefore currently
a build-size sample, not a runtime-correctness result.

Correctness validation

The package-owned rewrite path preserves linkage/COMDAT and has focused tests
for method-table initializer replacement. The ThinLTO combination also builds
and runs the interface/reflection cases used during the prototype:

globaldce_interface_matrix
globaldce_interface_slots
globaldce_reflect_method
globaldce_reflect_type_method
globaldce_typeid_dce
globaldce_unexported_method_identity

A small interface experiment removed all three dead Drop symbols while
preserving output:

Metric	ThinLTO baseline	ThinLTO + planner DCE
File size	122,112 B	121,328 B
__text	0x51a4	0x506c
Drop symbols	3	0

Known limitations

• Package cache use is temporarily disabled for ThinLTO + deadcode.
• The prototype regenerates temporary package archives but does not yet read or
  write cached rewritten archives.
• Reusing one in-memory package module for multiple entry-point plans is not
  supported; the original method table must become immutable or reloadable.
• ThinLTO backend cache directories are not wired into the build yet.
• The planner uses the current Meta reachability algorithm.
• MethodByName string/control-flow propagation is out of scope.
• Oz controls the LLGo pre-link pipeline, but an end-to-end size-oriented
  ThinLTO backend mode is not available through LLVM 19 ld64.lld.
• The Darwin pre-link SLP workaround should be revisited after the LLVM
  toolchain includes the upstream Mach-O LTO fix.

Follow-ups

• Make package bitcode immutable or reloadable so multiple plans can be applied
  independently.
• Define cache keys for the global plan and rewritten package bitcode.
• Wire a ThinLTO backend cache directory into the final link.
• Extend planner inputs when reflection/string-flow analysis is ready.
• Revisit temporary archive construction after the architecture is validated.
• Investigate and fix the remaining unreachable method called K8s startup
  failure before treating that benchmark as runtime validation.

Tests

Passed on the complete branch:

go test ./internal/build ./internal/crosscompile ./internal/deadcode ./internal/dcepass
go test -tags=dev ./internal/build -run '^(TestLLVMPassPipeline|TestThinLTODeadcode(LinkerArgs|Enabled)|TestDeadcodeDropEnabled)$' -count=1

git diff --check upstream/main...HEAD also passes.

[github-actions]

LLGo baseline benchmarks

a57aedc97cbc | workflow run | long-term charts

Program measurements

Platform	Workload	File size	vs base	Build	vs base	Run	vs base
Linux	cprintf	18656 B	+0.0%	312.802 ms	+4.9% (worse)	1.343 ms	+4.5% (worse)
Linux	fmtprintf	1874264 B	+0.0%	2.661 s	-0.3% (better)	3.311 ms	-0.0% (better)
Linux	println	67992 B	+0.0%	302.758 ms	+3.8% (worse)	1.650 ms	+2.1% (worse)
macOS	cprintf	84624 B	+0.0%	337.148 ms	-13.7% (better)	2.308 ms	-49.4% (better)
macOS	fmtprintf	1889968 B	+0.0%	2.657 s	+25.2% (worse)	13.611 ms	+29.6% (worse)
macOS	println	121168 B	+0.0%	324.737 ms	-1.8% (better)	3.675 ms	-20.2% (better)

Core language and compiler benchmarks

Platform	Benchmark	ns/op	vs base
Linux	BenchmarkLookupPCRandom	13.380 ns/op	+0.8% (worse)
Linux	BenchmarkMergeCompilerFlags	150.700 ns/op	-0.1% (better)
Linux	BenchmarkMergeLinkerFlags	94.410 ns/op	+0.1% (worse)
Linux	BenchmarkChannelBuffered	35.090 ns/op	+0.3% (worse)
Linux	BenchmarkChannelHandoff	27052 ns/op	-4.2% (better)
Linux	BenchmarkDefer	46.900 ns/op	+2.9% (worse)
Linux	BenchmarkDirectCall	1.558 ns/op	+0.1% (worse)
Linux	BenchmarkGlobalRead	1.557 ns/op	+0.1% (worse)
Linux	BenchmarkGlobalWrite	2.502 ns/op	+0.6% (worse)
Linux	BenchmarkGoroutine	33844 ns/op	-20.3% (better)
Linux	BenchmarkInterfaceCall	7.785 ns/op	+0.0% (worse)
Linux	BenchmarkRuntimeGetG	2.494 ns/op	+0.0% (worse)
macOS	BenchmarkLookupPCRandom	11.870 ns/op	-14.5% (better)
macOS	BenchmarkMergeCompilerFlags	119 ns/op	-14.1% (better)
macOS	BenchmarkMergeLinkerFlags	82.520 ns/op	-18.2% (better)
macOS	BenchmarkChannelBuffered	21.610 ns/op	+5.0% (worse)
macOS	BenchmarkChannelHandoff	6694 ns/op	-0.4% (better)
macOS	BenchmarkDefer	29.010 ns/op	+14.3% (worse)
macOS	BenchmarkDirectCall	1.104 ns/op	+16.8% (worse)
macOS	BenchmarkGlobalRead	1.029 ns/op	+7.9% (worse)
macOS	BenchmarkGlobalWrite	1.012 ns/op	-5.8% (better)
macOS	BenchmarkGoroutine	30646 ns/op	-4.5% (better)
macOS	BenchmarkInterfaceCall	4.774 ns/op	+16.5% (worse)
macOS	BenchmarkRuntimeGetG	2.075 ns/op	+9.9% (worse)

Compared with 9a344b47dddb measured in the same runner job.

[codecov]

Codecov Report

❌ Patch coverage is 57.81250% with 54 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/build/build.go	31.34%	40 Missing and 6 partials ⚠️
internal/dcepass/dcepass.go	83.33%	4 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!