Speed up loading bundles with many included files

[janniklasrose]

Summary

Each file in include was applied as its own mutator. Entering a mutator converts the whole configuration between its typed and dynamic representations (config.Root.MarkMutatorEntry → convert.FromTyped), so file N re-converted the N-1 resources already merged — making configuration load quadratic in the number of included files.

This came from a customer bundle with ~6000 jobs, one job per YAML file (the documented layout), plus ~18000 notebooks. Their CI spent ~29 minutes in configuration load before any API call.

Fix

Apply the per-file includes within the enclosing mutator scope, so the configuration is converted once per load rather than once per file. ProcessInclude merges through config.Root.Merge, which keeps both representations in sync, so it does not need a scope of its own.

The expanded include list now goes through Mutate for the same reason: with no per-file scope, nothing converts the typed field back into the dynamic tree afterwards, and the next ToTyped would restore the raw glob patterns. An empty list is dropped rather than written as [], since the typed field is omitempty.

Results

To size this, I generated bundles matching the customer's shape — real files on disk, one 8-task job per YAML file across nested directories, 4 wildcard include patterns, plus ~18000 notebook files at the 6000-job size. Measured with bundle validate (the load path, no deploy) on an Apple M4 Max:

Job files	Before	After	Speedup
3000	301 s	35 s	8.6x
4000	529 s	60 s	8.8x
5000	1237 s	93 s	13x
6000	1208 s	131 s	9.2x

At the customer's size, ~20 minutes becomes ~2 minutes (~89% less). Before the fix, doubling the file count roughly quadrupled the time; repeated runs at 3000 files spanned 301-316 s, so the growth is well outside measurement noise. A CPU profile at 800 files attributed 22% of samples to MarkMutatorEntry, nearly all under convert.FromTyped, plus the allocation and GC churn it caused.

Defining several jobs per file is an effective workaround for affected bundles and needs no CLI change: 1000 jobs one-per-file took 36 s, and the same jobs in a single file took 1.5 s.

What this does not fix

merge.Merge is independently quadratic — mergeMap allocates a new mapping and copies every existing entry, so file N also copies the N-1 resources already merged. That is untouched here and becomes the dominant term once this change lands, so load is much faster but not linear: at 100/200/400 files, time per file still rises from 91 µs to 120 µs to 159 µs.

A prototype that accumulates included resources into a single mapping instead of folding each file into the whole configuration was linear across 500-6000 files and reached ~1.2 s at 6000 files. It is deliberately not part of this PR: it has to preserve the merge semantics the current pairwise merge provides, and that deserves review on its own. Worth recording for whoever picks it up — a naive version that replaces instead of deep-merging on key collision drops fields from the earlier definition and collapses the accumulated locations, which also stops validate:unique_resource_keys from reporting duplicate resource keys, since it detects them by counting locations per key.

One behavior change worth flagging: per-include mutator timings are no longer recorded individually in telemetry; they roll up into ProcessRootIncludes.

Tests

Three unit tests in process_root_includes_test.go, covering what this change could plausibly break:

• the expanded include list is visible in both the typed and dynamic configuration (removing the per-file scope initially regressed this)
• an empty list stays absent from the dynamic tree, so bundle validate -o json does not gain an empty include (my first attempt at the above regressed this)
• merge semantics across included files are unchanged: per-key map merge with the later file winning, sequence concatenation, and accumulated locations

Full unit suite (6235 tests) and acceptance suite (2524) pass. bundle/templates/lakeflow-integrations fails identically on a clean tree in my environment (local Python/uv build issue), unrelated to this change.

This pull request and its description were written by Isaac.

[github-actions]

Approval status: pending

/bundle/ - needs approval

Files: bundle/config/loader/process_root_includes.go, bundle/config/loader/process_root_includes_test.go, bundle/mutator.go
Suggested: @denik
Also eligible: @pietern, @andrewnester, @lennartkats-db, @shreyas-goenka, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/include-load-performance.md
Based on git history:

• @denik -- recent work in .nextchanges/bundles/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: f448601

Run: 31163298763

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	4	4	291	1115	5:14
💚​	aws windows	4	4	293	1113	4:06
💚​	azure linux	4	4	290	1115	5:27
💚​	azure windows	4	4	292	1113	4:14
💚​	gcp linux	1	5	291	1115	4:22
💚​	gcp windows	1	5	293	1113	4:11

8 interesting tests: 4 RECOVERED, 4 SKIP

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
3:12	aws windows	TestAccept
3:06	gcp windows	TestAccept
3:02	azure windows	TestAccept