[Generator][Experimental] Add opt-in dependency-aware sharded Types generation#866
Draft
jpsim wants to merge 2 commits intoapple:mainfrom
Draft
[Generator][Experimental] Add opt-in dependency-aware sharded Types generation#866jpsim wants to merge 2 commits intoapple:mainfrom
jpsim wants to merge 2 commits intoapple:mainfrom
Conversation
Add foundational infrastructure for dependency-aware sharded code generation without enabling the feature. * `GraphAlgorithms.swift`: iterative Tarjan SCC, topological sort via min-heap, condensation DAG, longest-path layering, and LPT bin-packing * `ShardingConfig` in `Config.swift` with `typeShardCounts`, `operationLayerShardCounts`, validation, and optional `modulePrefix` for deterministic file naming * `StructuredSwiftRepresentation.file` replaced with `.files` array for multi-file output (backward-compatible via `init(file:)`) * `ImportDescription` gains `exported` flag; renderer emits `@_exported` prefix when set * `HeapModule` added as a dependency from swift-collections * Comprehensive tests for all graph algorithms
Add opt-in sharding that splits generated Types and Operations output across multiple files based on schema dependency layers, enabling parallel compilation in consuming build systems. * `SchemaDependencyGraph`: builds a dependency graph from `OpenAPI.ComponentDictionary<JSONSchema>`, computes SCC layers, and extracts per-operation schema references * `TypesFileTranslator` gains `translateFileSharded`, `translateSchemasSharded`, `translateOperationsSharded`, `ShardNamingStrategy` (default and prefixed), and `ShardImportResolver` for cross-shard `@_exported` imports * `runShardedGenerator()` in `GeneratorPipeline.swift` for multi-file output * `--sharding` CLI flag and `sharding` config file key * Remove verbose per-file log messages from `replaceFileContents` * Tests for sharding invariants, determinism, naming contracts, and end-to-end generation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds an experimental, fully opt-in sharded
Typesgeneration mode intended for very large OpenAPI specs.Default behavior is unchanged. This is an additive path for advanced users who need better generated-code compile times and are willing to adopt custom build integration.
Why experimental
This feature targets a narrow, high-scale use case and introduces additional generation topology (layering + sharding + inter-file imports). Marking it experimental lets us:
Real-world motivation / observed impact
In our production use case (large spec, filtered subset), generated output compilation was a major bottleneck.
With dependency-aware layering + sharding, we observed >2x faster compilation of generated output.
Example environment and workload:
These results are workload- and build-system-dependent, but they motivated proposing this upstream as an optional path for similarly large deployments.
Benchmark snapshot (real-world workload)
The chart below shows compile time of generated output in our production project (not generator runtime), measured on the same filtered spec input.
TypesoutputScope of this PR
ShardingConfigtoConfig(opt-in only), with validation.Typesoutput path (root + component/type/operation shard files).Non-goals
Stability / compatibility
shardingis not set, behavior is unchanged.Maintenance posture