(Markdown) code generation

[Seth Fitzsimmons (mojodna)]

Summary

Add overture-schema-codegen, a code generator that produces documentation from
Pydantic schema models.

Pydantic's model_json_schema() flattens the schema's domain vocabulary into JSON
Schema primitives. NewType names, constraint provenance, and custom constraint classes
disappear. Navigating Python's type annotation machinery -- NewType chains, nested
Annotated wrappers, union filtering, generic resolution -- is complex. The codegen
does it once. analyze_type() unwraps annotations into TypeInfo, a flat
target-independent representation that renderers consume without re-entering the type
system.

Architecture

Four layers with strict downward imports:

Rendering            ← Output formatting, all presentation decisions
Output Layout        ← What to generate, where it goes
Extraction           ← TypeInfo, FieldSpec, ModelSpec, UnionSpec
Discovery            ← discover_models() from overture-schema-core

analyze_type() is the central function. A single iterative loop peels NewType,
Annotated, Union, and container wrappers in fixed order, accumulating constraints tagged
with the NewType that contributed them. The result is a TypeInfo dataclass that
downstream modules consume without re-entering the type system.

Both concrete BaseModel subclasses and discriminated union type aliases (like Segment = Annotated[Union[RoadSegment, ...], ...]) satisfy the FeatureSpec protocol and flow
through the same pipeline. Union extraction finds the common base class, partitions
fields into shared and variant-specific, and extracts the discriminator mapping.

markdown_pipeline.py orchestrates the full pipeline without I/O: tree expansion,
supplementary type collection, path assignment, reverse references, and rendering.
Returns list[RenderedPage]. The CLI writes files to disk with Docusaurus frontmatter.

Design doc: packages/overture-schema-codegen/docs/design.md

Changes outside the codegen package

Preparatory fixes and refactors in core/system/CLI packages:

• Rename ModelKey.class_name to entry_point (carries module:Class path, not just the
  class name)
• Attach docstrings to NewTypes at runtime (so the codegen can extract them)
• Add resolve_discriminator_field_name() to system feature module
• Fix relative imports and f-string prefixes in core
• Use dict instead of Mapping in system test util type hints

Example (real) data added to theme pyproject.toml files (addresses, base, buildings,
divisions, places) under [examples.ModelName] sections.

What's in the package

Source:

Module	Purpose
type_analyzer.py	Iterative type unwrapping into TypeInfo
specs.py	Data structures shared between extraction and rendering
type_registry.py	Type name → per-target display string mapping
model_extraction.py	Pydantic model → ModelSpec, tree expansion
union_extraction.py	Union alias → UnionSpec, discriminator mapping
enum_extraction.py	Enum → EnumSpec
newtype_extraction.py	NewType → NewTypeSpec
primitive_extraction.py	Numeric primitives and geometry types
field_constraint_description.py	Constraint objects → display text
model_constraint_description.py	Model-level constraints → prose
module_layout.py	Python module paths → output directories
type_collection.py	Supplementary type discovery from field trees
path_assignment.py	Type names → output file paths
link_computation.py	Relative links between output pages
reverse_references.py	"Used By" reference computation
markdown_type_format.py	TypeInfo → markdown type strings with links
markdown_renderer.py	Jinja2 template driver for all page types
example_loader.py	TOML example loading, validation, flattening
markdown_pipeline.py	Pipeline orchestration (no I/O)
cli.py	Click CLI: generate and list commands
case_conversion.py	PascalCase → snake_case
docstring.py	Custom vs. auto-generated docstring detection

Tests: unit tests per module, golden file tests for
rendered markdown, integration tests against real schema models.

Design decisions worth reviewing

analyze_type is iterative, not recursive. The while True loop handles arbitrary
nesting depth (NewType wrapping Annotated wrapping NewType wrapping Annotated...)
without stack growth. Dict key/value types are the one exception where it recurses.

Cache insertion before recursion in expand_model_tree. The sub-model's ModelSpec
enters the cache before its fields are expanded. A back-edge encounter finds the cached
entry and marks starts_cycle=True rather than infinite-looping.

FeatureSpec is a Protocol, not a base class. ModelSpec and UnionSpec have
different field structures (flat list vs. annotated-field list with variant provenance).
A protocol lets them share a pipeline interface without forcing inheritance.

Constraint provenance via ConstraintSource. Each constraint records which NewType
contributed it. Field-level constraints with source=None render on the field;
constraints with a named source render on the NewType's own page. This prevents
duplication.

Test plan

• make check passes (pytest + doctests + ruff + mypy)
• make install && overture-codegen generate --format markdown --output-dir /tmp/schema-docs produces output
• Spot-check generated markdown for a union feature (e.g., Segment) and a model
  feature (e.g., Building) -- field tables, links, constraint descriptions, examples
• Verify cross-page links resolve correctly (supplementary types link back to
  features, features link to shared types)

The live schema reference contains Markdown produced by these changes (modulo some improvements from today).

[Adam Lastowka (Rachmanin0xFF)]

Haven't reviewed everything yet, but I found an issue in here while testing the parquet generator: this does not properly unpack nested lists. I don't think it's a problem for the markdown, but it surfaces in Divisions where we have list[NewType("Hierarchy", list[HierarchyItem])]. You can see the diff in the resulting arrow schemas:

Generated:
list<element: struct<division_id: string not null, subtype: string not null, name: string not null>>

Release Data (2026-02-18.0):
list<element: list<element: struct<division_id: string, subtype: string, name: string>>>

This should instead use something like a list_depth or a recursive unwrap.