Skip to content

refactor(extensions): move example YAML files out of the extensions folder - #1136

Open
nielspardon wants to merge 2 commits into
substrait-io:mainfrom
nielspardon:move-example-extensions-to-examples
Open

refactor(extensions): move example YAML files out of the extensions folder#1136
nielspardon wants to merge 2 commits into
substrait-io:mainfrom
nielspardon:move-example-extensions-to-examples

Conversation

@nielspardon

@nielspardon nielspardon commented Jul 21, 2026

Copy link
Copy Markdown
Member

Problem

The core extensions/ folder mixes official function extensions (functions_*.yaml) with example/placeholder files:

  • type_variations.yaml (extension:io.substrait:type_variations) — example type variations, no functions
  • extension_types.yaml (extension:io.substrait:extension_types) — the point/line user-defined types used as the docs' UDT example

Because they live alongside the real extensions, consumers ingest them inconsistently. This surfaced in duckdb-substrait-extension#239 (issue #214), where the now-removed unknown.yaml placeholder was expanded into bogus catch-all function registrations that clobbered valid ones.

Note: the third placeholder, extensions/unknown.yaml, was already removed in #1081, which replaced it with the built-in unbound type. This PR was rebased accordingly and no longer touches it.

Filtering behavior across the ecosystem for the two remaining files is inconsistent:

Consumer Strategy type_variations extension_types
DuckDB (post-#239) allowlist functions_*.yaml skip skip
substrait-python allowlist functions*.yaml skip skip
substrait-java hardcoded curated list excluded loaded
substrait-go load all loaded loaded
substrait-rs (+packaging) load all; rs skips by URN loaded (inert) skip (stale)

Change

Move the two example files to site/examples/extensions/ (the existing, schema-validated, docs-embeddable examples folder) and rename their URNs to the extension:example:* convention already used there. The core extensions/ folder now contains only official functions_*.yaml extensions.

References to extension:io.substrait:extension_types are updated to extension:example:extension_types in the docs (extensions/index.md, types/type_parsing.md), the distance_functions.yaml example dependency, and the parser test fixtures in tests/coverage/test_coverage.py. The type_variations URN is not referenced anywhere else.

Also adds a .DS_Store entry to .gitignore.

No tooling/config changes were needed: the coverage tests and function-doc generator already filter to functions_*, check-examples now validates the moved files against the same schema check-extensions used, yamllint runs repo-wide, and the mkdocs --8<-- include root is unchanged.

Verification

  • pixi run yamllint — clean
  • pixi run check-jsonschema — examples + extensions + dialects all ok
  • pixi run pytest tests/test_extensions.py tests/coverage/test_coverage.py — 128 passed
  • pixi run mkdocs build --strict — builds, includes resolve
  • git grep extension:io.substrait:{type_variations,extension_types} — no matches remain

Open questions for maintainers

  1. Is extension_types an example or an official extension? Unlike type_variations, it's genuinely load-bearing today (substrait-java and substrait-go load its point/line types). This PR treats it as an example; if we'd rather ship it officially, it should stay in / return to extensions/ under an official URN and only type_variations moves.
  2. Where should examples live? This PR uses the docs tree (site/examples/extensions/, not shipped in the release artifact). An alternative is shipping examples under extensions/examples/ so they travel with the extensions tree for downstream testing — the file moves and URN renames are identical; only the target directory and the check-extensions glob differ.

Downstream follow-ups (separate PRs)

Consumers that vendor the extensions folder will need updates when they next bump, since both files moved and the extension_types URN changed:

🤖 Generated with AI


This change is Reviewable

…older

The extensions/ folder mixed official function extensions (functions_*.yaml)
with example/placeholder files. Consumers filter these inconsistently, which
led to the DuckDB extension ingesting placeholder definitions and producing
bogus function registrations (substrait-io/duckdb-substrait-extension#239,
issue substrait-io#214).

Move the two remaining example files, type_variations.yaml and
extension_types.yaml, to site/examples/extensions/ (the existing,
schema-validated examples folder) and rename their URNs to the
extension:example:* convention used there, so the core extensions/ folder
contains only official function extensions.

References to extension:io.substrait:extension_types (docs, the
distance_functions example dependency and parser test fixtures) are updated
to extension:example:extension_types.

Note: extension_types.yaml is currently loaded by substrait-java and
substrait-go for its point/line user-defined types; whether it should instead
become a first-class official extension is left open for maintainer feedback.

The third placeholder, extensions/unknown.yaml, was removed separately in
substrait-io#1081, which replaced it with the built-in unbound type.
@nielspardon
nielspardon force-pushed the move-example-extensions-to-examples branch from c2ed733 to d2aeb13 Compare July 21, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant