refactor: consume the substrait-prost crate for core protobuf types

[nielspardon]

What

Stop vendoring and compiling the core Substrait protobuf files (the substrait and substrait.extensions packages) in this repository. Instead depend on the canonical pre-generated substrait-prost crate (0.87.0) and give the validator's protobuf introspection layer to those foreign types via runtime reflection.

This is the prost counterpart to the recent substrait-antlr (#527) and substrait-extensions (#529) migrations toward consuming the shared, version-tagged artifacts published from substrait-packaging.

Why it isn't just a dependency swap

The validator bolts an introspection layer onto every protobuf type at code-generation time: build.rs applied #[derive(ProtoMeta)] to every prost-generated type, producing impls of the local InputNode/ProtoMessage/ProtoOneOf/ProtoEnum traits that drive the traversal/validation engine. A derive can only run where a type is defined, so it cannot be applied to substrait-prost's foreign types.

The introspection traits are local, however, so the orphan rule permits impl InputNode for substrait_prost::Plan { … }. The question is where the per-type knowledge behind those impls comes from.

How: runtime reflection

An earlier revision of this PR answered that by reconstructing prost's code generation inside build.rs — decoding the embedded FileDescriptorSet and re-deriving Rust naming plus Box placement for recursive fields via a MessageGraph, mirroring prost-build. Per @vbarua's review (and the prost-reflect PoC in #538), that machinery is gone.

The neutral introspection now comes from upstream instead. substrait-io/substrait-packaging#35 added prost::Name + prost_reflect::ReflectMessage impls to substrait-prost behind a reflect feature, released in 0.87.0. ProtoMeta itself could not move upstream — it emits impls of the validator engine's traits, which would make neutral bindings depend on the validator and invert the layering — but Name/ReflectMessage are standard-ecosystem traits with no validator coupling, so they belong in the bindings crate.

With those available on the foreign types:

• build.rs's prost_meta generator emits thin, uniform trait impls — no boxing analysis, no per-field emission.
• Unknown-field detection defers to a single generic parse_proto_message_unknown, which enumerates fields through the descriptor and detects presence via a DynamicMessage.
• The ProtoMeta derive — still needed for the local substrait.validator package — is likewise reduced to reflection-based impls.

Generated output dropped from ~7,750 to ~4,557 lines.

One wrinkle worth flagging

descriptor().fields() enumerates oneof members, but the parse code marks a oneof under its declaration name. For substrait.Rel, proto_required_field!(x, y, rel_type, …) marks rel_type as parsed, while Rel's descriptor fields are the members read, filter, project, join, … A naive reflection walk therefore sees rel_type handled yet every member "unparsed", and flags them as unrecognized — a spurious not recognized by the validator warning on essentially every relation. The fix collapses each real oneof into a single unit keyed by the oneof name and addresses it through its Variant path element, matching how the typed traversal marked and rendered it.

Scope

• The validator-specific substrait.validator package is not provided by substrait-prost, so it is still compiled locally with the ProtoMeta derive.
• crate::input::proto::substrait now glob-re-exports the substrait-prost types, so the rest of the validator (~330 type references across 32 files) is unchanged.
• substrait-validator-derive is retained (used for the validator package).
• Full dynamic, descriptor-driven traversal — dropping the generated InputNode impls for the foreign types too — is a sensible follow-up, deliberately out of scope here.

Files

• rs/Cargo.toml — add substrait-prost (features embed-descriptor, reflect) as a normal + build dependency, plus prost-reflect; move substrait-extensions/substrait-antlr to 0.87.0.
• rs/build.rs — drop core-proto sync/compilation; compile only the substrait.validator package locally; add the prost_meta trait-impl generator.
• rs/src/input/proto.rs — glob re-export of substrait-prost, include of the generated impls, regression tests.
• rs/src/parse/traversal.rs — generic reflection-based unknown-field detection.
• derive/src/lib.rs — ProtoMeta reduced to reflection-based impls.

Verification

• cargo build/clippy/fmt clean across the workspace.
• Library tests pass, including regression tests pinning nested-message names (substrait.Type.List), URN acronym casing (substrait.extensions.SimpleExtensionURN), enum defaults/variants, and mixed-boxing oneofs.
• All 158 conformance test cases pass — end-to-end traversal over real plans (boxed/unboxed fields, oneofs, enums) produces identical diagnostics.
• Checked against substrait spec v0.87.0, the version pinned by the substrait submodule.

Notes

• Depends only on released crates: substrait-prost, substrait-extensions and substrait-antlr at 0.87.0. The earlier exact pin on the 0.87.0-alpha.4 pre-release has been dropped.

🤖 Generated with AI

[vbarua]

How did this work before?

/// substrait-prost ships plain prost-generated types without our ProtoMeta
/// derive,

Is this something we could include in substrait-prost, potentially behind a flag, for use here? The prost_meta stuff below seems a little extreme, and this code wasn't in this repo before so something was already generating it, I think?

[vbarua]

The validator bolts an introspection layer onto every protobuf type at code-generation time: build.rs applies #[derive(ProtoMeta)] to every prost-generated type, producing impls of the local InputNode/ProtoMessage/ProtoOneOf/ProtoEnum traits that drive the traversal/validation engine. A derive can only run where a type is defined, so it cannot be applied to substrait-prost's foreign types.

Could we apply [derive(ProtoMeta)] in substrait-prost?

[nielspardon]

I explored some options with my AI helper. The derive macro is tightly coupled with the validator specific proto definitions so can not as cleanly be moved without dragging those validator protos along.

---

How did this work before? ... something was already generating it, I think?

Yep — the impls were never checked in. build.rs attached #[derive(ProtoMeta)] to the locally-generated prost types via type_attribute, so they were produced at build time. That only works because a derive needs the type to be defined locally; once the types come from substrait-prost they're foreign, so prost_meta reconstructs the same impls from the embedded FileDescriptorSet instead.

Could we apply #[derive(ProtoMeta)] in substrait-prost?

Two snags:

1. It wouldn't let us drop the derive here anyway — the substrait.validator package isn't in substrait-prost, so we still apply ProtoMeta to it locally.
2. ProtoMeta emits impls of our traits (InputNode, etc.), and those are tied to the validator engine — parse_unknown takes &mut context::Context and the node methods return tree::Node, which between them pull in most of output::* and parse::*. For substrait-prost to host the impls it'd have to depend on the validator core: circular, and a layering inversion for neutral bindings.

The macro crate is already standalone and easy to publish, but publishing the macro isn't the blocker — the generated code needs the traits + their whole dependency cone in scope. Doing this properly means extracting a shared crate (traits + tree/primitive_data/context/...) for substrait-prost to depend on behind a flag. Doable, but a much bigger carve-out than prost_meta, which is why I kept substrait-prost validator-agnostic. Happy to revisit if we think the coupling's worth it.

[vbarua]

I hadn't realized that ProtoMeta was coupled to substrait-validator specific code. That doesn't feel like something we should include in substrait-prost.

I've been poking around this a little bit as well. Parsing the generated prost code at build time to dynamically build more rust code feels way too big 🧠 and gives me the heebie jeebies to be honest..

Poking around the context for how this works, I did see this comment:

substrait-validator/derive/src/lib.rs

Lines 8 to 12 in af55680

	//! recover the needed protobuf message metadata from there. Things would have
	//! been a *LOT* simpler and a *LOT* less brittle if prost would simply
	//! provide this information via traits of its own, but alas, there doesn't
	//! seem to be a way to do this without forking prost, and introspection
	//! seems to be a non-goal of that project.

which may be a bit out of date. prost-reflect might actually give us to close to enough introspection

I threw some llm compute at this to see if it's feasible: #538 to replicate some/most of what ProtoMeta gives us with just prost-reflect.

[nielspardon]

Thanks for the suggestion. My version of Claude agrees with you and your version of Claude 😄 I opened substrait-io/substrait-packaging#35 to add prost-reflect to substrait-prost.

[nielspardon]

Thanks for the prost-reflect illustration in #538 — that was the nudge that settled it. I agreed the prost_meta codegen was too much machinery, and your PoC made it concrete that runtime reflection could carry the same weight without reconstructing prost's box/naming decisions.

Two things fell into place to make it real here:

1. The neutral traits now live in substrait-prost. ProtoMeta couldn't move upstream (it's welded to the validator engine), but prost::Name and prost_reflect::ReflectMessage are neutral, standard-ecosystem introspection with zero validator coupling — so they belong in the bindings crate. That shipped in substrait-io/substrait-packaging#35 and is released as substrait-prost 0.87.0-alpha.4 behind a reflect feature. With it, the foreign core types carry Name + ReflectMessage, and the orphan-rule problem disappears.

2. I've rebuilt this PR on that. prost_meta loses the MessageGraph/boxing analysis and per-field emission entirely; unknown-field handling now defers to a single generic parse_proto_message_unknown that walks the descriptor and detects presence via a DynamicMessage. The generator is now thin, uniform trait impls (generated output ~7,750 → ~4,557 lines), and the ProtoMeta derive — still used for the local substrait.validator package — is likewise reduced to reflection-based impls.

One wrinkle worth noting for anyone taking this further: descriptor().fields() enumerates oneof members, but the parse code marks a oneof under its declaration name. Take substrait.Rel: the validator handles it with proto_required_field!(x, y, rel_type, …), marking rel_type as parsed — but Rel's descriptor fields are the members read, filter, project, join, … So a naive reflection walk sees rel_type handled yet every member "unparsed", and flags all of them as unrecognized (a spurious not recognized by the validator warning on essentially every relation). The fix is to collapse each real oneof into a single unit keyed by the oneof name (rel_type) and address it through its Variant path element, matching how the typed traversal marked and rendered it.

Verified locally against substrait spec v0.87.0: cargo build/clippy/fmt clean, library tests pass, and all 158 conformance cases pass. Just pushed it here.

Full dynamic (descriptor-driven) traversal — dropping the generated InputNode impls for the foreign types too — is a sensible next step, but I've kept it out of scope for this PR.

---

This summary was generated by AI.

[nielspardon]

Full dynamic (descriptor-driven) traversal — dropping the generated InputNode impls for the foreign types too — is a sensible next step, but I've kept it out of scope for this PR.

there seems to be an opportunity for a much bigger refactoring which I thought we best do as a follow-up

[vbarua]

The build.rs changes are a still a bit scary IMO, but they seem to work. I think it makes sense to try this and see how it goes.

[nielspardon]

Rebuilt on the released crates now that `0.87.0` is out: the exact pin on the `0.87.0-alpha.4` pre-release is gone, and `substrait-prost`/`substrait-extensions`/`substrait-antlr` are all plain `0.87.0` — matching the `v0.87.0` spec version pinned by the `substrait` submodule. cargo build/clippy/fmt clean, library tests pass, all 158 conformance cases pass.

One unrelated commit rode along: the Lint job was failing on tests/src/runner.rs with

error: redundant reference in `println!` argument
   --> tests/src/runner.rs:556:21
   = note: `-D clippy::useless-borrows-in-formatting` implied by `-D warnings`

That is pre-existing on main (the line dates to 498a4d3 and this PR never touches that file) — clippy 1.97 promoted useless_borrows_in_formatting, and since CI pins dtolnay/rust-toolchain@stable the job now fails on every open PR, e.g. #550 and #552. main itself was last built on 2026-07-01, before 1.97, so it never went red. Folded the one-line fix in here to unblock; happy to split it into its own PR against main instead if preferred, since that would clear the other PRs too.

Verified by reproducing CI exactly with clippy 1.97 locally (cargo clippy --all-features -- -D warnings -A clippy::doc-overindented-list-items): one occurrence, clean after the fix.