refactor: consume the substrait-protobuf package for core Python protobuf bindings

[nielspardon]

What

Migrate the Python package to the canonical pre-generated substrait-protobuf PyPI package for the core substrait / substrait.extensions protobuf bindings, instead of generating its own copy. This is the Python counterpart to #531 (which moves the Rust crate onto substrait-prost), but is independent of it — this PR is based on main and can merge in either order.

Changes

• py/substrait_validator/__init__.py — import Plan from the top-level substrait namespace (provided by substrait-protobuf) instead of the locally-generated .substrait.plan_pb2.
• py/build.rs — generate Python bindings only for the validator-specific substrait.validator package (substrait-protobuf does not provide it); no longer reads the Substrait submodule's protos. Also creates an __init__.py for every intermediate package directory, since the now-absent core bindings used to be what implicitly created substrait/__init__.py.
• py/pyproject.toml — add substrait-protobuf == 0.87.0 (pinned to the targeted Substrait version, mirroring the Rust pin) and raise the protobuf floor to >=5 (what substrait-protobuf requires).

Compatibility note

substrait-protobuf requires protobuf >=5 and Python >=3.10 (already this package's requires-python), so this drops any lingering protobuf-4 support.

Verification

• Full Python test suite passes (15 passed) in a venv with substrait-protobuf 0.87.0 / protobuf 6.33.6; sv.Plan is substrait.plan_pb2.Plan is True, and the locally-generated validator bindings coexist with substrait-protobuf with no descriptor-pool conflict.
• black / flake8 clean.

Follow-up

Once both this and #531 have merged, the core .proto files will no longer be needed anywhere in the repo (Rust → substrait-prost, Python → substrait-protobuf). A small follow-up will then drop the core-proto vendoring from rs/build.rs entirely.

🤖 Generated with AI

[vbarua]

One minor comments but makes sense overall.

[vbarua]

so we no longer need the Substrait submodule's protos -- nor bindings generated from them -- in this build

minor: this portion of the comment isn't that helpful IMO. It explains the change your making, but doesn't need to live in the code as it will be stale almost immediately after you merge it.

[nielspardon]

Good catch — dropped that clause. The comment now stops at what remains true after the merge:

// Note that we only generate Python bindings for the validator-specific
// `substrait.validator` package here. The bindings for the core `substrait`
// and `substrait.extensions` packages come from the `substrait-protobuf`
// PyPI package (a runtime dependency).

Applied the same principle to two other spots in this diff that would have gone stale the same way — a still contains stale copies of the core protos that is really just contains copies of the core protos, and a the generated files now live only in ... where the now was doing the dating. Substance kept in both, tense markers gone.

Also merged main (which now includes #531) into the branch and re-verified the combination: clippy/fmt clean, 21 library tests, all 158 conformance cases, black/flake8 clean, and the Python suite at 15 passed with substrait-protobuf 0.87.0 / protobuf 6.33.6. Rebuilding from a clean tree gives exactly the intended layout, which is the thing that comment describes:

py/substrait_validator/__init__.py
py/substrait_validator/substrait/__init__.py            <- intermediate, created by build.rs
py/substrait_validator/substrait/validator/tree_pb2.py
py/substrait_validator/substrait/validator/simple_extensions_pb2.py
py/substrait_validator/substrait/validator/type_system_pb2.py

no core plan_pb2.py/algebra_pb2.py, and sv.Plan is substrait.plan_pb2.Plan is True.

[nielspardon]

Added one more commit here (23d3869) for a packaging bug I hit while verifying this PR. It is pre-existing on main and not caused by these changes, but it is worth folding in because this PR is what makes the wheel newly depend on the generated intermediate substrait/__init__.py.

Symptom

Installing a locally built wheel and importing gives:

>>> import substrait_validator
ModuleNotFoundError: No module named 'substrait_validator.substrait'

The wheel contains only __init__.py and the .so — none of the generated substrait.validator bindings.

Root cause

1. py/build.rs writes the bindings into py/substrait_validator/substrait/, which py/.gitignore excludes as build output.
2. maturin >= 1.0 honors .gitignore when collecting python-source files. It applies this only inside a git checkout — I confirmed the trigger is merely the presence of a .git entry, by dropping an empty .git directory into a non-git copy of the tree and watching inclusion flip from 5 files to 0.
3. So every wheel built from a checkout drops the bindings. Unpacked sdists are not git checkouts, which is exactly why the sdist matrix row has always passed while the wheel rows did not catch this.
4. maturin 0.14.17 built every currently-published artifact and did not do this. CI resolves maturin>=1.0,<2 to 1.14.1 (confirmed in a recent run log), so the breakage is latent, not shipped — all 16 published 0.1.4 files are 0.14.17 builds tagged cp38/cp39 with Requires-Python: >=3. It would land on the next release.

Why CI could not see it

The wheel rows install the package editable first (for test dependencies), so the following step

python3 -m pip install --no-index --find-links=dist substrait-validator

matches an already-satisfied requirement by name and does nothing. substrait_validator.__file__ still points into the source tree afterward, so pytest exercised the source tree, never the artifact.

Fix

• py/pyproject.toml — declare the bindings with an explicit [tool.maturin] include. The paths are deliberately narrow rather than a glob over substrait/: a wide glob would also pick up core plan_pb2.py/algebra_pb2.py left in an existing build tree and shadow the ones substrait-protobuf provides. I verified that by planting a stale plan_pb2.py and confirming the narrow form excludes it.
• .github/workflows/python.yml — --force-reinstall --no-deps so the wheel actually replaces the editable install. Kept name-based matching instead of a dist/*.whl glob, which the Windows runner's shell does not expand.

Verification

Mirroring CI's exact sequence — editable install, then wheel install, then pytest — the wheel is now what gets tested and 15 passed, with substrait_validator resolving to site-packages rather than the source tree. The wheel contains exactly the five expected files and no stale core bindings. As a negative control, the CI change alone against the old packaging config reproduces the ModuleNotFoundError immediately, so this cannot silently regress.

Happy to split this into its own PR against main if you would rather keep this one scoped to the substrait-protobuf migration.