Address review feedback: de-duplicate file-scan proto conversions - #2
Open
adriangb wants to merge 4 commits into
Conversation
…source `FileScanConfig`'s new proto module carried private copies of the `PartitionedFile` / `FileGroup` wire logic that already existed as `TryFromProto` impls in `datafusion-proto`, so the two could drift. Put the single copy next to the types that own it, in a new `datafusion_datasource::proto` module (feature `proto`): - `FileRange::try_to_proto` / `try_from_proto` - `PartitionedFile::try_to_proto` / `try_from_proto` - `FileGroup::try_to_proto` / `try_from_proto` `datafusion-proto`'s `TryFromProto` impls for these types become one-line shims delegating to the above, so the central serializer and the per-source `try_to_proto` hooks cannot disagree, and the private `partitioned_file_*` / `file_group_*` helpers in `file_scan_config/proto.rs` are gone. Making them inherent methods (rather than moving `TryFromProto` into `datafusion-proto-models`) keeps `TryFromProto` local to `datafusion-proto`, which its other 45 impls rely on for the orphan rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Matches the naming used by the `ExecutionPlan` / `DataSource` / `FileSource` hooks, as raised in review. `FileScanConfig` also implements `DataSource::try_to_proto`, so the inherent encode method shadows the trait method for callers holding a concrete `FileScanConfig` (they differ in return type, so a mistaken call is a compile error, not silent misbehavior, and `dyn DataSource` callers are unaffected). Documented on the method; the decode side has no such overlap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`datafusion_proto::physical_plan::from_proto::parse_table_schema_from_proto` is now a pure delegation to `FileScanConfig::parse_table_schema_from_proto`, so mark it deprecated (as flagged in review) and point the in-crate callers at the `FileScanConfig` method directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The new `FileScanConfig` proto module carried its own copy of the
`Partitioning` and `PhysicalSortExprNode` wire logic, which already
existed inline in `RepartitionExec`'s hook and in `datafusion-proto`'s
`serialize_partitioning` / `parse_protobuf_partitioning` — three copies.
Apply the same principle as the `PartitionedFile` commit and put the
single copy next to the types that own it, taking the expression-level
context (`datafusion-physical-expr{,-common}` already have the `proto`
feature):
- `PhysicalSortExpr::try_to_proto` / `try_from_proto` (physical-expr-common)
- `Partitioning::try_to_proto` / `try_from_proto` (physical-expr)
To let plan hooks reach them, `ExecutionPlanEncodeCtx` /
`ExecutionPlanDecodeCtx` now back the expression-level contexts and hand
one out via `expr_ctx()`. `FileScanConfig`, `RepartitionExec` and
`datafusion-proto`'s central serializer all route through the type
methods, which also retires `serialize_range_partitioning`,
`serialize_range_split_point`, `parse_protobuf_range_partitioning` and
`parse_protobuf_range_split_point`.
Wire format is unchanged. Behavior differences: out-of-range partition
counts now error instead of wrapping or panicking on `unwrap`, and a
missing sort-expression child reports which field is missing rather than
"Unexpected empty physical expression".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adriangb
force-pushed
the
feedback/23683-partitioned-file-tryfromproto
branch
from
July 30, 2026 13:53
5e2e013 to
262b402
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Follow-up on my review comments on apache#23683. The theme was that
the new
file_scan_config/proto.rsre-implements wire logic that already existselsewhere, so the copies can drift. This addresses that plus the naming and
deprecation notes.
What changes are included in this PR?
Four commits, each independently buildable:
Move
PartitionedFile/FileGroupproto conversions intodatafusion-datasource— the single copy of that wire logic now lives nextto the types that own it, in a new
datafusion_datasource::protomodule(feature
proto):FileRange,PartitionedFileandFileGroupall getinherent
try_to_proto/try_from_proto.datafusion-proto'sTryFromProtoimpls for those types become one-line shims delegating tothem, and the private
partitioned_file_*/file_group_*helpers infile_scan_config/proto.rsare gone — so thepub(crate)visibilityquestion from @buraksenn's comment is moot too: these are public API on the
types now, usable from Proto: add DataSink serialization hook apache/datafusion#23752.
This uses inherent methods rather than moving
TryFromProtointodatafusion-proto-modelsfirst. Moving the trait on its own breaks the other45 impls in
datafusion-proto(E0117) — they rely on it being crate-localbecause both of their types are foreign — so the trait move only works as
part of relocating every impl to the crate that owns one of its types.
That's worth doing as its own mechanical precursor PR (see the description of
Add DataSource/FileSource proto hooks and FileScanConfig serde apache/datafusion#23683's parent epic); this PR doesn't block on it, and if
the trait does move later, these inherent methods are what the relocated
TryFromProtoimpls would call anyway.Rename
FileScanConfig::to_proto_conf/from_proto_conf→try_to_proto/try_from_proto, matching the hook naming convention.FileScanConfigalso implementsDataSource::try_to_proto, so the inherentencode method shadows the trait method for callers holding a concrete
FileScanConfig— they differ in return type, so a mistaken call is acompile error rather than silent misbehavior, and
dyn DataSourcecallersare unaffected. Documented on the method; the decode side has no overlap at
all.
Deprecate
parse_table_schema_from_protoindatafusion_proto::physical_plan::from_proto(it is now a pure delegation),point the in-crate callers at
FileScanConfig::parse_table_schema_from_proto,and add an upgrade-guide entry — answering the "presumably will be deprecated
in a future PR?" comment.
Put the
Partitioning/PhysicalSortExprconversion on the types(not from a review comment — an issue I hit while doing the above). The new
file_scan_config/proto.rscarried its own copy of thePartitioningandPhysicalSortExprNodewire logic, which was already duplicated inline inRepartitionExec's hook and indatafusion-proto'sserialize_partitioning/parse_protobuf_partitioning— i.e. this PR madeit three copies. Applying the same principle as commit 1, the single copy now
lives next to the types, taking the expression-level context
(
datafusion-physical-expr{,-common}already carry theprotofeature):PhysicalSortExpr::try_to_proto/try_from_protoPartitioning::try_to_proto/try_from_protoSo plan hooks can reach them,
ExecutionPlanEncodeCtx/ExecutionPlanDecodeCtxnow back the expression-level contexts and hand oneout via
expr_ctx()— useful beyond this PR, since any plan hook can nowpass its ctx to expression-level conversions.
FileScanConfig,RepartitionExecanddatafusion-proto's central serializer all routethrough the type methods, retiring
serialize_range_partitioning,serialize_range_split_point,parse_protobuf_range_partitioningandparse_protobuf_range_split_point. This one is the easiest to split into itsown PR against
mainif you'd rather keep this branch focused.The wire format is unchanged throughout. Behavior differences, both in commit 4:
out-of-range partition counts now error instead of wrapping or panicking on an
unwrap, and a missing sort-expression child reports which field is missinginstead of "Unexpected empty physical expression".
Note: a pre-existing round-trip asymmetry (not fixed here)
While adding tests for
PartitionedFile::try_to_protoI found thatPartitionedFilestatistics do not round-trip cleanly, onmaintoday andindependent of this branch — filed as apache#23998. Not fixed here
since it changes decode semantics.
Are these changes tested?
Yes.
datafusion_datasource::protofor thePartitionedFile/FileGroupround trips and the invalid-path error.datafusion-prototests now exercise the delegating shims (andwould catch an accidental self-recursion in them).
FileScanConfigserde tests from this branch cover all four partitioningvariants through the shared ctx helpers.
(
cargo test --profile ci --workspace --lib --tests --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption),plus
cargo fmt,ci/scripts/rust_clippy.shandci/scripts/doc_prettier_check.shclean.Are there any user-facing changes?
Yes, all additive or deprecating:
try_to_proto/try_from_protoonFileRange,PartitionedFileandFileGroup(featureprotoondatafusion-datasource).try_to_proto/try_from_protoonPartitioningandPhysicalSortExpr, andexpr_ctx()on theExecutionPlanencode/decodecontexts.
datafusion_proto::physical_plan::from_proto::parse_table_schema_from_protois deprecated (documented in the 55.0.0 upgrade guide).
FileScanConfighelpers added by this branch are renamed before release,so nothing published changes name.