Skip to content

Address review feedback: de-duplicate file-scan proto conversions - #2

Open
adriangb wants to merge 4 commits into
kumarUjjawal:issue-23497-datasource-proto-hooksfrom
pydantic:feedback/23683-partitioned-file-tryfromproto
Open

Address review feedback: de-duplicate file-scan proto conversions#2
adriangb wants to merge 4 commits into
kumarUjjawal:issue-23497-datasource-proto-hooksfrom
pydantic:feedback/23683-partitioned-file-tryfromproto

Conversation

@adriangb

@adriangb adriangb commented Jul 30, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Follow-up on my review comments on apache#23683. The theme was that
the new file_scan_config/proto.rs re-implements wire logic that already exists
elsewhere, so the copies can drift. This addresses that plus the naming and
deprecation notes.

Note on how to take this. Two of the four commits here are generic
plumbing that applies to main today, independent of the data-source hooks,
so I've also put them up as standalone precursor PRs against main:

If those land first, rebasing apache#23683 on main collapses those two commits to
just the call sites in file_scan_config/proto.rs, and apache#23683 stays focused on
what it is actually about: the DataSource / FileSource try_to_proto
hooks, the FileScanConfig spine and the delegation through DataSourceExec.
That is the shape I'd prefer — this branch exists mainly so you can see the end
state, and you should feel free to take it as guidance rather than merge it.

What changes are included in this PR?

Four commits, each independently buildable:

  1. Move PartitionedFile / FileGroup proto conversions into
    datafusion-datasource
    — the single copy of that wire logic now lives next
    to the types that own it, in a new datafusion_datasource::proto module
    (feature proto): FileRange, PartitionedFile and FileGroup all get
    inherent try_to_proto / try_from_proto. datafusion-proto's
    TryFromProto impls for those types become one-line shims delegating to
    them, and the private partitioned_file_* / file_group_* helpers in
    file_scan_config/proto.rs are gone — so the pub(crate) visibility
    question from @buraksenn's comment is moot too: these are public API on the
    types now, usable from Proto: add DataSink serialization hook apache/datafusion#23752.

    This uses inherent methods rather than moving TryFromProto into
    datafusion-proto-models first. Moving the trait on its own breaks the other
    45 impls in datafusion-proto (E0117) — they rely on it being crate-local
    because both of their types are foreign — so the trait move only works as
    part of relocating every impl to the crate that owns one of its types.
    That's worth doing as its own mechanical precursor PR (see the description of
    Add DataSource/FileSource proto hooks and FileScanConfig serde apache/datafusion#23683's parent epic); this PR doesn't block on it, and if
    the trait does move later, these inherent methods are what the relocated
    TryFromProto impls would call anyway.

  2. Rename FileScanConfig::to_proto_conf / from_proto_conf
    try_to_proto / try_from_proto, matching the hook naming convention.
    FileScanConfig also implements DataSource::try_to_proto, so the inherent
    encode method shadows the trait method for callers holding a concrete
    FileScanConfig — they differ in return type, so a mistaken call is a
    compile error rather than silent misbehavior, and dyn DataSource callers
    are unaffected. Documented on the method; the decode side has no overlap at
    all.

  3. Deprecate parse_table_schema_from_proto in
    datafusion_proto::physical_plan::from_proto (it is now a pure delegation),
    point the in-crate callers at FileScanConfig::parse_table_schema_from_proto,
    and add an upgrade-guide entry — answering the "presumably will be deprecated
    in a future PR?" comment.

  4. Put the Partitioning / PhysicalSortExpr conversion on the types
    (not from a review comment — an issue I hit while doing the above). The new
    file_scan_config/proto.rs carried its own copy of the Partitioning and
    PhysicalSortExprNode wire logic, which was already duplicated inline in
    RepartitionExec's hook and in datafusion-proto's
    serialize_partitioning / parse_protobuf_partitioning — i.e. this PR made
    it three copies. Applying the same principle as commit 1, the single copy now
    lives next to the types, taking the expression-level context
    (datafusion-physical-expr{,-common} already carry the proto feature):

    • PhysicalSortExpr::try_to_proto / try_from_proto
    • Partitioning::try_to_proto / try_from_proto

    So plan hooks can reach them, ExecutionPlanEncodeCtx /
    ExecutionPlanDecodeCtx now back the expression-level contexts and hand one
    out via expr_ctx() — useful beyond this PR, since any plan hook can now
    pass its ctx to expression-level conversions. FileScanConfig,
    RepartitionExec and datafusion-proto's central serializer all route
    through the type methods, retiring serialize_range_partitioning,
    serialize_range_split_point, parse_protobuf_range_partitioning and
    parse_protobuf_range_split_point. This one is the easiest to split into its
    own PR against main if you'd rather keep this branch focused.

The wire format is unchanged throughout. Behavior differences, both in commit 4:
out-of-range partition counts now error instead of wrapping or panicking on an
unwrap, and a missing sort-expression child reports which field is missing
instead of "Unexpected empty physical expression".

Note: a pre-existing round-trip asymmetry (not fixed here)

While adding tests for PartitionedFile::try_to_proto I found that
PartitionedFile statistics do not round-trip cleanly, on main today and
independent of this branch — filed as apache#23998. Not fixed here
since it changes decode semantics.

Are these changes tested?

Yes.

  • New unit tests in datafusion_datasource::proto for the PartitionedFile /
    FileGroup round trips and the invalid-path error.
  • The existing datafusion-proto tests now exercise the delegating shims (and
    would catch an accidental self-recursion in them).
  • FileScanConfig serde tests from this branch cover all four partitioning
    variants through the shared ctx helpers.
  • Full workspace suite: 10084 passed / 0 failed
    (cargo test --profile ci --workspace --lib --tests --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption),
    plus cargo fmt, ci/scripts/rust_clippy.sh and
    ci/scripts/doc_prettier_check.sh clean.

Are there any user-facing changes?

Yes, all additive or deprecating:

  • New public try_to_proto / try_from_proto on FileRange,
    PartitionedFile and FileGroup (feature proto on
    datafusion-datasource).
  • New try_to_proto / try_from_proto on Partitioning and
    PhysicalSortExpr, and expr_ctx() on the ExecutionPlan encode/decode
    contexts.
  • datafusion_proto::physical_plan::from_proto::parse_table_schema_from_proto
    is deprecated (documented in the 55.0.0 upgrade guide).
  • The FileScanConfig helpers added by this branch are renamed before release,
    so nothing published changes name.

…source

`FileScanConfig`'s new proto module carried private copies of the
`PartitionedFile` / `FileGroup` wire logic that already existed as
`TryFromProto` impls in `datafusion-proto`, so the two could drift.

Put the single copy next to the types that own it, in a new
`datafusion_datasource::proto` module (feature `proto`):

- `FileRange::try_to_proto` / `try_from_proto`
- `PartitionedFile::try_to_proto` / `try_from_proto`
- `FileGroup::try_to_proto` / `try_from_proto`

`datafusion-proto`'s `TryFromProto` impls for these types become one-line
shims delegating to the above, so the central serializer and the
per-source `try_to_proto` hooks cannot disagree, and the private
`partitioned_file_*` / `file_group_*` helpers in
`file_scan_config/proto.rs` are gone.

Making them inherent methods (rather than moving `TryFromProto` into
`datafusion-proto-models`) keeps `TryFromProto` local to
`datafusion-proto`, which its other 45 impls rely on for the orphan rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation physical-plan datasource proto labels Jul 30, 2026
adriangb and others added 3 commits July 30, 2026 08:29
Matches the naming used by the `ExecutionPlan` / `DataSource` /
`FileSource` hooks, as raised in review.

`FileScanConfig` also implements `DataSource::try_to_proto`, so the
inherent encode method shadows the trait method for callers holding a
concrete `FileScanConfig` (they differ in return type, so a mistaken call
is a compile error, not silent misbehavior, and `dyn DataSource` callers
are unaffected). Documented on the method; the decode side has no such
overlap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`datafusion_proto::physical_plan::from_proto::parse_table_schema_from_proto`
is now a pure delegation to `FileScanConfig::parse_table_schema_from_proto`,
so mark it deprecated (as flagged in review) and point the in-crate
callers at the `FileScanConfig` method directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The new `FileScanConfig` proto module carried its own copy of the
`Partitioning` and `PhysicalSortExprNode` wire logic, which already
existed inline in `RepartitionExec`'s hook and in `datafusion-proto`'s
`serialize_partitioning` / `parse_protobuf_partitioning` — three copies.

Apply the same principle as the `PartitionedFile` commit and put the
single copy next to the types that own it, taking the expression-level
context (`datafusion-physical-expr{,-common}` already have the `proto`
feature):

- `PhysicalSortExpr::try_to_proto` / `try_from_proto` (physical-expr-common)
- `Partitioning::try_to_proto` / `try_from_proto` (physical-expr)

To let plan hooks reach them, `ExecutionPlanEncodeCtx` /
`ExecutionPlanDecodeCtx` now back the expression-level contexts and hand
one out via `expr_ctx()`. `FileScanConfig`, `RepartitionExec` and
`datafusion-proto`'s central serializer all route through the type
methods, which also retires `serialize_range_partitioning`,
`serialize_range_split_point`, `parse_protobuf_range_partitioning` and
`parse_protobuf_range_split_point`.

Wire format is unchanged. Behavior differences: out-of-range partition
counts now error instead of wrapping or panicking on `unwrap`, and a
missing sort-expression child reports which field is missing rather than
"Unexpected empty physical expression".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adriangb
adriangb force-pushed the feedback/23683-partitioned-file-tryfromproto branch from 5e2e013 to 262b402 Compare July 30, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource documentation Improvements or additions to documentation physical-plan proto

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant