Spark 4.1: Mark overrides of deprecated Spark API as deprecated#16028
Spark 4.1: Mark overrides of deprecated Spark API as deprecated#16028manuzhang wants to merge 5 commits into
Conversation
manuzhang
force-pushed
the
deprecate-SparkTable-schema
branch
from
9b89e14 to
fe74213
Compare
manuzhang
changed the title
schema() on Spark Table implementations
|
@huaxingao Fixed previous issues and I extended the scope of this PR to all deprecated Spark API. Please help review again. Thanks! |
manuzhang
requested review from
amogh-jahagirdar,
aokolnychyi,
huaxingao and
szehon-ho
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
@huaxingao @szehon-ho Please help review when you have time. Thanks! |
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
Thanks for the updates @manuzhang! The approach looks good -- the active path now bypasses Spark's deprecated APIs. Left two small nits inline. A few existing threads still open before I approve:
- Javadoc style
/** ... */so@deprecatedis actually picked up, plus the missing@DeprecatedonRollbackStagedTable(@huaxingao's comments). - Switching the
@deprecatedtext to{@link #columns()}for consistency acrossBaseSparkTable/SparkChangelogTable/RollbackStagedTable. - Dropping the now-redundant deprecated
createTable(StructType, ...)override inSparkRewriteTableCatalog(it only throwsUnsupportedOperationException, so the defaultTableInfochain routes there anyway).
Once those are in I think this is good to go.
|
@szehon-ho I've updated as requested. Please help review again. Thanks! |
Mark Spark 4.1 overrides of deprecated catalog and table schema methods as deprecated ahead of 1.12.0. Implement the TableInfo catalog methods and route legacy StructType overloads through a shared Spark3Util helper so the active path avoids Spark's deprecated APIs. Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Szehon Ho <szehon.apache@gmail.com>
Co-authored-by: Szehon Ho <szehon.apache@gmail.com>
Co-authored-by: Szehon Ho <szehon.apache@gmail.com>
Drop the redundant rewrite catalog StructType override, keep the StructType tableInfo helper package-private, and share the cached Spark schema conversion for columns(). Co-authored-by: Codex <codex@openai.com>
manuzhang
force-pushed
the
deprecate-SparkTable-schema
branch
from
895fc3d to
db2a162
Compare
3 participants
Mark Spark 4.1 overrides of deprecated catalog and table schema methods as deprecated ahead of 1.12.0. Implement the TableInfo catalog methods and route legacy StructType overloads through a shared Spark3Util helper so the active path avoids Spark's deprecated APIs.
Co-authored-by: @codex