fix(schema): enforce executable type evolution - #681
Conversation
ALTER TYPE admitted Arrow-castable pairs without guaranteeing Paimon-compatible reads of historical files. Predicate pruning and global indexes could also retain stale physical-type semantics. Add explicit evolution executors and history preflight checks, apply safe residual filtering and writer normalization, and fence schema-incompatible global index entries across catalog and DataFusion paths. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
| {"name": "_INDEX_META", "type": ["null", "bytes"], "default": null}, | ||
| {"name": "_SOURCE_META", "type": ["null", "bytes"], "default": null} | ||
| {"name": "_SOURCE_META", "type": ["null", "bytes"], "default": null}, | ||
| {"name": "_BUILD_SCHEMA_ID", "type": ["null", "long"], "default": null} |
There was a problem hiding this comment.
It records the table schema ID used to build a global-index entry. It closes a race with ALTER COLUMN TYPE: schema evolution does not advance the data snapshot, so an index built with the old field type may still pass the latest-snapshot commit guard. Readers compare the indexed field types from this schema with the current schema and ignore incompatible entries. Legacy and Java entries leave it null and are handled conservatively through schema history.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
The addition of _BUILD_SCHEMA_ID breaks compatibility with older Rust readers. Testing has confirmed that the new writer produces two Index Manifests, causing the current main reader to report unknown FileKind: 7.
|
@JingsongLi Thanks, confirmed. The nested |
2 participants
Purpose
ALTER COLUMN TYPEpreviously accepted Arrow-castable type pairs without guaranteeing Paimon-compatible conversion of historical files. Predicates, statistics, and global indexes could therefore use incompatible physical-type semantics.Brief change log
Tests
API and Format
Adds an optional
build_schema_idto global-index metadata. Legacy index entries remain readable.Documentation
Documents the supported type-evolution matrix and safety restrictions.