EPIC: v3 Support Tracking

[nssalian]

This tracks the progress on adding full v3 support to iceberg-rust.
Based on the pyiceberg tracking (apache/iceberg-python#1818)
and iceberg-go tracking (apache/iceberg-go#589).

New DataTypes

• Variant - PR feat: Variant Support #2188 open
  • Non-shredded read/write
  • Shredded read
  • Shredded write
  • Variant statistics (skip min/max on parent, allow on shredded children)
• Geometry / Geography
  • Support geo types #1884

Default Values

• initial-default and write-default support

Row Lineage

• first-row-id tracking on manifest entries
• next-row-id monotonicity validation
• Row lineage assignment on write

Deletion Vectors

• DV read support (Puffin + Roaring bitmap)
• DV write support
• Scanner integration

Multi-arg Transforms

• Metadata read (partition spec with multi-arg transforms)
• Eval support (when spec stabilizes)

Misc v3 Spec Items

• Stop writing distinct_counts on v3 manifests
• Emit null current-snapshot-id for empty v3 metadata (not -1)
• Support zstd metadata compression codec
• Gate v3-only metadata fields when reading v1/v2
• iceberg v3 has to set first-row-id #1652

Related

• Iceberg spec v3 changes: https://iceberg.apache.org/spec/#version-3
• iceberg-go v3 tracking: feat: v3 support tracking issue iceberg-go#589
• pyiceberg v3 tracking: V3 Tracking issue iceberg-python#1818
• Java variant shredding: Spark: Support writing shredded variant in Iceberg-Spark iceberg#14297

Describe the solution you'd like

These would be grouped and owned by one or more contributors.

[Shekharrajak]

Joined the Iceberg - Variant Community Sync call and happy to take Deletion Vectors → read support . Already following it to have in datafusion-comet apache/datafusion-comet#3376

[dannycjones]

There's these issues we can track under this epic:

• iceberg v3 has to set first-row-id #1652
• Support geo types #1884

@nssalian can we add it to the main issue body?

[wirybeaver]

Geometry/Geography support is now in progress in draft PR #2653: #2653