Expand IcebergIO write API to cover all write options - #5986
Conversation
| writeProperties: Map[String, String] = IcebergIO.WriteParam.DefaultWriteProperties, | ||
| sortFields: List[String] = IcebergIO.WriteParam.DefaultSortFields, | ||
| partitionFields: List[String] = IcebergIO.WriteParam.DefaultPartitionFields, | ||
| extraConfigProperties: Map[String, AnyRef] = IcebergIO.WriteParam.DefaultExtraConfigProperties, |
There was a problem hiding this comment.
I thought this was more flexible than adding a dedicated named param for every new option added to the Iceberg write api (i.e. distribution_mode, autosharing, etc...)
There was a problem hiding this comment.
Yeah might be better this way indeed till the API stabilizes
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5986 +/- ##
==========================================
- Coverage 61.99% 61.98% -0.01%
==========================================
Files 323 323
Lines 11737 11743 +6
Branches 793 839 +46
==========================================
+ Hits 7276 7279 +3
- Misses 4461 4464 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| withClue( | ||
| "Iceberg writeProperties are not supported in Beam versions <= 2.76. Once Beam is upgraded, flip this assertion to `true`." | ||
| ) { | ||
| hasBloom shouldBe false |
There was a problem hiding this comment.
This won't work until the next Beam release w/ apache/beam#39250 ; I did validate that this test passes when using a locally built Beam snapshot 👍
| table: String, | ||
| catalogName: String = null, | ||
| catalogProperties: Map[String, String] = IcebergIO.WriteParam.DefaultCatalogProperties, | ||
| hadoopConfigProperties: Map[String, String] = |
There was a problem hiding this comment.
this is a breaking change. We assume that there are not many affected users? Should we add some doc though?
There was a problem hiding this comment.
yeah... not ideal, but AFAIK there are no users of this API. We weren't even publishing the scio-managed artifact until about a month ago, in 0.15.7 😅
3 participants
Expands iceberg write API to support
distribution_modeconfigPropertiestowritePropertiesto match Iceberg convention https://iceberg.apache.org/docs/latest/configuration/#write-properties