Skip to content

test(recipe): bump Spark 4 integration Iceberg runtime to 1.11.0 #1224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
laskoviymishka merged 2 commits into from
Jun 18, 2026
Merged

test(recipe): bump Spark 4 integration Iceberg runtime to 1.11.0 #1224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8c7a8e1
test(recipe): bump Spark 4 integration Iceberg runtime to 1.11.0
nssalian Jun 18, 2026
d2f50ac
Spark 3.5 fix
nssalian Jun 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion internal/recipe/Dockerfile.spark4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ FROM apache/spark:4.0.2-python3

USER root

ARG ICEBERG_VERSION=1.10.0
ARG ICEBERG_VERSION=1.11.0
ARG SCALA_VERSION=2.13

RUN curl --progress-bar -L --retry 3 \
Expand Down
13 changes: 8 additions & 5 deletions internal/recipe/provision.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@

spark = SparkSession.builder.getOrCreate()

# Spark 4 (Iceberg 1.11+) warns on a bare "default." in ALTER extensions; Spark 3.5 (Iceberg 1.8) can't plan them with an explicit catalog. Qualify only on 4+.
alter_ns = "spark_catalog.default" if int(spark.version.split(".")[0]) >= 4 else "default"

catalogs = {
'rest': load_catalog(
"rest",
Expand Down Expand Up @@ -138,7 +141,7 @@

# Partitioning is not really needed, but there is a bug:
# https://github.com/apache/iceberg/pull/7685
spark.sql(f"ALTER TABLE default.test_positional_mor_deletes ADD PARTITION FIELD years(dt) AS dt_years")
spark.sql(f"ALTER TABLE {alter_ns}.test_positional_mor_deletes ADD PARTITION FIELD years(dt) AS dt_years")

spark.sql(
f"""
Expand All @@ -159,9 +162,9 @@
"""
)

spark.sql(f"ALTER TABLE default.test_positional_mor_deletes CREATE TAG tag_12")
spark.sql(f"ALTER TABLE {alter_ns}.test_positional_mor_deletes CREATE TAG tag_12")

spark.sql(f"ALTER TABLE default.test_positional_mor_deletes CREATE BRANCH without_5")
spark.sql(f"ALTER TABLE {alter_ns}.test_positional_mor_deletes CREATE BRANCH without_5")

# WAP must be enabled for spark.wap.branch to scope DML to the branch
spark.sql(f"ALTER TABLE default.test_positional_mor_deletes SET TBLPROPERTIES ('write.wap.enabled'='true')")
Expand Down Expand Up @@ -189,7 +192,7 @@
"""
)

spark.sql(f"ALTER TABLE default.test_positional_mor_double_deletes ADD PARTITION FIELD years(dt) AS dt_years")
spark.sql(f"ALTER TABLE {alter_ns}.test_positional_mor_double_deletes ADD PARTITION FIELD years(dt) AS dt_years")

spark.sql(
f"""
Expand Down Expand Up @@ -272,7 +275,7 @@
"""
)

spark.sql(f"ALTER TABLE default.{table_name} ADD PARTITION FIELD {partition}")
spark.sql(f"ALTER TABLE {alter_ns}.{table_name} ADD PARTITION FIELD {partition}")

spark.sql(
f"""
Expand Down
Loading