feat(build): bump Scala/Java services to Java 17 LTS#4938
feat(build): bump Scala/Java services to Java 17 LTS#4938bobbai00 wants to merge 10 commits intoapache:mainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4938 +/- ##
============================================
+ Coverage 42.49% 42.87% +0.37%
+ Complexity 2180 1216 -964
============================================
Files 1005 738 -267
Lines 37429 29862 -7567
Branches 3914 2906 -1008
============================================
- Hits 15907 12803 -3104
+ Misses 20558 16436 -4122
+ Partials 964 623 -341
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
25ae13f to
64ffded
Compare
aglinxinyuan
left a comment
There was a problem hiding this comment.
Do we have to include it as part of the release? If it's not required, let's not include it in the release and do more testing.
The discussion also mentioned that we need to have JDK_JAVA_OPTIONS. Is that still required? I don't see it anywhere in this PR.
### What changes were proposed in this PR? Backport #4549 (initial AGENTS.md + CLAUDE.md) and #4825 (AGENTS.md rewrite) to `release/v1.1.0-incubating`. Two cherry-picks: #4549 applies clean; #4825 has trivial content conflicts that resolve by taking the post-rewrite state (i.e. `upstream/main:AGENTS.md` verbatim — what the squashed result of both PRs would produce on main). ### Any related issues, documentation, discussions? Backport of #4549 and #4825. Unblocks the auto-backport simulation for #4938 (Java 17 bump) and any future PR that touches `AGENTS.md`. ### How was this PR tested? Doc-only change. Verified `diff <(git show HEAD:AGENTS.md) <(git show upstream/main:AGENTS.md)` is empty. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) --------- Co-authored-by: Xinyuan Lin <xinyual3@uci.edu> Co-authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Got it. I mentioned it in the PR description and The main reason I want to include it in the release branch is because Java 11 already doesn't have an LTS version. Since we use Java as the runtime image, it would be great to have an LTS base image as the runtime. |
64ffded to
c16f305
Compare
Java 11 is extended to be supported till 2032. https://www.oracle.com/java/technologies/java-se-support-roadmap.html Do we HAVE to upgrade before release?? |
c16f305 to
18f8813
Compare
Sorry, I missed the JDK_JAVA_OPTIONS part. |
fc43932 to
d7b171c
Compare
We don't have to. I removed the tag. |
|
How about bash script in bin/? do they need to be updated for |
|
|
||
| When running computing-unit master/worker outside their Docker images on | ||
| JDK 17+, also `export JDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED"`. | ||
| Without it, [Apache Arrow Java](https://arrow.apache.org/docs/java/install.html) fails on off-heap allocation. |
There was a problem hiding this comment.
The link does not work for me
There was a problem hiding this comment.
I fixed the link. Please check
3563991 to
40e94aa
Compare
aglinxinyuan
left a comment
There was a problem hiding this comment.
Do you want to add that to the IntelliJ configuration in the .run folder so developers can also run with JDK_JAVA_OPTIONS on their local machines? Please also don't solely rely on test cases for this big change. Let's run a real workflow to manually test it.
Added |
During my daily development, I use Java 17 all the time and manually ran lots of workflows. So I am pretty confident with the changes. To be safe, can you try this branch and manually run some workflows to test it? |
Many scripts are outdated currently. I prefer NOT to update them and have a separate PR to handle them. |
|
Mirror: After switching to Java 17 and running the workflows, this error occurred: |
Can you try: |
Ok, it seems like the only JDK_JAVA_OPTIONS we need are "--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED". We don't need all of them. Please update the option in the PR. The ref seems incorrect by the way. |
d371b91 to
4f24c90
Compare
Fixed. |
2eea970 to
cb03534
Compare
Java 11 premier support has ended and the temurin:11-* base images
accumulate fixable critical/high CVEs flagged by Docker Hub. Java 17 is
an active LTS through Sep 2029 and unblocks libraries that already
require 17+.
- bin/*.dockerfile (8 services): build base
sbtscala/scala-sbt:eclipse-temurin-jammy-11.0.17_8_1.9.3_2.13.11 ->
...-17.0.5_8_1.9.3_2.13.11; runtime eclipse-temurin:11-{jdk,jre}-jammy
-> 17-{jdk,jre}-jammy.
- computing-unit-master/worker: add ENV JDK_JAVA_OPTIONS="--add-opens=
java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED" so Apache
Arrow's off-heap memory module keeps working under Java 17's strong
encapsulation.
- .github/workflows/build.yml + build-and-push-images.yml: java-version
11 -> 17 (matrix and setup-java steps).
- AGENTS.md: toolchain table.
Refs discussion apache#4001. Closes apache#4937.
cb03534 to
ae79618
Compare
|
This CI run never finishes. I canceled it after 5 hours. https://github.com/apache/texera/actions/runs/25367504114/job/74383231727 |
Mirror Apache Pekko's project/JdkOptions.scala pattern: keep the required --add-opens flags in one sbt-side place and apply them via Test / javaOptions on the amber subproject (with Test / fork := true). This makes `sbt test` work out of the box on JDK 17 — no workflow env var needed, and local devs running tests on JDK 17 don't need to remember to export anything. - project/JdkOptions.scala: 5 opens (4 from Pekko + java.util for Ehcache SizeOf used by Tuple.inMemSize). - amber/build.sbt: Test / fork := true, Test / javaOptions ++= JdkOptions.versionSpecificJavaOptions. - .github/workflows/build.yml: drop JDK_JAVA_OPTIONS env from amber, amber-integration, and platform jobs. Dockerfile ENV and IntelliJ run configs are unchanged — they're not sbt-driven.
The previous commit only set Test / fork + javaOptions in amber's own build.sbt, but ArrowUtilsSpec lives under common/workflow-core and common/workflow-operator. When the CI invokes WorkflowCore/jacoco and WorkflowOperator/jacoco those tests still ran without --add-opens and failed at MemoryUtil init. Move the settings to ThisBuild scope in the root build.sbt so every subproject's test JVM forks with the same flags; drop the now-duplicate amber/build.sbt block.
Forking changed the test JVM's CWD to the subproject directory, so tests reading repo-root-relative paths (e.g. MockTexeraDB -> sql/texera_ddl.sql) failed with NoSuchFileException. Set ThisBuild / Test / baseDirectory back to the build root.
Earlier commits introduced Test / fork := true and a JdkOptions plugin to inject JDK 17 --add-opens flags. Forking changed the test JVM's CWD to each subproject directory, which broke tests that read repo-root-relative paths (sql/texera_ddl.sql from MockTexeraDB), and required a baseDirectory pin to fix. Replace all of that with a single .jvmopts at the repo root: sbt's launcher reads it and applies the flags to sbt's own JVM startup. With Test / fork at its default (false), tests run in sbt's JVM and inherit the flags — no fork overhead, no CWD change, no baseDirectory pin, no sbt plugin file. Drops: - project/JdkOptions.scala - ThisBuild / Test / fork := true - ThisBuild / Test / javaOptions ++= ... - ThisBuild / Test / baseDirectory := ... Adds: - .jvmopts (5 --add-opens lines, attribution comments)
# Conflicts: # bin/computing-unit-master.dockerfile # bin/computing-unit-worker.dockerfile
| # | ||
| # JDK 17+ --add-opens flags required by: | ||
| # - Apache Pekko (dispatcher / mailbox / scheduler use sun.misc.Unsafe | ||
| # via jdk.internal.misc, plus java.lang for virtual threads and |
There was a problem hiding this comment.
do we have virtual threads?
| # JDK 17+ --add-opens flags required by: | ||
| # - Apache Pekko (dispatcher / mailbox / scheduler use sun.misc.Unsafe | ||
| # via jdk.internal.misc, plus java.lang for virtual threads and | ||
| # sun.nio.ch for Aeron / NIO channels). |
There was a problem hiding this comment.
do we have Aeron / NIO channels?
| @@ -0,0 +1,18 @@ | |||
| # JVM options for sbt's launcher script. Read by `sbt` from the build | |||
There was a problem hiding this comment.
Add a header for this file.
|
The amber run again hangs and drains our CI. canceled manually. please check https://github.com/apache/texera/actions/runs/25420608272/job/74561719299?pr=4938 @bobbai00 @aglinxinyuan |

What changes were proposed in this PR?
Bump all Scala/Java microservices from Java 11 to Java 17 LTS.
8 Dockerfiles under
bin/: build basesbtscala/scala-sbt:eclipse-temurin-jammy-11.0.17_8_1.9.3_2.13.11→...-17.0.5_8_1.9.3_2.13.11(same sbt 1.9.3 + Scala 2.13.11); runtimeeclipse-temurin:11-{jdk,jre}-jammy→17-{jdk,jre}-jammy.bin/computing-unit-{master,worker}.dockerfile: addENV JDK_JAVA_OPTIONSwith three opens (single space-separated value):java.nioopen is required by Apache Arrow Java under JDK 17+ for off-heap memory.java.langandjava.utilopens are required by Ehcache'sSizeOfreflection used byTuple.inMemSize(refs discussion Support Java 17 as runtime for Scala/Java based micro services #4001).The other 6 service images don't exercise either path.
.github/workflows/build.yml+build-and-push-images.yml:java-version: 11→17(matrix entries + 5setup-javasteps); also set the sameJDK_JAVA_OPTIONSenv on the amber/amber-integration/platform jobs sosbt testJVMs inherit the opens..run/ComputingUnit{Master,Worker}.run.xml: same env in IntelliJ run configs.AGENTS.md: toolchain table; quick note on the Java 17--add-opensrequirement for non-container runs.Any related issues, documentation, discussions?
Closes #4937. Refs discussion #4001 (Java 17 readiness assessment — only minor changes needed).
How was this PR tested?
CI exercises end-to-end:
build.ymlbuilds + tests under Java 17,build-and-push-images.ymlproduces multi-arch images. The Arrow open is exercised by theamber-integrationjob, the Ehcache opens byamber(which loadsTupleextensively viaDPThread/SizeOf).Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)