[GLUTEN-10113][VL] Pass hadoop fs related session level configurations to native by zhouyuan · Pull Request #12367 · apache/gluten

zhouyuan · 2026-06-25T13:14:43Z

What changes are proposed in this pull request?

This patch adds the runtime hadoop related configurations to native.

How was this patch tested?

new UT

Was this patch authored or co-authored using generative AI tooling?

IBM BoB

Related issue: #10113

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-25T13:15:15Z

Run Gluten Clickhouse CI on x86

Signed-off-by: Yuan <yuanzhou@apache.org>

Copilot

Pull request overview

This PR extends the Velox backend’s partition serialization so that Hadoop filesystem-related session configuration (notably fs.azure.*, fs.s3a.*, fs.gs.*) is captured on the driver and carried across the RDD partition boundary to executors, then forwarded into the native runtime via extraConf.

Changes:

Added an fsConf: Map[String, String] field to GlutenPartition to carry selected Hadoop FS config across to executors.
Updated VeloxIteratorApi.genPartitions to collect fs.* keys from the driver-side Hadoop configuration and embed them into each GlutenPartition.
Updated VeloxIteratorApi.genFirstStageIterator to merge GlutenPartition.fsConf into the native extraConf, and added a new Velox UT suite to validate propagation behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala	Adds `GlutenPartition.fsConf` to transport FS-related Hadoop config across partition boundaries.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala	Captures driver-side FS Hadoop config into partitions and forwards it into native `extraConf`.
backends-velox/src/test/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApiFsConfSuite.scala	Adds unit tests verifying that selected `fs.*` keys are embedded into `GlutenPartition.fsConf`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-25T13:23:44Z

Run Gluten Clickhouse CI on x86

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-25T15:17:22Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala:51

GlutenPartition now contains fsConf, which can include filesystem credentials (e.g., S3/ABFS secrets). Because this is a case class, the default toString will include the full fsConf map and may leak secrets into Spark logs / exception messages / UI when partitions are stringified. Override toString to avoid printing values (e.g., print only keys).

  override def preferredLocations(): Array[String] =
    splitInfos.flatMap(_.preferredLocations().asScala)
}

github-actions · 2026-06-25T17:03:26Z

Run Gluten Clickhouse CI on x86

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-25T17:22:08Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

+    val partitionFsConf = inputPartition.asInstanceOf[GlutenPartition].fsConf
+    val extraConf = (partitionFsConf +
+      (GlutenConfig.COLUMNAR_CUDF_ENABLED.key -> enableCudf.toString)).asJava
    val transKernel = NativePlanEvaluator.create(BackendsApiManager.getBackendName, extraConf)


+      "fs.azure.account.auth.type.myaccount.dfs.core.windows.net" -> "OAuth",
+      "fs.azure.account.oauth.provider.type" -> "ClientCredentials"
+    ) {
+      val partitions = api.genPartitions(emptyWsCtx, Seq(Seq.empty), Seq.empty)


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-06-26T14:17:58Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala:51

GlutenPartition is a case class, so its generated toString will include the full fsConf map (including credential values). This can leak secrets via exceptions/logging (e.g., $split string interpolation) or Spark UI/debug output. Consider overriding toString to avoid printing values (e.g., only print the keys).

  override def preferredLocations(): Array[String] =
    splitInfos.flatMap(_.preferredLocations().asScala)
}

+    val fsPrefixes = Seq("fs.azure.", "fs.s3a.", "fs.gs.")
+    val hadoopConf = leaves.headOption
+      .map(_ => org.apache.spark.sql.SparkSession.active.sessionState.newHadoopConf())
+      .getOrElse(org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration)


+    val partitionFsConf = inputPartition.asInstanceOf[GlutenPartition].fsConf
+    val extraConf = (partitionFsConf +
+      (GlutenConfig.COLUMNAR_CUDF_ENABLED.key -> enableCudf.toString)).asJava
    val transKernel = NativePlanEvaluator.create(BackendsApiManager.getBackendName, extraConf)



+  test("genPartitions embeds fs.s3a.* keys from Hadoop conf into GlutenPartition.fsConf") {
+    withHadoopConf(
+      "fs.s3a.access.key" -> "AKIAIOSFODNN7EXAMPLE",
+      "fs.s3a.secret.key" -> "wJalrXUtnFEMI"
+    ) {
+      val partitions = api.genPartitions(emptyWsCtx, Seq(Seq.empty), Seq.empty)
+      assert(partitions.size == 1)
+      val fsConf = partitions.head.asInstanceOf[GlutenPartition].fsConf
+      assert(
+        fsConf.contains("fs.s3a.access.key"),
+        s"Expected fs.s3a.access.key not found; got: ${fsConf.keys.mkString(", ")}")
+      assert(fsConf("fs.s3a.access.key") == "AKIAIOSFODNN7EXAMPLE")
+      assert(fsConf.contains("fs.s3a.secret.key"))
+      assert(fsConf("fs.s3a.secret.key") == "wJalrXUtnFEMI")
+    }
+  }


+  test("genPartitions embeds fs.azure.* keys from Hadoop conf into GlutenPartition.fsConf") {
+    withHadoopConf(
+      "fs.azure.account.auth.type.myaccount.dfs.core.windows.net" -> "OAuth",
+      "fs.azure.account.oauth.provider.type" -> "ClientCredentials"
+    ) {
+      val partitions = api.genPartitions(emptyWsCtx, Seq(Seq.empty), Seq.empty)


[VL] Passing hadoop related configurations to native

af6b23b

Signed-off-by: Yuan <yuanzhou@apache.org>

Copilot AI review requested due to automatic review settings June 25, 2026 13:14

github-actions Bot added CORE works for Gluten Core VELOX labels Jun 25, 2026

Copilot started reviewing on behalf of zhouyuan June 25, 2026 13:15 View session

add test

1d6f087

Signed-off-by: Yuan <yuanzhou@apache.org>

zhouyuan force-pushed the wip_pass_fs_options branch from e37d128 to 1d6f087 Compare June 25, 2026 13:19

Copilot AI reviewed Jun 25, 2026

View reviewed changes

fix

6595e1f

Signed-off-by: Yuan <yuanzhou@apache.org>

Copilot AI review requested due to automatic review settings June 25, 2026 15:16

Copilot started reviewing on behalf of zhouyuan June 25, 2026 15:17 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

fix

9b41c7f

Signed-off-by: Yuan <yuanzhou@apache.org>

zhouyuan force-pushed the wip_pass_fs_options branch from 2220efc to 9b41c7f Compare June 25, 2026 17:21

Copilot AI review requested due to automatic review settings June 25, 2026 17:21

Copilot started reviewing on behalf of zhouyuan June 25, 2026 17:29 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Potential fix for pull request finding

3e5abcd

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 26, 2026 14:17

Copilot started reviewing on behalf of zhouyuan June 26, 2026 14:17 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

zhouyuan requested a review from marin-ma June 26, 2026 15:07

marin-ma approved these changes Jun 26, 2026

View reviewed changes

Uh oh!

Conversation

zhouyuan commented Jun 25, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhouyuan commented Jun 25, 2026 •

edited by github-actions Bot

Loading