issue #7567 : VFS support for Databricks UC Volumes (Files API)

core/src/main/java/org/apache/hop/metadata/api/HopMetadataPropertyType.java

@@ -101,6 +101,7 @@ public enum HopMetadataPropertyType {
   VFS_MINIO_CONNECTION,
   VFS_S3_CONNECTION,
   VFS_WEBDAV_CONNECTION,
+  VFS_DATABRICKS_CONNECTION,
 
   // DATA STREAM
   DATA_STREAM,

docs/hop-user-manual/modules/ROOT/nav.adoc

@@ -396,6 +396,7 @@ under the License.
 * xref:vfs.adoc[Virtual File System]
 ** xref:vfs/aws-s3-vfs.adoc[Amazon Web Services S3]
 ** xref:vfs/azure-blob-storage-vfs.adoc[Microsoft Azure Storage]
+** xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace]
 ** xref:vfs/dropbox-vfs.adoc[Dropbox]
 ** xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage]
 ** xref:vfs/google-drive-vfs.adoc[Google Drive]
@@ -455,6 +456,8 @@ under the License.
 ** xref:metadata-types/azure-authentication.adoc[Azure Connection]
 ** xref:metadata-types/beam-file-definition.adoc[Beam File Definition]
 ** xref:metadata-types/cassandra/cassandra-connection.adoc[Cassandra Connection]
+** xref:metadata-types/databricks-connection.adoc[Databricks Connection]
+** xref:metadata-types/databricks-vfs-connection.adoc[Databricks VFS Connection]
 ** xref:metadata-types/data-set.adoc[Data Set]
 ** xref:metadata-types/execution-data-profile.adoc[]
 ** xref:metadata-types/execution-information-location.adoc[]

docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspective-execution-information.adoc

@@ -31,6 +31,8 @@ The Execution Information perspective provides an overview of execution informat
 
 The execution information displayed in this perspective is not collected automatically.  You need to specify to which xref:metadata-types/execution-information-location.adoc[location] you want to send execution information in the xref:metadata-types/pipeline-run-config.adoc[Pipeline run configuration] or xref:metadata-types/workflow-run-config.adoc[Workflow run configuration] of your choice.
 
+TIP: Locations can use a VFS URI (for example a Databricks Volume via a named scheme). That lets you inspect native Spark jobs that wrote execution JSON on the cluster from this perspective on your laptop. See xref:pipeline/spark/databricks.adoc#exec-info-on-volume[Execution information on a UC Volume].
+
 == Navigation
 
 You can navigate to it by clicking the image:icons/execution.svg[width="24px"] icon from the vertical perspective toolbar in the Hop GUI.

docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-connection.adoc

@@ -53,8 +53,24 @@ Use this metadata type when orchestrating jobs from Hop (for example the upcomin
 
 Use **Test connection** in the editor to call the workspace identity endpoint (SCIM *Me*) or fall back to listing jobs. A successful test shows the resolved user name or host.
 
+== VFS
+
+For Hop file dialogs and VFS-aware transforms, create a separate
+xref:metadata-types/databricks-vfs-connection.adoc[Databricks VFS Connection] under **File storage**.
+That object’s **name** is the URI scheme; it **references this connection** for host and PAT.
+
+Example: Jobs connection `prod-workspace`, VFS connection name `dbx-jars`:
+
+----
+dbx-jars:///Volumes/apache-hop/default/jars/data.csv
+----
+
+See xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace VFS].
+
 == Related
 
+* xref:metadata-types/databricks-vfs-connection.adoc[Databricks VFS Connection] — named scheme for Volumes / Workspace
+* xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace VFS]
 * xref:pipeline/spark/databricks.adoc[Native Spark on Databricks] — Deploy & run, classic clusters, Volumes
 * xref:workflow/actions/databricks-job-run.adoc[Databricks Job Run] — trigger and wait for Jobs API runs
 * xref:database/databases/databricks.adoc[Databricks database (JDBC)] — SQL warehouse access

docs/hop-user-manual/modules/ROOT/pages/metadata-types/databricks-vfs-connection.adoc

@@ -0,0 +1,74 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:imagesdir: ../../assets/images/
+:page-pagination:
+:description: Named Databricks VFS connection for Unity Catalog Volumes and Workspace files (Files API)
+
+= Databricks VFS Connection
+
+== Description
+
+*Databricks VFS Connection* registers a **named VFS scheme** for Unity Catalog Volumes and Workspace files via the Databricks **Files API**.
+
+* The metadata **Name** is the URI scheme (like xref:metadata-types/minio-connection.adoc[MinIO Connection]).
+* **Authentication** is not stored on this object. Select an existing xref:metadata-types/databricks-connection.adoc[Databricks Connection] (workspace host + PAT).
+
+Example: VFS name `db-volume`, root path `/Volumes/apache-hop/default/testing`, Databricks Connection `prod-workspace`:
+
+----
+db-volume:///input/customers-1M.txt
+----
+
+resolves to workspace path `/Volumes/apache-hop/default/testing/input/customers-1M.txt`.
+
+This is **Hop-side I/O** only (GUI, local engine, classic file transforms).
+**Native Spark** Dataset I/O does **not** use Hop VFS schemes — use absolute `/Volumes/…` or cloud URIs (`s3a://…`, …) on the cluster.
+Do not use this connection as the primary bulk path for large Spark datasets (workspace Files API is not a high-throughput object-store client).
+See xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace VFS] and
+xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems on native Spark].
+
+== Options
+
+[options="header",cols="1,3"]
+|===
+|Option |Description
+
+|Name (URI scheme)
+|VFS scheme used in paths: `{name}:///…`. Prefer short names; do not reuse built-in schemes (`file`, `s3`, …).
+
+|Description
+|Optional free-text description.
+
+|Databricks Connection
+|Existing Jobs/workspace connection that supplies host and personal access token.
+
+|Root path
+|Absolute UC Volume or Workspace path that is the **root of this scheme**. Shortens URIs: `{name}:///input` → `{root}/input`. Leave empty if every URI must include full `/Volumes/…` or `/Workspace/…` paths.
+|===
+
+== Tips
+
+* Create the **Databricks Connection** first and use **Test connection** there (or on this editor, which validates the referenced connection).
+* Set **Root path** to your testing volume (for example `/Volumes/apache-hop/default/testing`) so workflow paths stay short.
+* In File → Open, type `{name}:///` to list the root path contents (when root is set).
+* One Jobs connection can back several VFS schemes (different root volumes).
+
+== Related
+
+* xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace VFS]
+* xref:metadata-types/databricks-connection.adoc[Databricks Connection]
+* xref:pipeline/spark/databricks.adoc[Native Spark on Databricks]

docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc

@@ -72,6 +72,8 @@ Options:
 * *Persistence delay*: This is the maximum time to wait before execution information is written to disk, expressed in milliseconds.
 * *Maximum cache age*: This is the maximum time to keep execution information around in memory, before clearing it out, expressed in milliseconds.  It's advised to keep this below the expected duration of your workflow or pipeline.
 
+The root folder can be a local path or a **VFS URI**. For example, on Databricks you can store execution JSON on a Unity Catalog Volume via a xref:metadata-types/databricks-vfs-connection.adoc[Databricks VFS Connection] (`db-volume:///executions`) so a native Spark job on the cluster and the laptop GUI share the same location. Full walkthrough: xref:pipeline/spark/databricks.adoc#exec-info-on-volume[Execution information on a UC Volume].
+
 === Remote location
 
 A remote location allows you to write execution information to a Hop Server.

docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/databricks.adoc

@@ -16,7 +16,7 @@ under the License.
 ////
 [[NativeSparkOnDatabricks]]
 :imagesdir: ../../../assets/images
-:description: Run Apache Hop native Spark (MainSpark JAR) jobs on Databricks — workspace tiers, classic clusters, UC Volumes, compute modes, and Deploy & run lessons learned.
+:description: Run Apache Hop native Spark (MainSpark JAR) jobs on Databricks — workspace tiers, classic clusters, UC Volumes, execution information on Volumes, compute modes, and Deploy & run lessons learned.
 :toc:
 :toclevels: 3
 
@@ -37,6 +37,7 @@ A minimal Hop project for this path typically includes:
 * A workflow with **Databricks Job Run** (Deploy & run) and optional **Databricks Job Wait**
 * A `work/` folder with the **native-provided** fat jar
 * Metadata: Databricks Connection, Native Spark pipeline run configuration
+* Optionally (for remote execution history from the laptop GUI): Databricks VFS Connection, **Caching File** execution information location on a UC Volume, and an environment config with `DATABRICKS_HOST` / `DATABRICKS_TOKEN` — see <<exec-info-on-volume>>
 
 .Sample `hello-databricks.hpl` in the project explorer (Generate rows → UUID → Dummy)
 image::pipeline/spark/databricks-pipeline-hello-databricks.png[Hello Databricks sample pipeline in Hop GUI: Generate rows, Random value (uuid), Dummy,width=85%]
@@ -176,6 +177,151 @@ If logs still show `POST /api/2.0/dbfs/create` for a `/Volumes/…` path, the ru
 
 Job library and MainSpark parameter paths for volumes stay as absolute `/Volumes/…` (no `dbfs:` scheme).
 
+=== Hop VFS for Volumes
+
+From Hop GUI and local transforms you can use a xref:metadata-types/databricks-vfs-connection.adoc[Databricks VFS Connection] (scheme = metadata name, auth via referenced Databricks Connection) to browse, read, and write the same Files API paths (for example `dbx-jars:///Volumes/apache-hop/default/jars/…`).
+That is **Hop-side I/O only** — it does not replace absolute `/Volumes/…` paths on the cluster or in Jobs API library URIs.
+See xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace VFS].
+
+[[exec-info-on-volume]]
+=== Execution information on a UC Volume
+
+You can store Hop **execution information** (pipeline graph, logs, transform metrics, and row samples from an execution data profile) on a **Unity Catalog Volume** and open it from the laptop Hop GUI — even though the pipeline ran on Databricks under `MainSpark`.
+
+That works because:
+
+* The **Caching File** location writes one JSON file per top-level execution (plus nested state/samples inside it).
+* The root folder can be a **Databricks VFS** URI (for example `db-volume:///executions`).
+* On the cluster, MainSpark resolves named VFS schemes from the **exported project metadata** and writes through the **Files API** using the Databricks Connection PAT.
+* On the laptop, the same location metadata + environment variables let the xref:hop-gui/perspective-execution-information.adoc[Execution Information Perspective] list and open those remote files.
+
+This is for **execution metadata** only (logs, metrics, sample rows), not for bulk Spark datasets. Prefer **Caching File** over plain File so writes are batched (fewer Files API round-trips). Native Spark Dataset I/O still uses `/Volumes/…` or cloud URIs — not the Hop VFS scheme.
+
+==== Pattern overview
+
+[options="header",cols="1,3"]
+|===
+|Piece |Role
+
+|xref:metadata-types/databricks-connection.adoc[Databricks Connection]
+|Workspace host + PAT (Jobs API and Files API auth for VFS)
+
+|Environment / lifecycle config
+|`DATABRICKS_HOST`, `DATABRICKS_TOKEN` (and optional cluster id). Must be active in Hop GUI **and** deployed to the job (`--HopConfigFile` / env-config artifact)
+
+|xref:metadata-types/databricks-vfs-connection.adoc[Databricks VFS Connection]
+|Named scheme (e.g. `db-volume`) + optional root path under a Volume
+
+|xref:metadata-types/execution-information-location.adoc[Execution Information Location] (**Caching File**)
+|Root folder = `{scheme}:///executions` (or another folder under the VFS root)
+
+|xref:metadata-types/pipeline-run-config.adoc[Pipeline run configuration] (**Native Spark**)
+|**Execution information location** = the Caching File location; optional **execution data profile** for sample rows
+
+|Deploy & run package
+|Exports connection, VFS, location, run config, and (when configured) the env config so the **driver** can write the same URI
+|===
+
+==== 1. Databricks Connection (variables for secrets)
+
+Create a Jobs/PAT connection. Prefer variables so the same metadata works on the laptop and on the cluster after Deploy & run ships the environment config.
+
+.Databricks Connection: workspace host and PAT via `${DATABRICKS_HOST}` / encrypted token variable
+image::pipeline/spark/databricks-connection.png[Databricks Connection metadata editor with variable-based host and personal access token,width=85%]
+
+==== 2. Environment configuration file
+
+Define at least host and token in a Hop **environment** or lifecycle configuration file that you use when opening the project in GUI and when Deploy & run attaches env config for MainSpark:
+
+----
+DATABRICKS_HOST=https://dbc-….cloud.databricks.com
+DATABRICKS_TOKEN=Encrypted …
+----
+
+Optional: `DATABRICKS_CLUSTER_ID` for existing-cluster deploy.
+
+.Environment configuration: `DATABRICKS_HOST`, encrypted `DATABRICKS_TOKEN`, optional cluster id
+image::pipeline/spark/databricks-environment-configuration-file.png[Hop environment configuration file editor with Databricks host, token, and cluster id variables,width=90%]
+
+[IMPORTANT]
+====
+If the GUI can browse the Volume but the cluster writes a **local** path such as `db-volume:///…` under the driver cwd, the driver did **not** resolve the named VFS scheme or the host/token variables. Confirm:
+
+* Env config is deployed (`env-config-….json` / `--HopConfigFile`) and contains the same variables.
+* Exported **metadata** includes the Databricks Connection, VFS connection, and execution location (Deploy & run package / metadata export).
+* The PAT can **write** to the Volume path.
+====
+
+==== 3. Databricks VFS Connection (scheme + Volume root)
+
+Create a VFS connection whose **name is the URI scheme**. Point **Root path** at a folder on a UC Volume (create that folder in Catalog Explorer if needed).
+
+.Databricks VFS Connection: scheme `db-volume`, connection `databricks-free`, root `/Volumes/…/testing`
+image::pipeline/spark/databricks-vfs-connection-to-volume.png[Databricks VFS Connection editor with scheme name, Databricks Connection, and Volume root path,width=85%]
+
+With root path `/Volumes/apache-hop/default/testing`:
+
+* Hop URI `db-volume:///executions` maps to workspace path `/Volumes/apache-hop/default/testing/executions`
+* Leave root empty only if you prefer full paths in every URI (`db-volume:///Volumes/…`)
+
+See xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace VFS].
+
+==== 4. Caching File execution information location
+
+Create an xref:metadata-types/execution-information-location.adoc[Execution Information Location]:
+
+* **Location type**: **Caching File location**
+* **Folder**: the VFS URI, e.g. `db-volume:///executions`
+* **Create folder?**: optional; you can also create `executions` on the Volume first
+* **Persistence delay**: multi-second (e.g. `5000`) so the driver does not write on every metric tick
+* **Maximum cache age**: long enough to cover the pipeline (e.g. hours for long jobs)
+
+.Caching File location `db-executions` with folder `db-volume:///executions`
+image::pipeline/spark/databricks-volume-caching-execution-information-location.png[Execution information location editor: Caching File type and Databricks VFS folder URI,width=85%]
+
+==== 5. Native Spark run configuration
+
+On the **Native Spark** pipeline run configuration used by Deploy & run:
+
+* **Execution information location** = the Caching File location (e.g. `db-executions`)
+* **Execution data profile** = optional but recommended (e.g. `first-last`) so the perspective can show sample rows
+* Engine type **Native Spark pipeline engine**; fat jar path is used for local `spark-submit` — on Databricks, Deploy & run supplies the library jar separately
+
+.Native Spark run configuration: execution location `db-executions`, data profile `first-last`
+image::pipeline/spark/databricks-native-spark-pipeline-run-configuration.png[Native Spark pipeline run configuration with execution information location and data profile,width=75%]
+
+==== 6. Run on Databricks, inspect from Hop GUI
+
+. Deploy & run the pipeline (or workflow with Job Run) with the run configuration above and env config attached.
+. Wait until the job finishes (or is far enough along that the cache has flushed at least once).
+. In Hop GUI, open the **Execution Information** perspective (`Ctrl-Shift-I`).
+. Select the location (e.g. `db-executions`), refresh, and open the pipeline execution.
+. Use **Info**, **Log**, **Metrics**, and **Data** tabs — same as a local run.
+
+.Execution Information perspective: `hello-databricks` from Volume location `db-executions` with cluster log text
+image::pipeline/spark/databricks-execution-perspective-looking-at-spark-execution.png[Execution Information perspective showing a Spark pipeline execution loaded from a Databricks Volume,width=95%]
+
+Typical log lines on a successful remote capture:
+
+----
+Using execution information location 'db-executions' …
+Prepared native Spark pipeline engine with run configuration 'databricks-native' …
+Reusing active SparkSession (version=4.1.0, …)
+Spark pipeline finished, result row count=…
+Registered … transform sample set(s) from Spark executors
+----
+
+JSON files on the Volume look like `{execution-id}.json` under the mapped folder (e.g. `/Volumes/…/testing/executions/`).
+
+==== Tips and limits
+
+* **Time filter** in the perspective defaults to a short window (e.g. `< 1h`). Widen it if the run is older.
+* **Parents only** / status filters still apply; turn them off if you expect a child execution.
+* Files API is **single-stream HTTP** — fine for execution JSON (tens to a few hundred KB typical); do not use this path for large data landings.
+* **Create parent folder** on the Volume before the first run if create-folder is off or list permissions are restricted.
+* Laptop and cluster must share the **same** connection name, VFS scheme name, location name, and variable names so metadata exported for Deploy & run matches what the GUI uses.
+* Deleting executions from the perspective removes the remote JSON (subject to PAT permissions).
+
 == Compute modes (Deploy & run)
 
 The **Cluster / compute** field accepts one of:
@@ -531,12 +677,17 @@ Databricks does **not** expose a standard content MD5/SHA for volume files compa
 * [ ] Cluster / compute = existing classic id **or** `new_cluster` with valid `node_type_id` / `spark_version`
 * [ ] Job id empty on first create; wait mode and timeout sized for upload + **job cluster cold start** (often 8+ minutes) + pipeline run
 * [ ] After success: note Job ID, Run ID, run page URL variables; open run in Databricks UI
+* [ ] (Optional remote execution history) VFS connection + Caching File location on a Volume; run config points at that location; env vars on driver; GUI Execution Information perspective can list the run — see <<exec-info-on-volume>>
 
 == Related
 
 * xref:workflow/actions/databricks-job-run.adoc[Databricks Job Run]
 * xref:workflow/actions/databricks-job-wait.adoc[Databricks Job Wait]
 * xref:metadata-types/databricks-connection.adoc[Databricks Connection]
+* xref:metadata-types/databricks-vfs-connection.adoc[Databricks VFS Connection]
+* xref:vfs/databricks-vfs.adoc[Databricks Volumes / Workspace VFS]
+* xref:metadata-types/execution-information-location.adoc[Execution Information Location]
+* xref:hop-gui/perspective-execution-information.adoc[Execution Information Perspective]
 * xref:pipeline/spark/getting-started-with-native-spark.adoc[Getting started with the native Spark engine]
 * xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native Spark run configuration]
 * xref:pipeline/spark/paths-and-filesystems.adoc[Paths and file systems]