VFS support for Databricks Unity Catalog Volumes (Files API)

[mattcasters]

Feature request

Add an Apache Commons VFS provider for Databricks Unity Catalog Volumes (and optionally Workspace files), so Hop can browse, read, write, list, and delete volume paths the same way it does for MinIO, Azure, S3, Google Storage, etc.

Motivation

• Modern Databricks workspaces often have public DBFS root disabled. Artifact storage for Jobs / native Spark deploy is typically a UC Volume (/Volumes/<catalog>/<schema>/<volume>/…).
• Hop’s Databricks Job Run (Deploy & run) already uploads to Volumes via the Files API (PUT /api/2.0/fs/files/…). That is a special-case client, not reusable Hop VFS.
• A proper VFS scheme would allow:
  • File dialogs / browse volumes from Hop GUI
  • File Input/Output and other VFS-aware transforms/actions against volumes
  • Optional reuse for deploy (source fat jar or target volume URIs)
  • Consistent auth via existing Databricks Connection metadata (host + PAT)

Background: API already exists

Databricks exposes volumes and workspace files through the workspace Files API:

• Docs: https://docs.databricks.com/api/workspace/files
• Volume file guide: https://docs.databricks.com/aws/en/volumes/volume-files

Operation	Endpoint (typical)
Upload	PUT /api/2.0/fs/files{path}?overwrite=true (application/octet-stream)
Download	GET /api/2.0/fs/files{path} (HTTP Range supported)
Create directory	PUT /api/2.0/fs/directories{path}/
List directory	GET /api/2.0/fs/directories{path}
Delete	DELETE on files/directories

Auth: Bearer PAT (same as Jobs API / Hop Databricks Connection).

Note: Legacy DBFS API (/api/2.0/dbfs/*) is separate and often blocked for public root; VFS should target the Files API, not DBFS root.

Hop already implements Volume upload in the Databricks tech plugin (RestDatabricksJobsClient / Files API path encoding) for Deploy & run.

Proposed design (high level)

Align with existing Hop VFS plugins (e.g. MinIO named connections):

1. Extract a small Databricks Files client (upload / download / list / mkdir / delete) shared by Jobs deploy and VFS.

2. Add VFS under plugins/tech/databricks (or sibling module):

   • @VfsPlugin + IVfs
   • AbstractOriginatingFileProvider → FileSystem → FileObject

3. Auth: reuse Databricks Connection metadata (host, token, optional API base).

4. URI scheme (suggested): connection-named scheme, e.g.

   databricks-free:///Volumes/apache-hop/default/jars/hop-native.jar
   

   Path = absolute UC/Workspace path. Alternative: dbxvol://{connection}/Volumes/....

5. Capabilities (MVP): READ/WRITE content, LIST, CREATE/DELETE, type/size; optional Range GET later.

6. WireMock unit tests; docs under native Spark / Databricks.

Out of scope (MVP)

• Full Databricks Java SDK (pure HttpClient is enough, consistent with Jobs client)
• Delta table APIs (tabular data)
• Replacing cloud object-store VFS (s3a/abfss) that volumes already sit on
• Spark-side path dialect (/Volumes/... on cluster remains separate from Hop VFS URIs)

Path dialect note

Hop VFS URIs are for Hop-side I/O only. Cluster Spark / Jobs still use /Volumes/... (or cloud URIs). Document this clearly (same class of issue as Hop s3:// vs Spark s3a://).

Acceptance criteria

• Named Databricks Connection can be used as a VFS scheme
• Browse / list UC Volume directories in Hop file dialogs
• Read and write files via VFS (including large jars within Files API limits)
• Deploy path can optionally share the same Files client (no behaviour regression for Job Run)
• Unit tests (WireMock) + short user docs

Related

• Databricks Job Run Deploy & run (existing Files API upload to Volumes)
• Metadata: Databricks Connection (Jobs API / PAT)
• Prior art: MinIO / WebDAV / Azure VFS plugins in Hop

Priority / when

Useful after classic-cluster Deploy & run is solid on Premium/classic-capable workspaces; not blocking first JAR deploy if Volume base path + Files API upload already work.

[mattcasters]

Examine relationship to #3941
Examine relationship to #2639