Add a Databricks SQL cache to PyAirbyte

[iherdt-airbyte]

Summary

Add a native Databricks SQL cache implementation to PyAirbyte (airbyte.caches.databricks), alongside the existing DuckDB, MotherDuck, Postgres, Snowflake, and BigQuery caches. It would pair with the existing destination-databricks (Delta Lake on Unity Catalog) the same way SnowflakeCache and BigQueryCache pair with their destination connectors.

Motivation

Code-first data engineering teams want to run an entire pipeline inside their own Python/Airflow stack: extract with PyAirbyte, transform with dbt, and land in Databricks, all in one repo and one runtime, without bouncing between an orchestrator, a separate sync service, and a connector UI.

Today PyAirbyte can write to Databricks only via the destination-databricks connector through Destination.write(). Because that destination is Java-based, it runs as a Docker image, so the path requires Docker on every Airflow worker. That breaks the "pure Python, no containers" model that makes the SQL caches attractive in the first place.

A native Databricks cache, implemented in-process like the other SQL caches, would:

• Remove the Docker dependency for Databricks-bound pipelines.
• Let cache and final destination be the same system, instead of using a throwaway DuckDB buffer.
• Give teams a single, portable, Python-native end-to-end flow (read -> cache in Databricks -> dbt -> done) runnable anywhere PyAirbyte runs.

Proposed solution

• Add DatabricksCache under airbyte.caches.databricks, following the existing CacheBase / SqlConfig pattern used by SnowflakeCache and BigQueryCache.
• Back it with a SQLAlchemy-compatible Databricks dialect (e.g. databricks-sql-connector / databricks-sqlalchemy) against a SQL warehouse / Unity Catalog.
• Set paired_destination_name / paired_destination_config_class to destination-databricks so clone_as_cloud_destination_config() and destination pairing work consistently with the other caches.
• Support the usual cache responsibilities: stream catalog, state tracking, and read/write through the SQL backend.

Alternatives considered

• Destination.write() with the Databricks destination connector works today but requires Docker on the runtime, which defeats the Python-native goal.
• DuckDB/Postgres/Snowflake/BigQuery cache as an intermediate buffer, then load to Databricks adds a hop and still needs the Docker-based destination for the final load.

Additional context

An enterprise design partner with a Databricks + Airflow + dbt stack has offered to help test an alpha and work with us toward release. Happy to connect the maintainers with them to validate the design and shake out auth (token vs OAuth) and Unity Catalog specifics.

[mtrienis]

Definitely +1 - IMO, this will really fill a gap for data engineers who want a programmable solution. Please let me know if I can help with this.