Skip to content

refactor(data): migrate Daft SQL bindings to new connectors - #63

Merged
jiangxt2 merged 3 commits into
masterfrom
feat/split-daft-olap-bindings
Aug 16, 2026
Merged

refactor(data): migrate Daft SQL bindings to new connectors#63
jiangxt2 merged 3 commits into
masterfrom
feat/split-daft-olap-bindings

Conversation

@jiangxt2

Copy link
Copy Markdown
Owner

Purpose

This PR migrates the Daft ClickHouse/Doris reads from the daft-olap-connectors package to two per-connector public facades: daft-clickhouse and daft-doris. The old package identity is removed from binding descriptors, dependency checks, install hints and capability receipts, and explicit parallel sharding now fails closed because the target connectors only support single and auto splits.

What Changes

  • Add DaftClickHouseBinding and DaftDorisBinding backed by the public daft_clickhouse.read_clickhouse and daft_doris.read_doris facades; new binding IDs daft_clickhouse.daft.clickhouse and daft_doris.daft.doris.
  • Remove the daft-olap-connectors identity from descriptors, dependency checks, install hints and capability receipts; old package references are cleared from docs and E2E notes.
  • Fail closed on explicit parallel shard requirements (diagnostic parallel_sql_read_unsupported) and reject unsupported Doris transports.
  • Declare database drivers in extras (clickhouse-connect[arrow,async], PyMySQL, ADBC) and lock Daft 0.7.23; the unpublished daft-clickhouse and daft-doris wheels stay outside uv.lock and are installed separately in E2E/runtime environments.
  • Update binding tests for fail-closed parallel, Doris mysql/flight transports, descriptor identity and missing-package diagnostics.

Breaking change: the Daft SQL binding IDs change from daft_olap.daft.clickhouse / daft_olap.daft.doris to daft_clickhouse.daft.clickhouse / daft_doris.daft.doris without an old-ID fallback; persisted configurations or receipts using the old IDs require migration.

Test Plan

  • Affected data binding tests pass (75 passed, 1 skipped)
  • Ruff format --check, ruff check and diff check pass
  • Integration tests pass (pending local connector wheel injection and real Docker/Ray E2E)

Open Source Checklist

  • No internal credentials, URLs, or tokens exposed
  • New external dependencies reviewed for license compatibility (pending review of adbc-driver-manager, adbc-driver-flightsql and related driver additions)

Replace the daft-olap-connectors integration with per-connector Daft
bindings backed by the public daft-clickhouse and daft-doris facades.

- Add DaftClickHouseBinding / DaftDorisBinding using
  daft_clickhouse.read_clickhouse and daft_doris.read_doris; new binding
  IDs daft_clickhouse.daft.clickhouse and daft_doris.daft.doris
- Replace the old distribution identity in descriptors, dependency
  checks, install hints and capability receipts
- Fail closed on explicit 'parallel' sharding
  (parallel_sql_read_unsupported) and reject unsupported Doris transports
- Declare drivers in extras (clickhouse-connect[arrow,async], PyMySQL,
  ADBC) and lock daft 0.7.23
- Keep the unpublished daft-clickhouse and daft-doris wheels outside
  uv.lock; E2E and runtimes install them separately
- Update tests for fail-closed parallel, mysql/flight transports,
  descriptor identity and missing-package diagnostics; clear old package
  references from docs and E2E notes

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
The CI public-API checker requires every __all__ export to carry a @publicapi annotation; DaftClickHouseBinding and DaftDorisBinding were exported by the new facade modules without one, failing the Lint job.

- Mark both classes @publicapi(stability=Stability.ALPHA), matching their adapter-level support status

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
DaftClickHouseBinding and DaftDorisBinding are now @publicapi targets; the generated data.md page was stale, failing the docs contracts check in Unit Tests and Documentation Build jobs.

- Regenerate docs/reference/api/data.md via tools/generate_public_api_reference.py

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2
jiangxt2 marked this pull request as ready for review August 16, 2026 07:01
@jiangxt2
jiangxt2 merged commit 2ce4e6d into master Aug 16, 2026
14 checks passed
@jiangxt2
jiangxt2 deleted the feat/split-daft-olap-bindings branch August 16, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant