daft-clickhouse

daft-clickhouse is an independent Daft community connector for reading ClickHouse physical tables and appending Arrow batches to existing ClickHouse MergeTree-family tables.

pip install "daft-clickhouse[clickhouse]"

import daft
from daft_clickhouse import read_clickhouse, write_clickhouse

events = read_clickhouse(
    host="localhost",
    database="analytics",
    table="events",
    username="connector",
    password="...",
)

write_clickhouse(
    daft.from_pydict({"id": [1], "value": ["new"]}),
    host="localhost",
    database="analytics",
    table="write_events",
    username="connector",
    password="...",
).collect()

Writing is deliberately append-only and currently requires Daft NativeRunner. The target table must already exist, input columns are validated against its discovered schema, and each confirmed batch is reported as source input rows and bytes. The connector does not provide upsert, overwrite, DDL, connector-owned retries, global transactions, or exactly-once guarantees. retry_policy="none" rejects any future connector retry policy rather than silently changing delivery semantics. It does not monkey-patch Daft's existing DataFrame.write_clickhouse method.

The default write path is PyArrow Table -> clickhouse-connect.insert_arrow(). insert_mode="async" is explicit and still waits for the server-side asynchronous insert flush. The accepted target engines are MergeTree, ReplacingMergeTree, SummingMergeTree, AggregatingMergeTree, CollapsingMergeTree, and ReplicatedMergeTree. Other engines, including Distributed targets, are rejected until their complete write contract is certified.

Daft 0.7.23 does not propagate synchronous consumer early-close into a Python async DataSource task. Read consumers should fully drain results; cancellation and strict end-to-end backpressure remain upstream compatibility limitations. split="auto" is for a direct, single-server endpoint only and is not certified behind load balancing or ClickHouse Cloud routing.

See writing, consistency, and security for the public contract.