feat: QWP ingestion protocol

CHANGELOG.rst

@@ -5,6 +5,123 @@ Changelog
 
 =========
 
+Unreleased
+----------
+
+Features
+~~~~~~~~
+
+QWP Ingestion Protocol
+**********************
+
+Adds support for the QuestDB Wire Protocol (QWP) alongside the existing
+ILP transports.
+
+- **QWP/UDP** (``qwpudp::``): fire-and-forget datagram ingestion,
+  defaulting to port 9007. New configuration keys ``max_datagram_size``
+  and ``multicast_ttl``; ``protocol_version`` does not apply.
+- **QWP/WebSocket** (``qwpws::`` / ``qwpwss::``): acknowledged streaming
+  ingestion with frame-sequence-number (FSN) tracking. New ``Sender``
+  methods ``flush_and_get_fsn``, ``flush_and_keep_and_get_fsn``,
+  ``published_fsn``, ``acked_fsn``, ``await_acked_fsn``, ``drive_once``,
+  ``poll_qwp_ws_error``, ``qwp_ws_errors_dropped`` and ``close_drain``.
+  Server diagnostics are reported through a ``qwp_ws_error_handler``
+  callback or polled as :class:`QwpWsError` values; terminal server
+  rejections raise :class:`IngressServerRejectionError`.
+
+Additional configuration keys ``tls_roots_password``,
+``retry_max_backoff_millis`` and ``qwp_ws_progress`` are also accepted.
+Every new key is equally available as a ``Sender`` / ``Sender.from_conf`` /
+``Sender.from_env`` keyword argument (``max_datagram_size``,
+``multicast_ttl``, ``tls_roots_password``, ``retry_max_backoff``,
+``qwp_ws_progress`` and ``qwp_ws_error_handler``).
+
+Buffer Factories
+****************
+
+``Buffer.ilp()`` and ``Buffer.qwp()`` construct protocol-specific
+buffers. Direct ``Buffer(...)`` construction is deprecated in favour of
+these factories and ``Sender.new_buffer()``.
+
+Query Egress
+************
+
+Adds :class:`Client` with :meth:`Client.query`, returning a
+:class:`QueryResult` that streams rows as Arrow record batches over the
+QWP/WebSocket read endpoint. Results can be consumed via ``to_arrow``,
+``to_pandas``, ``to_polars``, ``iter_arrow``, ``iter_pandas``,
+``iter_polars`` or the Arrow C stream PyCapsule protocol
+(``__arrow_c_stream__``). ``to_polars`` / ``iter_polars`` use pyarrow to
+buffer failover-safe batches; ``__arrow_c_stream__`` (consumed as
+``polars.from_arrow(result)``) is the pyarrow-free polars path. SYMBOL
+columns are dictionary-encoded on the wire and map to a pandas
+``Categorical`` (``to_pandas`` / ``iter_pandas``) or a polars
+``Categorical`` (``to_polars`` / ``iter_polars``), the latter sharing one
+persistent ``Categories`` identity across streamed batches so
+``polars.concat`` stitches them without a categories-mismatch error.
+
+``to_pandas`` / ``iter_pandas`` default to a native (no-pyarrow) build
+straight from the QWP column buffers: a nullable integer column becomes a
+pandas nullable ``Int*`` when it contains nulls and plain numpy otherwise,
+``double`` stays numpy with ``NaN``, ``TIMESTAMP`` → ``datetime64``, and
+the QuestDB column kinds are recorded in ``df.attrs['questdb']`` for a type
+round-trip through :meth:`Client.dataframe`. Pass
+``dtype_backend="pyarrow"`` / ``"numpy_nullable"`` (or ``types_mapper=``)
+to select the pyarrow-backed conversion instead.
+
+:class:`Client` is a context manager and exposes :meth:`Client.close` and
+:meth:`Client.reap_idle` for pooled-connection lifecycle management.
+
+Columnar DataFrame Ingestion
+****************************
+
+Adds :meth:`Client.dataframe`, ingesting pandas / polars / pyarrow and
+any Arrow C Data Interface object over QWP/WebSocket. A
+``schema_overrides`` keyword reclassifies columns as ``symbol``,
+``ipv4``, ``char`` or ``geohash`` (e.g. ``{'addr': 'ipv4', 'loc':
+('geohash', 20)}``).
+
+The designated-timestamp argument ``at`` is the timestamp column itself,
+given by name (``str``) or position (``int``); unlike
+:meth:`Sender.dataframe` / :meth:`Buffer.dataframe` it does not accept a
+scalar ``datetime`` / ``TimestampNanos`` / ``ServerTimestamp``. A frame
+produced by :meth:`QueryResult.to_pandas` round-trips back to the same
+QuestDB column types automatically: the kinds recorded in
+``df.attrs['questdb']`` and pandas nullable extension dtypes are honoured.
+
+Errors
+******
+
+Adds :class:`UnsupportedDataFrameShapeError` (raised when a DataFrame
+cannot be expressed on the QWP columnar path) and the
+:class:`IngressErrorCode` members ``ServerRejection``,
+``ArrowUnsupportedColumnKind``, ``ArrowIngest`` and ``Cancelled``.
+:class:`IngressError` gains a ``qwp_ws_error`` property exposing the
+structured :class:`QwpWsError` view on a server-side QWP/WebSocket
+rejection.
+
+Build & dependencies
+~~~~~~~~~~~~~~~~~~~~~~
+
+- The minimum supported Python is raised from 3.8 to **3.10**; Python 3.8
+  and 3.9 are no longer supported.
+- ``numpy>=1.21.0`` is now a hard runtime dependency (previously it was
+  pulled in only via the ``dataframe`` extra).
+- **pyarrow is now optional.** It is imported lazily, only when actually
+  needed (``pd.ArrowDtype`` columns, pyarrow sources, ``schema_overrides``,
+  and the ``to_arrow`` / ``iter_arrow`` / ``dtype_backend`` helpers). The
+  ``to_polars`` / ``__arrow_c_stream__`` egress paths and the default
+  ``to_pandas`` / ``iter_pandas`` work without pyarrow.
+- The ``dataframe`` extra now pins ``pandas>=1.3.5`` and
+  ``pyarrow>=10.0.1``.
+
+Deprecations
+~~~~~~~~~~~~
+
+- Direct ``Buffer(...)`` construction is deprecated and emits a
+  ``DeprecationWarning``. Use ``Buffer.ilp()``, ``Buffer.qwp()`` or
+  ``Sender.new_buffer()`` instead.
+
 4.1.0 (2025-11-28)
 ------------------