fix: dtype roundtrip guard

CONTRACT.md

@@ -0,0 +1,70 @@
+# Public API Contract
+
+This document defines the stability promise `pl-row-encode` makes to its users and the
+rules that keep that promise mechanically enforceable.
+
+## The promise (SemVer)
+
+We follow [Semantic Versioning](https://semver.org/). For a given **major** version:
+
+- The **public surface** is frozen: the names exported from `pl_row_encode.__all__`, and
+  their call signatures, do not change in a breaking way.
+- The **documented behavior** of those functions does not change in a breaking way.
+- The **token wire format** is stable: a token produced by any release within a major
+  version decodes correctly with any later release in the same major version.
+
+Additions (new functions, new *optional* keyword arguments, new accepted input types) are
+allowed in **minor** releases. Bug fixes go in **patch** releases.
+
+Anything that breaks one of the three guarantees above requires a **major** version bump
+and an entry in the changelog.
+
+## How it is enforced
+
+The contract is not a gentleman's agreement — it is executable. Two layers live under
+`tests/contract/`:
+
+| File | Guards |
+|------|--------|
+| `test_surface.py` | The exported symbols and their exact signatures. |
+| `test_behavior.py` | Documented round-trip behavior, dtypes, errors, and the token wire format (via a frozen golden token). |
+
+### The append-only rule
+
+Within a major version, the tests in `tests/contract/` are **append-only**:
+
+- You **may** add new contract tests (covering new, additive API).
+- You **may not** weaken, change, or delete an existing contract test.
+
+If a change requires editing an existing contract assertion, that change is breaking by
+definition. Do not edit the assertion to make CI green — instead, bump the major version
+and record the break here and in the changelog.
+
+A red contract suite means "this is a breaking change." That is the whole point.
+
+## Dtype safety: encodable implies decodable
+
+The core invariant is that a token we hand back can always be decoded — we never emit
+bytes that only blow up on the return trip. `encode` classifies every (recursively
+nested) input dtype into three tiers at encode time:
+
+- **Known-good** — dtypes verified to round-trip losslessly (the allowlist in
+  `classify_dtype`, guarded by the dtype-matrix property tests). These encode silently.
+- **Known-bad** — `Categorical`. Its category→string mapping lives in an external string
+  cache, not in the token, so decoding would panic. Hard-rejected at encode with an
+  actionable `ComputeError` (use `Enum`, or cast to `String`).
+- **Unknown** — anything we haven't vetted. We cannot safely probe it (the probe is the
+  dangerous decode itself), so encode emits a `UserWarning` that the token may fail or
+  panic on decode, then proceeds.
+
+The allowlist is keyed to the **compiled** polars crate version (`Cargo.toml`), not the
+user's runtime polars: the plugin decodes with its own embedded `polars-row`, so the set
+of safe dtypes is fixed at build time. Bumping the polars crate is the trigger to re-run
+the dtype-matrix tests and re-validate the allowlist.
+
+## Updating the surface snapshot
+
+`test_surface.py` pins exact signatures. When you make an **additive** change (e.g. a new
+function, or a new optional keyword argument), update the snapshot in the same PR and
+release it as a minor version. When you find yourself wanting to update it to *remove* or
+*retype* something, that is a major change.

pyproject.toml

@@ -37,6 +37,12 @@ build-backend = "maturin"
 python-source = "."
 module-name = "pl_row_encode._internal"
 
+[tool.pytest.ini_options]
+# Run the >>> examples in the public docstrings as tests, so the documented
+# usage can't silently rot. This is the third contract layer alongside
+# tests/contract/ (see CONTRACT.md).
+addopts = "--doctest-modules"
+
 [dependency-groups]
 dev = [
     "ruff>=0.15.15",

src/lib.rs

@@ -41,30 +41,88 @@ fn fields_from_prefixed(buf: &[u8]) -> PolarsResult<Vec<Field>> {
     deserialize_header(&buf[4..end])
 }
 
-/// Categorical's category->string mapping lives in an external string cache, not in the
-/// dtype or the row bytes, so a token can't carry it -- decoding would panic in
-/// `cat_to_str`. Reject it (recursing through nested containers) at encode time with an
-/// actionable message instead of emitting a token that only blows up on decode. Enum is
-/// fine: its categories live in the dtype and ride along in the serialized header.
-fn reject_categorical(dtype: &DataType) -> PolarsResult<()> {
+/// Classify a dtype against three tiers to uphold the invariant that anything we encode
+/// can be decoded (see CONTRACT.md):
+///
+///   * **known-good** -- dtypes whose polars-row encode/decode we've verified round-trips
+///     losslessly (the allowlist below, guarded by the dtype-matrix property tests). These
+///     encode silently.
+///   * **known-bad** -- `Categorical`. Its category->string mapping lives in an external
+///     string cache, not in the dtype or row bytes, so a token can't carry it; decoding
+///     panics in `cat_to_str`. Hard-rejected with an actionable message instead of
+///     emitting a token that blows up on decode. (Enum is fine: its categories live in the
+///     dtype and ride along in the serialized header.)
+///   * **unknown** -- anything else (new/exotic dtypes we haven't vetted). We can't safely
+///     probe it (the probe is the dangerous decode), so its name is collected and the
+///     caller emits a warning: the token may fail or panic on decode.
+///
+/// Recurses through `List` / `Array` / `Struct` so a nested offender is caught too. The
+/// allowlist is keyed to the compiled polars crate version (Cargo.toml), not the user's
+/// runtime polars, since the plugin decodes with its own embedded `polars-row`.
+fn classify_dtype(dtype: &DataType, unknown: &mut Vec<String>) -> PolarsResult<()> {
     match dtype {
         DataType::Categorical(_, _) => polars_bail!(
             ComputeError:
             "Categorical columns cannot be encoded: the category mapping is not embeddable \
              in a token. Use Enum, or cast to String before encoding."
         ),
-        DataType::List(inner) => reject_categorical(inner),
-        DataType::Array(inner, _) => reject_categorical(inner),
+
+        // Allowlist: leaves verified to round-trip (see tests/test_property.py).
+        DataType::Boolean
+        | DataType::UInt8
+        | DataType::UInt16
+        | DataType::UInt32
+        | DataType::UInt64
+        | DataType::Int8
+        | DataType::Int16
+        | DataType::Int32
+        | DataType::Int64
+        | DataType::Int128
+        | DataType::Float32
+        | DataType::Float64
+        | DataType::Decimal(_, _)
+        | DataType::String
+        | DataType::Binary
+        | DataType::Date
+        | DataType::Time
+        | DataType::Datetime(_, _)
+        | DataType::Duration(_)
+        | DataType::Enum(_, _) => Ok(()),
+
+        // Containers: known-good iff their inner dtype(s) are.
+        DataType::List(inner) | DataType::Array(inner, _) => classify_dtype(inner, unknown),
         DataType::Struct(fields) => {
             for f in fields {
-                reject_categorical(f.dtype())?;
+                classify_dtype(f.dtype(), unknown)?;
             }
             Ok(())
         }
-        _ => Ok(()),
+
+        other => {
+            unknown.push(format!("{other}"));
+            Ok(())
+        }
     }
 }
 
+/// Emit a Python `UserWarning` naming dtypes that aren't on the known-good allowlist.
+/// Reaching Python from inside a polars expr requires grabbing the GIL; failures to warn
+/// are swallowed (a warning must never break an encode).
+fn warn_unknown_dtypes(names: &[String]) {
+    use pyo3::prelude::*;
+
+    let msg = format!(
+        "pl_row_encode: dtype(s) [{}] are not in the known round-trip-safe set; the \
+         resulting token may fail or panic on decode. If decode works, please report it \
+         so the dtype can be allowlisted.",
+        names.join(", "),
+    );
+    let _ = Python::attach(|py| -> PyResult<()> {
+        py.import("warnings")?.call_method1("warn", (msg,))?;
+        Ok(())
+    });
+}
+
 #[polars_expr(output_type=Binary)]
 fn row_encode(inputs: &[Series]) -> PolarsResult<Series> {
     if inputs.is_empty() {
@@ -74,8 +132,12 @@ fn row_encode(inputs: &[Series]) -> PolarsResult<Series> {
     let columns: Vec<Column> = inputs.iter().cloned().map(Column::from).collect();
     let fields: Vec<Field> = inputs.iter().map(|s| s.field().into_owned()).collect();
 
+    let mut unknown: Vec<String> = Vec::new();
     for field in &fields {
-        reject_categorical(field.dtype())?;
+        classify_dtype(field.dtype(), &mut unknown)?;
+    }
+    if !unknown.is_empty() {
+        warn_unknown_dtypes(&unknown);
     }
 
     let header = serialize_header(&fields)?;

tests/contract/test_behavior.py

@@ -0,0 +1,121 @@
+"""Freeze the *behavior* users depend on: round-trips, dtypes, errors, wire format.
+
+APPEND-ONLY within a major version. You may add tests for new, additive behavior; you may
+not weaken or delete an assertion here. A red test in this file means a breaking change.
+See CONTRACT.md.
+"""
+
+import polars as pl
+import pytest
+from polars.testing import assert_series_equal
+
+from pl_row_encode import (
+    decode,
+    decode_peek,
+    decode_series,
+    encode,
+    encode_series,
+    get_header,
+)
+
+# --- round-trip behavior --------------------------------------------------
+
+
+def test_encode_produces_binary_token():
+    df = pl.DataFrame({"x": [1, 2], "y": ["a", "b"]})
+    out = df.select(encode("x", "y").alias("tok"))
+    assert out.schema == pl.Schema({"tok": pl.Binary})
+
+
+def test_lazy_roundtrip_via_decode_with_header():
+    df = pl.DataFrame({"x": [1, 2], "y": ["a", "b"]}).select(
+        encode("x", "y").alias("tok")
+    )
+    header = get_header(df.to_series())
+    rows = (
+        df.select(decode("tok", schema_header=header).alias("row"))
+        .to_series()
+        .to_list()
+    )
+    assert rows == [{"x": 1, "y": "a"}, {"x": 2, "y": "b"}]
+
+
+def test_lazy_roundtrip_via_decode_peek():
+    df = pl.DataFrame({"x": [1, 2], "y": ["a", "b"]}).select(
+        encode("x", "y").alias("tok")
+    )
+    rows = df.select(decode_peek(df, "tok").alias("row")).to_series().to_list()
+    assert rows == [{"x": 1, "y": "a"}, {"x": 2, "y": "b"}]
+
+
+def test_eager_roundtrip_via_series():
+    tok = encode_series(pl.Series("x", [10, 20]))
+    assert tok.dtype == pl.Binary
+    assert decode_series(tok).to_list() == [{"x": 10}, {"x": 20}]
+
+
+def test_encode_series_matches_encode_expr():
+    df = pl.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
+    via_expr = df.select(tok=encode("a", "b"))["tok"]
+    via_series = encode_series(df["a"], df["b"])
+    assert_series_equal(via_series.rename("tok"), via_expr)
+
+
+# --- documented errors ----------------------------------------------------
+
+
+def test_encode_requires_input():
+    with pytest.raises(ValueError, match="at least one column"):
+        encode()
+
+
+def test_encode_series_requires_input():
+    with pytest.raises(ValueError, match="at least one series"):
+        encode_series()
+
+
+def test_get_header_rejects_all_null_series():
+    with pytest.raises(ValueError, match="all-null"):
+        get_header(pl.Series("x", [None], dtype=pl.Binary))
+
+
+def test_get_header_rejects_short_token():
+    with pytest.raises(ValueError, match="too short"):
+        get_header(b"\x00")
+
+
+def test_decode_peek_rejects_no_non_null_token():
+    df = pl.DataFrame({"tok": pl.Series([None], dtype=pl.Binary)})
+    with pytest.raises(ValueError, match="no non-null token"):
+        decode_peek(df, "tok")
+
+
+# --- header shape ---------------------------------------------------------
+
+
+def test_header_is_length_prefixed():
+    import struct
+
+    tok = encode_series(pl.Series("x", [1, 2, 3]))
+    header = get_header(tok)
+    (header_len,) = struct.unpack_from("<I", header, 0)
+    assert header_len == len(header) - 4
+    # header sniffed from a single token equals header sniffed from the series
+    assert get_header(tok[0]) == header
+
+
+# --- token wire format (cross-version stability) --------------------------
+
+# A token produced by an early release. Any later release in the same major version MUST
+# still decode it to the same rows. If this test fails, the on-disk/over-the-wire token
+# format changed -- that is a breaking change for anyone who persisted tokens. Bump major.
+GOLDEN_TOKEN = bytes.fromhex(
+    "220000000200000000000000010000000000000078090000000100000000000000790e000000018000"
+    "0000000000010161"
+)
+GOLDEN_ROWS = [{"x": 1, "y": "a"}]
+
+
+def test_golden_token_still_decodes():
+    decoded = decode_series(pl.Series("tok", [GOLDEN_TOKEN], dtype=pl.Binary)).to_list()
+    assert decoded == GOLDEN_ROWS

tests/contract/test_surface.py

@@ -0,0 +1,59 @@
+"""Freeze the *shape* of the public API: exported names and exact signatures.
+
+APPEND-ONLY within a major version. A failure here means the public surface drifted.
+If the drift is additive (new symbol, new optional kwarg), update the snapshot in this
+file and ship a minor release. If it removes or retypes something, it is a breaking
+change: bump the major version and record it in CONTRACT.md.
+
+See CONTRACT.md for the full policy.
+"""
+
+import inspect
+
+import pl_row_encode as m
+
+# The frozen set of public symbols. Add to this set for additive releases; never remove.
+EXPORTED = {
+    "encode",
+    "encode_series",
+    "get_header",
+    "decode",
+    "decode_peek",
+    "decode_series",
+}
+
+# Exact string form of each public signature. Adding an *optional* keyword argument is
+# additive (update the snapshot, minor release); changing or removing a parameter is
+# breaking (major release).
+SIGNATURES = {
+    "encode": "(*exprs: 'IntoExpr') -> 'pl.Expr'",
+    "encode_series": "(*series: 'pl.Series') -> 'pl.Series'",
+    "get_header": "(token: 'bytes | pl.Series') -> 'bytes'",
+    "decode": "(expr: 'IntoExpr', *, schema_header: 'bytes') -> 'pl.Expr'",
+    "decode_peek": "(frame: 'Frame', column: 'str') -> 'pl.Expr'",
+    "decode_series": "(s: 'pl.Series') -> 'pl.Series'",
+}
+
+
+def test_all_matches_frozen_set():
+    assert set(m.__all__) == EXPORTED
+
+
+def test_exported_symbols_are_importable():
+    for name in EXPORTED:
+        assert hasattr(m, name), f"{name} is exported in __all__ but not importable"
+
+
+def test_no_undeclared_signatures():
+    # Every exported symbol must have a pinned signature below, so new public API can't
+    # slip in without consciously freezing its signature.
+    assert set(SIGNATURES) == EXPORTED
+
+
+def test_signatures_frozen():
+    for name, expected in SIGNATURES.items():
+        actual = str(inspect.signature(getattr(m, name)))
+        assert actual == expected, (
+            f"signature of {name} changed: {actual!r} != frozen {expected!r}. "
+            "If additive, update SIGNATURES (minor release); if not, bump major."
+        )