docs: act on DX review — accuracy, taxonomy, funnel, and clarity fixes

Addresses the senior-DX review findings:

- Pattern count: reconcile four-vs-five — function-patterns intro now says five,
  tutorial says "three more patterns", home table adds the Buffering row.
- Home "Why VGI?" table: drop the false "single-threaded" claim; reframe the
  in-process vs. worker-process axis honestly and add an Arrow-IPC cost note.
- Taxonomy: use "pattern" in prose throughout (kept "shape" only for the visual
  glyphs); "kind" stays an internal CSS/asset name.
- Accuracy: Cloudflare DO backend uses httpx (not stdlib) — corrected. Pushdown
  how-to now uses params.current_pushdown_filters (already a decoded
  PushdownFilters), not a nonexistent params.pushdown_filters + deserialize.
- Funnel: how-to "Next steps" now link sibling how-to/concept before raw
  reference; reconcile state-storage vs shared-storage and concepts vs lifecycle.
- On-ramps: gloss vgi-serve on first use; explain the Python Client vs SQL
  ATTACH and the injected ctx; link execution_id to state storage.
- Add a troubleshooting callout to tutorial step 2; realistic DuckDB result
  header; how-to index "Reference & tooling" split; contributing-docs carve-out
  for advanced illustrative pages + tighter Next-steps rule.

Strict build green; 67 doc/e2e assertions pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: rustyconover

docs/contributing-docs.md

@@ -28,8 +28,13 @@ Every tutorial, how-to, and concept page **must** contain, in order:
 2. **Prerequisites** — assumed knowledge, prior steps, and required extras (e.g.
    `pip install vgi-python[http]`), with links. Use a list or an admonition.
 3. **At least one complete, runnable example** — no `...` elisions in the primary example. It must
-   pass the documentation-example tests (see below).
-4. **A "Next steps" section** — links to the logical next page(s). No dead ends.
+   pass the documentation-example tests (see below). *Exception:* advanced pages whose feature
+   isn't exercisable from a self-contained snippet (HTTP serving, auth, optimizer pushdown) may
+   lead with an illustrative `test="skip"` sketch — but label it illustrative and point to a
+   runnable worker or the reference for the real thing.
+4. **A "Next steps" section** that advances the reader along the funnel: prefer a sibling **how-to**
+   or a **concept** page, then the **reference** for the full contract. Don't jump straight from a
+   how-to into auto-generated reference. No dead ends.
 
 ## Example rules
 

docs/how-to/catalogs.md

@@ -60,6 +60,6 @@ constraints, generated columns, column comments, filter requirements — is cove
 
 ## Next steps
 
-- **Persist per-group state** → [State storage](../shared-storage.md).
-- **Full catalog options** (tables, views, constraints) → [Catalog Interface](../catalog-interface.md).
+- **Persist per-group state** → [State storage](state-storage.md).
+- **Full catalog options** (tables, views, constraints) → [Catalog Interface reference](../catalog-interface.md).
 - **Exact API** → [API Reference: Catalogs](../api/catalogs.md).

docs/how-to/function-patterns.md

@@ -4,7 +4,7 @@ description: "How to write each of the four VGI function patterns: scalar, table
 
 # Function patterns
 
-**What this is:** a recipe showing each of the four VGI function shapes, with a complete,
+**What this is:** a recipe showing each of the five VGI function patterns, with a complete,
 runnable worker for each. **Who it's for:** developers who've finished the
 [tutorial](../tutorial/index.md) and want to know which pattern fits their problem.
 
@@ -199,8 +199,9 @@ When a function must see the **whole** input before it can produce *any* output
 top-k, or a full reduction — use a buffering function. Unlike table-in-out (which emits per input
 batch), it runs in three phases: `process` (the **sink** — stash each batch, return a `state_id`),
 `combine` (reduce all the partials once), and `finalize` (the **source** — stream the result out).
-Because the phases can run in different worker processes, state lives in `params.storage` (shared
-storage scoped by `execution_id`), not in memory.
+Because the phases can run in different worker processes, state can't live in memory — it goes in
+`params.storage`, a shared store keyed to this call (its `execution_id`). See
+[Persist state across workers](state-storage.md) for the backends behind it.
 
 ```python
 --8<-- "examples/row_count_worker.py"
@@ -219,8 +220,7 @@ SELECT * FROM buffers.row_count((SELECT * FROM big_table));
 
 ## Next steps
 
-- **Persist state across invocations** → [State storage](../shared-storage.md).
-- **Let the optimizer prune work** → [Filter pushdown](../filter-pushdown.md) and
-  [Column statistics](../column-statistics.md).
-- **Understand the call lifecycle** → [Concepts: lifecycle](../lifecycle.md).
+- **Persist state across workers** → [State storage](state-storage.md).
+- **Let the optimizer prune work** → [Integrate with the optimizer](pushdown-and-statistics.md).
+- **Understand the call lifecycle** → [Concepts](../concepts/index.md).
 - **Exact signatures** → [API Reference: Functions](../api/functions.md).

docs/how-to/http-auth.md

@@ -21,7 +21,10 @@ The same worker that runs over a subprocess also runs over HTTP — add `--http`
 vgi-serve my_worker.py --http
 ```
 
-The client connects with `transport="http"` instead of spawning a subprocess:
+DuckDB still attaches the worker over HTTP the usual way (`ATTACH ... (TYPE vgi, LOCATION
+'http://...')`). You only need the **Python `Client`** when you're calling the worker *from Python*
+rather than from SQL — for tests, scripts, or another service. It connects with `transport="http"`
+instead of spawning a subprocess, and exposes the same call methods:
 
 ```python test="skip"
 from vgi.client import Client
@@ -39,8 +42,10 @@ The quickest setup is static bearer tokens via an environment variable — comma
 VGI_BEARER_TOKENS="token1=alice,token2=bob" vgi-serve my_worker.py --http
 ```
 
-Unauthenticated requests get HTTP 401. Authenticated requests carry the principal in the
-`AuthContext` your function can read via the injected `ctx`:
+Unauthenticated requests get HTTP 401. Authenticated requests carry the principal in an
+`AuthContext`. Your function reads it through an optional `ctx` parameter: declare `ctx` in the
+signature and the framework injects a per-call `CallContext` (auth, logging, transport info) — you
+don't pass it from SQL.
 
 ```python test="skip"
 class Secret(ScalarFunction):

docs/how-to/index.md

@@ -20,11 +20,13 @@ real. Each guide assumes you can already write and run a basic worker.
   with bearer/JWT auth.
 - **[Integrate with the optimizer](pushdown-and-statistics.md)** — accept pushed-down filters and
   report column statistics.
-- **Describe your functions** — metadata for introspection: [Function Metadata](../metadata.md)
-- **Use the CLI** — invoke functions and inspect workers from the shell: [CLI](../cli.md)
+Each recipe ends with a **Next steps** section that links onward to a concept page and the full
+reference for that topic.
 
-Each recipe links to a deeper **reference** page (Function API, Catalog Interface, Shared Storage,
-Authentication, Filter Pushdown, Column Statistics) for the full contract.
+## Reference & tooling
+
+- **[Function Metadata](../metadata.md)** — describe your functions for introspection.
+- **[CLI](../cli.md)** — invoke functions and inspect workers from the shell.
 
 ## Next steps
 

docs/how-to/pushdown-and-statistics.md

@@ -16,8 +16,10 @@ pushed-down `WHERE` predicates and reporting column statistics so the planner ca
 ## Filter pushdown
 
 A table function can receive the `WHERE` predicates DuckDB would otherwise apply *after* the scan,
-and apply them at the source. Opt in with `filter_pushdown = True` in the function's `Meta`, then
-read the decoded filters in `process` via `params`:
+and apply them at the source. Opt in with `filter_pushdown = True` in the function's `Meta`. The
+framework deserializes the predicates for you and exposes them on `params.current_pushdown_filters`
+as a `PushdownFilters` tree (or `None` when no filter applies), refreshed before each `process`
+call:
 
 ```python test="skip"
 class Events(TableFunctionGenerator[EventsArgs]):
@@ -26,13 +28,14 @@ class Events(TableFunctionGenerator[EventsArgs]):
 
     @classmethod
     def process(cls, params, state, out):
-        filters = params.pushdown_filters   # decoded predicate tree (or None)
+        filters = params.current_pushdown_filters   # PushdownFilters tree, or None
         # apply `filters` while generating rows, then out.emit(...) / out.finish()
 ```
 
-Filters arrive as a hybrid JSON + Arrow structure; decode and evaluate them with
-`deserialize_filters` (see [API: Filter Pushdown](../api/filters.md)). The wire format and a worked
-example are in the [Filter Pushdown reference](../filter-pushdown.md).
+`PushdownFilters` is already decoded — you don't call `deserialize_filters` yourself (that helper is
+for the raw wire bytes). To have the framework apply the filters to your output automatically,
+set `auto_apply_filters = True` in `Meta`. The node types and a worked example are in the
+[Filter Pushdown reference](../filter-pushdown.md) and [API: Filter Pushdown](../api/filters.md).
 
 ## Column statistics
 
@@ -56,7 +59,8 @@ Table(
 EXPLAIN SELECT * FROM mydb.data.departments WHERE id > 100;   -- Physical Plan: EMPTY_RESULT
 ```
 
-Full details — RPC-based dynamic statistics, TTLs, spatial bounds — are in the
+(The snippets above are illustrative — `schema` and the `departments` table stand in for your own
+catalog.) Full details — RPC-based dynamic statistics, TTLs, spatial bounds — are in the
 [Column Statistics reference](../column-statistics.md).
 
 ## Next steps

docs/how-to/state-storage.md

@@ -12,7 +12,12 @@ function that coordinates partial results across workers.
 
 - You've built an aggregate or multi-worker function (see
   [Function patterns → Aggregate](function-patterns.md#aggregate)).
-- For cloud backends: the relevant extra (`vgi-python[azure]`).
+- For the Azure backend: `pip install vgi-python[azure]`. SQLite and Cloudflare DO need no extra.
+
+!!! note "`vgi-serve`"
+    The commands below use `vgi-serve`, the CLI installed with `vgi-python` that runs a worker
+    module as a long-lived process (the production counterpart to the tutorial's `uv run`). The
+    `--http` flag serves it over HTTP instead of stdin/stdout.
 
 ## Two kinds of "state" — don't confuse them
 
@@ -51,7 +56,7 @@ VGI_WORKER_SHARED_STORAGE=cloudflare-do vgi-serve my_worker.py --http
 |---|---|---|---|
 | SQLite | `sqlite` (default) | local / subprocess | none (stdlib) |
 | Azure SQL | `azure-sql` | Azure deployments | `vgi-python[azure]` |
-| Cloudflare DO | `cloudflare-do` | edge / multi-cloud | none (stdlib) |
+| Cloudflare DO | `cloudflare-do` | edge / multi-cloud | none extra — uses `httpx`, which ships with `vgi-python`; needs a Worker endpoint + token |
 
 The per-backend setup (connection strings, credentials, table provisioning) is documented in the
 [Shared Storage reference](../shared-storage.md).

docs/index.md

@@ -73,8 +73,12 @@ Stock `duckdb` works too — `INSTALL vgi FROM community; LOAD vgi;`.
 | C/C++ compilation required | Any language with an Apache Arrow library |
 | Tied to a DuckDB version | Version independent |
 | Complex build/release cycle | Ship a script or executable |
-| Runs in-process | Process isolation |
-| Single-threaded | Parallel workers |
+| Runs in DuckDB's process | Isolated worker process — a crash or a heavy dependency can't take DuckDB down |
+| Native code in DuckDB's threads | Your code, optionally fanned out across worker processes |
+
+**The tradeoff:** data crosses a process boundary as Apache Arrow IPC. That's fast and columnar,
+but not free — co-locate workers (subprocess transport) for latency-sensitive paths, and reach for
+VGI when the productivity and isolation win outweighs the hop.
 
 **Use cases:** call REST APIs from SQL, run ML inference, process data with pandas/numpy, build
 custom ETL transforms, expose external data sources as queryable tables and views.
@@ -87,6 +91,7 @@ custom ETL transforms, expose external data sources as queryable tables and view
 | **Table** | `TableFunctionGenerator` | `SELECT * FROM func(args)` | Generate data |
 | **Table-in-out** | `TableInOutFunction` | `SELECT * FROM func((SELECT ...))` | Streaming transforms, filtering |
 | **Aggregate** | `AggregateFunction` | `SELECT func(col) ... GROUP BY` | Grouped accumulation |
+| **Buffering** | `TableBufferingFunction` | `SELECT * FROM func((SELECT ...))` | Sees every row first (sort, top-k) |
 
 See the [API Reference](api/index.md) for the full surface, or jump into the guides below.
 

docs/tutorial/scalar.md

@@ -66,9 +66,12 @@ What's going on:
 
 ```sql
 SELECT calc.double(21);
--- ┌──────────────┐
--- │ 42           │
--- └──────────────┘
+-- ┌──────────────────┐
+-- │ double(21)       │
+-- │     int64        │
+-- ├──────────────────┤
+-- │       42         │
+-- └──────────────────┘
 ```
 
 Over a real column:

docs/tutorial/table.md

@@ -32,8 +32,8 @@ The new pieces, compared to step 1:
 ??? info "Scalar vs. table — when do I use which?"
     Use a **scalar** function when output has exactly one row per input row (a transform). Use a
     **table** function when you generate rows independent of any input — a sequence, a data source,
-    an API result set. There are two more shapes (table-in-out and aggregate) covered in the
-    [how-to guides](../how-to/function-patterns.md).
+    an API result set. There are three more patterns (table-in-out, aggregate, and buffering)
+    covered in the [how-to guides](../how-to/function-patterns.md).
 
 ??? info "Generating a lot of rows? Stream with state"
     `process` is actually called *repeatedly* until you call `out.finish()`. For large results you
@@ -64,10 +64,19 @@ SELECT * FROM calc.series(3);
 
 That's both function patterns running from SQL. 🎉
 
+??? success "It didn't work?"
+    - **`Binder Error: table function ... does not exist`** — the SQL name is the snake_case of the
+      class name (`Series` → `series`) and a table function is called in `FROM`, not `SELECT`:
+      `SELECT * FROM calc.series(3)`, not `SELECT calc.series(3)`.
+    - **The query hangs and never returns** — a table generator must call `out.finish()` when it has
+      no more rows. Without it the framework keeps calling `process` forever.
+    - **`ATTACH` errors after editing** — if `calc` is already attached from step 1, `DETACH calc;`
+      first (or attach under a new name).
+
 ## Next steps
 
-- **More function shapes** → [How-to: function patterns](../how-to/function-patterns.md) covers
-  table-in-out (streaming transforms), aggregates, and a string-valued scalar.
+- **More function patterns** → [How-to: function patterns](../how-to/function-patterns.md) covers
+  table-in-out (streaming transforms), aggregates, buffering, and a string-valued scalar.
 - **Understand what just happened** → [Concepts: worker lifecycle](../concepts/index.md) explains
   bind → init → process → finish and the transports.
 - **Look up the exact API** → the [API Reference](../api/index.md) documents every class and