Data tables perform poorly with small batch sizes

[mootari]

Describe the bug

When querying a large result set with small batch sizes (2048 for DuckDB), the table output can take a long time to settle even after all batches are available.

I ran into this with Motherduck where the full result got streamed within 2 seconds after the first batch but the table took an entire minute to catch up (timed separately).

To Reproduce

Example: https://observablehq.com/d/40ae382ca9c82fcb

Steps to reproduce the behavior:

1. Create a notebook with db = DuckDBClient.of()
2. Create an SQL cell with db as source and the query select range, range::text str from range(0, 6000000)
3. Observe that it takes several seconds until all rows are available

Expected behavior

The table should render near instantaneously.

Screenshots

Table batches:

[mbostock]

I tested this in our prerelease web editor for Notebooks 2.0 and the results were returned in less than two seconds. I didn’t investigate whether the slowness is coming from the old SQL cell display (vs. Inputs.table) or from the old DuckDBClient implementation (vs. Notebook Kit’s).

In any case, some performance improvements should be available soon, but I don’t think a fully-functioning query results display in the upcoming launch.