Skip to content

linq: table group_by fusion — TableAdapter rides plan_group_by_core#3103

Merged
borisbat merged 4 commits into
masterfrom
bbatkin/linq-table-groupby
Jun 11, 2026
Merged

linq: table group_by fusion — TableAdapter rides plan_group_by_core#3103
borisbat merged 4 commits into
masterfrom
bbatkin/linq-table-groupby

Conversation

@borisbat

Copy link
Copy Markdown
Collaborator

What

each_kv(tab) / keys(tab) / values(tab) chains with group_by declined at the can_group_by_source gate (TableAdapter inherited the base false) and cascaded to tier-2: materialize a kv array, re-enter the array lane. That cascade was the entire groupby_* gap in the m7 column.

The group_by splice pattern is fully adapter-generic (can_group_by_sourcebuild_group_by_adapterplan_group_by_core), so enabling tables is two overrides on TableAdapter, mirroring DecsAdapter:

  • can_group_by() → true
  • build_group_by_adapter() → fresh TableAdapter. The upstream_join arm returns null: join |> group_by over a table lead stays on tier-2 (named deferred edge in LINQ_TO_TABLE.md; the stage-5 terminator-path join is untouched).

plan_group_by_core hands the bucket-fill body to wrap_source_loop, so the kv usage-pruner sees the whole accumulation (key expr + reducer updates + upstream where/select segments) — a group key over kv.value.brand walks values(tab) alone. The non-copyable-values gate and the elemType const-scrub are inherited from adapter construction.

Numbers

benchmarks/sql m7 (table) groupby_* family, ns/op (results.md re-swept in this PR):

before after
INTERP (count / family range) 163 / 144–201 31 / 30–50
JIT (count / family range) 43.5 / 44–73 8.4 / 8.4–11

~5× in both tiers — the JIT side compounds with #3100's inline slot walk. join_groupby_* unchanged (cascade, by design); INTERP matrix flat elsewhere.

Tests

8 new sub-tests in tests/linq/test_linq_table_source.das (67/67 INTERP + JIT, AOT-compiled in test_aot): kv/keys/values lanes, upstream where+select segments, having + trailing where + order, count terminator, empty table, fused-vs-tier-2 agreement.

Gates

  • full INTERP 11001 / 0 failed, full AOT 10313 / 0 failed
  • JIT: linq 1971, jit_tests 295 — pass
  • CI lint 0 issues; formatter clean
  • docs: Sphinx latex + html -W clean, pdflatex pass 1 clean (linq_fold_patterns.rst table-source row updated + new table group_by pattern row; LINQ_TO_TABLE.md stage 7 status + findings + deferred edge)

🤖 Generated with Claude Code

each_kv/keys/values |> group_by chains declined at can_group_by_source
(TableAdapter inherited the base false) and cascaded to tier-2: materialize
a kv array, re-enter the array lane. The group_by splice pattern is fully
adapter-generic, so enabling tables is two overrides mirroring DecsAdapter:

- can_group_by() -> true
- build_group_by_adapter() -> fresh TableAdapter (clone tabExpr, fresh
  srcName, same elemType/lane). The upstream_join arm returns null:
  join |> group_by over a table lead stays on tier-2 (named deferred edge
  in LINQ_TO_TABLE.md; the stage-5 terminator-path join is untouched).

plan_group_by_core hands the bucket-fill body to wrap_source_loop, so the
kv usage-pruner sees the whole accumulation (key expr + reducer updates +
upstream where/select segments) — a group key over kv.value.brand walks
values(tab) alone. The non-copyable-values gate and the -const elemType
scrub are inherited from adapter construction.

m7 groupby_* (ns/op): INTERP 144-201 -> 30-50 (count 163->31, ~5x);
JIT 44-73 -> 8.4-11 (count 43.5->8.4, rides #3100's inline slot walk).
join_groupby_* unchanged (cascade). INTERP matrix flat elsewhere.

Tests: 8 new group_by sub-tests in test_linq_table_source.das (67/67
INTERP+JIT; AOT-compiled in test_aot) — kv/keys/values lanes, upstream
segments, having + trailing where + order, count terminator, empty table,
fused-vs-tier-2 agreement.

Gates: full INTERP 11001/0, full AOT 10313/0, JIT linq 1971 + jit_tests
295, CI lint 0, Sphinx latex+html -W clean + pdflatex pass 1, formatter
clean. results.md re-swept; linq_fold_patterns.rst table row + new
group_by pattern row; LINQ_TO_TABLE.md stage 7 status + findings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 11, 2026 21:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Enables group_by fusion for table-backed LINQ sources (each_kv(tab), keys(tab), values(tab)) by letting TableAdapter participate in the existing adapter-generic plan_group_by_core path, avoiding the tier-2 “materialize kv array then regroup” cascade.

Changes:

  • Enable table-source group_by fusion via TableAdapter capability + adapter construction hook.
  • Add new table-source group_by tests covering kv/keys/values lanes and composed segments (having/where/order/count/empty).
  • Update docs and benchmark notes/results to reflect the new fused behavior and performance deltas.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
daslib/linq_fold_table.das Turns on group_by support for table sources and wires build_group_by_adapter to reuse plan_group_by_core with a fresh TableAdapter (declines upstream-join shape to keep the documented deferred edge).
tests/linq/test_linq_table_source.das Adds a new test_table_group_by suite validating fused behavior across lanes and common operator compositions.
doc/source/reference/linq_fold_patterns.rst Documents the new table-lead group_by fusion pattern and its limits (e.g., join→group_by over table lead still cascades).
benchmarks/sql/results.md Updates benchmark narrative and measured numbers to reflect the new fused table groupby_* behavior.
benchmarks/sql/LINQ_TO_TABLE.md Advances table-fold “stage” status and documents the new group_by fusion plus remaining deferred edges.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/linq/test_linq_table_source.das Outdated
borisbat and others added 3 commits June 11, 2026 15:02
… inference cleanups

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…catch)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@borisbat borisbat merged commit e6d896b into master Jun 11, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants