feat: multi-axis structured summaries (Phase 2.5)#22
Merged
Conversation
Phase 2.5 schema groundwork. The summarizer is about to start writing a
structured {topics, entities, numbers, one_line} object per section so
retrieval has more axes to match on; this commit adds the storage shape
and the wire model without changing any existing behaviour.
- New migration 0005 adds sections.summary_axes JSONB. Forward-only
additive: NULL on every existing row, no backfill required. The
down migration drops the column cleanly.
- tree.SummaryAxes carries the four axes; Section + SectionView gain
an optional *SummaryAxes pointer so nil distinguishes 'not yet
generated' from 'generated, no entities found'. Existing
Section.Summary stays put — it'll hold axes.OneLine so older SDKs
that only read 'summary' keep working.
- db.UpsertSection writes the new column; db.UpdateSectionSummaryAxes
mirrors UpdateSectionSummary so the summarize stage can patch the
two fields independently as it processes each section.
- JSONB marshal/unmarshal follows the candidate_questions pattern:
nil pointer marshals to SQL NULL, a non-nil pointer always marshals
to an object, garbled bytes degrade to nil. Roundtrip tests cover
nil / empty / full / unicode shapes.
…ries
The summarizer now produces a structured {topics, entities, numbers,
one_line} object per section via JSON mode, replacing the one-line
prompt. Persistence writes BOTH the new axes blob (via
UpdateSectionSummaryAxes) and the one_line into the existing summary
column, so older callers that read summary continue to work unchanged.
- structuredSummaryFor builds the JSON-mode request, mirroring the
HyDE retry-on-parse-failure shape (defaultSummaryAxesRetries=2).
On final parse failure the section ends up with the model's raw
text as OneLine and empty axis lists — never calls p.fail, so a
single LLM-formatting blip doesn't dead-letter the document.
- parseSummaryAxes tolerates code-fence wrappers and prose leakage
(same contract as ParseSelection / parseHyDEResponse), trims
blank/whitespace-only axis entries, and rejects non-JSON outright
so the retry loop fires.
- Per-axis caps (SummaryAxesMaxTopics/Entities/Numbers) trim
oversized model output so a misbehaving model can't blow up the
retrieval prompt budget downstream. Defaults: 4/8/6.
- legacyOneLineSummary is the pre-2.5 single-sentence path; reached
when SummaryAxesEnabled is false. Keeps the structured-vs-legacy
code paths cleanly separated so the opt-out is one bool away.
- summarize now patches summary + summary_axes independently — the
summary column always gets axes.OneLine; the JSONB column is only
written when SummaryAxesEnabled is on, so a future opt-out cleanly
leaves summary_axes NULL.
Tests cover: happy-path axes parse, axis caps enforcement, parse
failure falls back to raw text in OneLine, transport / ErrNotImplemented
fallback, retry count, code-fence / blank-entry tolerance.
…rompt
The retrieval prompt now renders entities and numbers from each
section's structured summary on the same outline line as the
title/summary, giving the model direct surface-form access to
proper-noun and numeric anchors at selection time. Pre-Phase-2.5
sections (axes == nil) skip the new suffix, so un-backfilled
documents see exactly the same prompt as before.
Outline shape per section:
- [sec_a] Long-Term Debt — issued debt securities — entities: 3M Company, JPMorgan — numbers: $4.2B, 2034
- writeSectionLine appends '— entities: ...' / '— numbers: ...'
when sv.SummaryAxes is non-nil and the corresponding axis is
non-empty. Truncated to the first 3 entries each via firstN to
keep per-section prompt cost bounded.
- Config block ingest.summary_axes (Enabled, MaxTopics, MaxEntities,
MaxNumbers) wires the toggle + caps through to the ingest
pipeline. Defaults: enabled=true, 4/8/6. Env overrides:
VLE_INGEST_SUMMARY_AXES_{ENABLED,MAX_TOPICS,MAX_ENTITIES,MAX_NUMBERS}.
Validate rejects negatives.
- Both cmd/server and cmd/engine wire the new config fields into
NewPipeline so production binaries pick up the structured path
out of the box.
- openapi.yaml gains a shared SummaryAxes schema referenced from
SectionView, Section, and QuerySection. Marked Omitted on
pre-2.5 sections.
- config.example.yaml + config.server.example.yaml gain a
summary_axes block with inline documentation.
Tests cover: axes-bearing tree renders entities + numbers with proper
truncation; pre-2.5 tree (no axes anywhere) renders unchanged; config
defaults + env overrides + negative-cap validation.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
Warning Review limit reached
More reviews will be available in 54 minutes and 8 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (17)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2.5 of the retrieval-quality roadmap. The summarizer used to return a one-line string per section; this PR extends it to a structured
{topics, entities, numbers, one_line}object so retrieval has richer signal at selection time. Both the existingsummarycolumn (which keeps holding the one-line sentence) and a newsummary_axesJSONB column are populated by the ingest pipeline; the retrieval prompt renders entities + numbers on the same outline line as the title/summary so the model sees the section's proper-noun and numeric anchors directly.Schema
Forward-only-additive. Existing rows carry
NULLand the retrieval prompt skips the axes block whennil, so un-backfilled documents see no prompt change. The down migration drops the column cleanly. No backfill in scope for this PR — operators wanting axes on pre-2.5 docs would run a separate one-off tool.Backwards-compat strategy
Section.Summary(the plain text column) continues to be written by the pipeline, now sourced fromaxes.OneLine. Older SDKs / API consumers that only readsummarykeep working unchanged.- [id] Title — summary) whensummary_axesisnil. The new— entities: ... — numbers: ...suffixes only appear on sections that carry a non-empty axes block.summary_axes = NULL; new ingests write the structured blob.Opt-out
ingest.summary_axes.enabled: false(orVLE_INGEST_SUMMARY_AXES_ENABLED=false) reverts the summarize stage to the legacy single-sentence prompt. The retrieval-side rendering is conditional on the axes pointer being non-nil, so disabling the ingest path is enough — no separate retrieval flag needed.Design rationale
topicsfor paraphrased subject-matter queries,entitiesfor proper-noun lookups ("3M's senior unsecured notes"),numbersfor value-anchored questions ("the $4.2B figure on long-term debt"). The retrieval model gets one extra string per axis on the same outline line — single-digit token overhead per section, but covers the three query patterns vector RAG handles poorly.UPDATEs (summary+summary_axes) so the two columns can drift independently as the pipeline evolves — for instance a future backfill tool can fillsummary_axesfor older sections without touchingsummary.Test plan
pkg/db/sections_marshal_test.go— roundtrip nil / empty / full / unicode shapes throughmarshalSummaryAxes/unmarshalSummaryAxes; garbled JSONB degrades to nil; empty object is preserved (not coerced to NULL) so the pipeline can distinguish "generated, all axes empty" from "not generated".pkg/ingest/summary_axes_test.go— happy-path axes parse, per-axis cap enforcement, parse failure falls back to raw text in OneLine, transport / ErrNotImplemented fallback, retry count, code-fence / blank-entry tolerance, legacy single-line path whenSummaryAxesEnabled=false.pkg/retrieval/retrieval_test.go— outline rendering surfacesentities:+numbers:prefixes (truncated to first 3); axes-empty section omits the labels; pre-2.5 tree without axes anywhere renders unchanged.pkg/config/config_test.go— defaults (enabled=true, 4/8/6); env override (enable/disable + caps); garbled env preserves defaults; negative caps failValidate.go build ./...,go vet ./...,go test ./...all green.Migration sequence
0005_sections_summary_axes.{up,down}.sql. Latest pre-PR migration was0004_sections_extrasso this slots in directly.