Skip to content

feat(analytics): add dashboard model filter#861

Open
rodboev wants to merge 26 commits into
kenn-io:mainfrom
rodboev:pr/633-analytics-model-filter
Open

feat(analytics): add dashboard model filter#861
rodboev wants to merge 26 commits into
kenn-io:mainfrom
rodboev:pr/633-analytics-model-filter

Conversation

@rodboev

@rodboev rodboev commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

The analytics dashboard can already narrow by project, machine, agent, and time, but it still rolls every chart and summary together across all models. Issue #633 asks for model-name filtering on that main dashboard, and the missing piece is end-to-end plumbing: the analytics HTTP input does not accept model, the shared analytics filter type has no model field, the backend predicate builders do not carry model-aware message scoping through analytics and trends, and the frontend analytics store and toolbar have no dashboard-local model filter state.

This change threads a comma-separated model filter through the analytics route layer, the shared analytics filter object, the analytics, trends, and filtered-model lookup helpers, and the dashboard request state. When a model filter is active, the dashboard first scopes to sessions that contain at least one matching model message, then row-based analytics derive message, token, tool-call, trend-term, session-shape, and velocity metrics from matching model rows inside that session set. Summary totals, activity, heatmap, projects, tools, skills, top sessions, signals, trend terms, session shape, and velocity now stay aligned across SQLite, PostgreSQL, and DuckDB-backed modes, including mixed-model sessions and hour/day slices.

On the frontend, the analytics store gains model filter state, request params, clear and toggle helpers, and active-filter chips, and the analytics toolbar gains a model dropdown that keeps known models stable across refreshes and filtered reloads. Backend analytics tests now cover the new filter behavior across SQLite, PostgreSQL pgtests, and DuckDB, including mixed-model summaries, activity, hour-of-week, trends, tools, skills, top sessions, session shape, and velocity, and the analytics store tests verify that the dashboard sends model and clears it correctly.

Fixes #633

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (41a49ab)

Analytics model filtering has two medium-severity correctness/scalability issues; no high or critical findings.

Medium

  • internal/db/analytics.go:267 - The model filter is implemented as a session-level EXISTS, but message-aggregating analytics queries still aggregate all message rows from matching sessions. In a mixed-model session matching gpt-4o, claude-* messages can still contribute to hour-of-week, trends, activity, and message counts. Apply the model predicate to message-row aggregations, or explicitly separate session-scoped vs message-scoped model filtering and add mixed-model coverage for SQLite and PostgreSQL.

  • internal/db/analytics.go:406 - getAnalyticsModelsForSessionIDs builds a single IN (...) list for every filtered session. On the Go summary path, common for DST timezones, archives exceeding SQLite’s bind-variable limit can make the summary endpoint fail while collecting models. The PostgreSQL mirror has the same unchunked pattern at internal/postgres/analytics.go:452. Chunk session IDs with the existing helpers and merge/dedupe/sort returned model names.


Panel: ci_default_security | Synthesis: codex, 12s | Members: codex_default (codex/default, done, 4m24s), codex_security (codex/security, done, 13s) | Total: 4m49s

@rodboev

rodboev commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Applied the chunking fix in d5134519 for both SQLite and PostgreSQL model-list lookups, so the summary path no longer builds one oversized IN (...) clause.

I left the mixed-model behavior session-scoped on purpose. The issue asks for filtering the main dashboard by model, and specifically calls out seeing model usage in sessions. This PR keeps the dashboard scoped to sessions that contain at least one matching model, which is the contract already described in the PR body. That keeps session-level cards, projects, tools, skills, and top sessions on one consistent population instead of partially rewriting some of them into message-level metrics.

A message-scoped model-usage dashboard still seems useful, I just see that as a separate slice from this session dashboard filter.

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (d513451)

Summary verdict: one medium issue remains; no high or critical findings reported.

Medium

  • internal/db/trends.go:71; internal/postgres/trends.go:35
    The trends model filter is applied through the shared session predicate, which only checks that the session has at least one matching model. The outer query still counts every eligible m row in that session, so mixed-model sessions leak non-selected model messages into message_count and term totals.
    Fix: For message-row aggregations, clear sessionFilter.Model before building the session WHERE, then apply the CSV model predicate directly to outer m.model in both SQLite and PostgreSQL paths. Add mixed-model tests.

Panel: ci_default_security | Synthesis: codex, 6s | Members: codex_default (codex/default, done, 5m8s), codex_security (codex/security, done, 15s) | Total: 5m29s

@rodboev

rodboev commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

The trends paths now clear sessionFilter.Model before building the shared session predicate, then apply the selected model directly to outer m.model in both SQLite and PostgreSQL. That keeps mixed-model sessions from leaking non-selected messages into trend message_count and term totals.

I also added mixed-model coverage for both backends, plus a PostgreSQL query-shape test that keeps the model predicate on the outer message rows.

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (4e09c6c)

Medium issue found; security review found no additional issues.

Medium

  • frontend/src/lib/api/generated/services/AnalyticsService.ts:628
    getApiV1AnalyticsSignalSessions does not accept or include the new model query parameter, so signal evidence requests drop the active model filter even though analytics.signalEvidenceParams() includes it. This can show unfiltered signal examples for model-filtered signal totals.
    Fix: Add model to this method’s destructuring, parameter type, and query object, and cover the signal evidence request path in a frontend test.

Panel: ci_default_security | Synthesis: codex, 6s | Members: codex_default (codex/default, done, 7m21s), codex_security (codex/security, done, 52s) | Total: 8m19s

@rodboev

rodboev commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

AnalyticsService.getApiV1AnalyticsSignalSessions() now accepts model and includes it in the query object, so signal evidence requests stay aligned with the active analytics model filter.

I also added a focused frontend test for that request path, and npm test -- --run AnalyticsService.test.ts passed locally.

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (be14b6f)

Summary verdict: Medium-risk correctness issues remain in model-filtered analytics for mixed-model sessions.

Medium

  • Location: internal/db/analytics.go:1085, internal/db/analytics.go:1637
    Problem: Model-filtered message aggregations still count every message in sessions that contain the selected model. In mixed-model sessions, Activity and Hour-of-Week can include off-model messages because the model predicate is only a session-level EXISTS. PostgreSQL counterparts reportedly use the same query shape.
    Fix: Apply the model CSV predicate to the joined messages m rows for message-level aggregations in both SQLite and PostgreSQL, and add mixed-model session test coverage.

  • Location: internal/db/analytics.go:548, internal/db/analytics.go:584
    Problem: Model and hour/day filters are evaluated against separate message existence checks. A session with gpt-4o at 09:00 and another model at 10:00 can match model=gpt-4o&hour=10, so time-filtered panels can include sessions where the selected model was not active in the selected time bucket. PostgreSQL filteredSessionIDs has the same issue.
    Fix: When time filters and model filters are both active, apply the model predicate inside the same message timestamp check used for the time filter in both backends.


Panel: ci_default_security | Synthesis: codex, 10s | Members: codex_default (codex/default, done, 6m41s), codex_security (codex/security, done, 1m7s) | Total: 7m58s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (f6342d2)

Medium issue found: tool-call analytics can overcount across mixed-model sessions.

Medium

  • internal/db/analytics.go:1249, internal/db/analytics.go:2210, internal/postgres/analytics.go:823, internal/postgres/analytics.go:1627
    • The new model filter is applied when selecting sessions, but tool-call aggregations still count every tool_calls row in those sessions.
    • In mixed-model sessions, ActivityEntry.ToolCalls, the Tools panel, and Skills panel can include tool calls from unselected models even though message counts are filtered by m.model.
    • Fix by joining tool_calls back to messages in the activity/tools/skills tool-call queries and applying the same model CSV predicate in both SQLite and PostgreSQL. Add mixed-model tests with tool calls on selected and unselected model messages.

Panel: ci_default_security | Synthesis: codex, 7s | Members: codex_default (codex/default, done, 5m22s), codex_security (codex/security, done, 14s) | Total: 5m43s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (25f8649)

Summary verdict: One medium backend parity issue remains; no high or critical findings.

Medium

  • internal/duckdb/analytics_usage.go:127: AnalyticsFilter.Model is wired into the API and SQLite/PostgreSQL paths, but DuckDB analytics still ignores it. duckBuildAnalyticsWhere has no model predicate, and DuckDB activity/trends/tool/skill queries do not constrain joined messages by m.model, so agentsview duckdb serve returns unfiltered analytics for ?model=... and the summary does not populate model options.
    • Fix: Add DuckDB parity for the model filter across analytics/trends queries, including message-level predicates where SQLite/PostgreSQL now apply them, and add DuckDB tests for model-filtered analytics.

Panel: ci_default_security | Synthesis: codex, 7s | Members: codex_default (codex/default, done, 6m31s), codex_security (codex/security, done, 1m2s) | Total: 7m40s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (42f2026)

Summary verdict: One medium correctness issue remains; no security findings were reported.

Medium

  • Location: internal/db/analytics.go:837
    • Problem: The model filter only narrows sessions, but the summary still sums session-level message_count. Mixed-model sessions selected by model=gpt-4o can include messages from other models, while activity, tools, and trends count only matching message rows. The same session-level pattern remains in heatmap/projects and in PostgreSQL/DuckDB summaries.
    • Fix: For model-filtered message metrics, aggregate from messages with the same m.model predicate, or keep model filtering consistently session-scoped across all analytics endpoints and tests.

Panel: ci_default_security | Synthesis: codex, 7s | Members: codex_default (codex/default, done, 6m5s), codex_security (codex/security, done, 48s) | Total: 7m0s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (7e798f2)

Medium-risk issue remains: AnalyticsSummary.Models can include models outside the active message-level filter scope.

Medium

  • internal/db/analytics.go:1075, internal/postgres/analytics.go:679, internal/duckdb/analytics_usage.go:606
    • AnalyticsSummary.Models is populated from every model in the filtered sessions, not from the same message-level model/time scope used for the summary counts. A mixed-model session filtered with model=gpt-4o can still return unrelated models like claude-3-5-sonnet in models, making the response and dropdown options inconsistent with the active filter.
    • Suggested fix: pass the active AnalyticsFilter into the model-list helper and apply the same model/hour/day message predicates, or derive the list from the same filtered message rows used for counts. Add mixed-model regression coverage for SQLite, PostgreSQL, and DuckDB.

Panel: ci_default_security | Synthesis: codex, 7s | Members: codex_default (codex/default, done, 6m8s), codex_security (codex/security, done, 25s) | Total: 6m40s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (8378a7a)

Medium issues remain in analytics model filtering; no High or Critical findings were reported.

Medium

  • Location: internal/db/analytics.go:1150
    Problem: summary.models only uses the message-level filtered lookup when Model is set. With only hour/dow active, mixed-model sessions can contribute models from messages outside the selected time window, so the model dropdown can show models that are not actually present in the filtered message scope. The same pattern appears in the PostgreSQL and DuckDB summary paths.
    Fix: Use the filtered model lookup whenever f.HasTimeFilter() or a real model filter is active, and make the SQLite fast path constrain the joined message rows by the same time predicates.

  • Location: internal/db/analytics.go:4835
    Problem: Top sessions now accepts the model filter, but the messages metric still orders and returns the full session message_count. A session with one selected-model message and many other-model messages can rank above a session with many selected-model messages, making the model-filtered dashboard misleading. PostgreSQL and DuckDB have the same session-total behavior.
    Fix: When metric == "messages" and a model filter is active, rank and return per-session counts of matching model messages across all backends.


Panel: ci_default_security | Synthesis: codex, 9s | Members: codex_default (codex/default, done, 6m12s), codex_security (codex/security, done, 1m19s) | Total: 7m40s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (0a096a9)

Summary verdict: The PR has medium-severity analytics correctness issues in model-filtered aggregations; no security issues were found.

Medium

  • Location: internal/db/analytics.go:1375
    Problem: Model + hour/day filters only use timeIDs to admit sessions; the activity query then counts every message for that model in the admitted session. A session with gpt-4o messages at 09:00 and 10:00 will report both messages for model=gpt-4o&hour=10. The same pattern affects tool-call counts and the PostgreSQL/DuckDB paths.
    Fix: Apply the hour/day timestamp predicate to the message and tool-call aggregation queries when a time filter is active, or reuse a shared helper that filters rows by both model and message time.

  • Location: internal/db/analytics.go:1651
    Problem: Model-filtered requests recompute message counts, but output-token metrics still use full session totals. Mixed-model sessions will leak off-model tokens into output_tokens heatmaps, top-session output-token ranking, and summary token totals; PostgreSQL and DuckDB keep the same session-total behavior.
    Fix: Aggregate messages.output_tokens and token coverage from rows matching the selected model and active time filter, and use those model-scoped totals across SQLite, PostgreSQL, and DuckDB.


Panel: ci_default_security | Synthesis: codex, 12s | Members: codex_default (codex/default, done, 12m26s), codex_security (codex/security, done, 44s) | Total: 13m22s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (f049114)

Medium confidence: one substantive issue remains in model-filtered velocity analytics; no security issues were reported.

Medium

  • internal/db/analytics.go:3494 - The model filter reaches the velocity endpoint, but velocity still loads all messages and tool calls for any session containing the selected model. In mixed-model sessions, off-model timings, message rates, character rates, and tool rates can leak into the filtered velocity panel. The same issue exists in the PostgreSQL and DuckDB velocity paths.

    Fix: Make velocity model-aware across SQLite/PostgreSQL/DuckDB, or stop sending model to the velocity endpoint if velocity is intentionally session-scoped.


Panel: ci_default_security | Synthesis: codex, 7s | Members: codex_default (codex/default, done, 10m54s), codex_security (codex/security, done, 1m36s) | Total: 12m37s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (3a5aa0d)

Medium findings:

  • internal/postgres/analytics.go:618; internal/duckdb/analytics_usage.go:976
    Model-filtered tool-call counting requires a parseable message timestamp even when no hour/day filter is active. GetAnalyticsVelocity uses this helper for any model filter, so PostgreSQL and DuckDB undercount selected-model tool calls for sessions with NULL/missing tool-call message timestamps, while SQLite counts them unless a time filter is set.
    Fix: only parse/apply the timestamp check when f.HasTimeFilter() is true; otherwise count every tool call whose joined message matches the selected model.

Panel: ci_default_security | Synthesis: codex, 6s | Members: codex_default (codex/default, done, 9m15s), codex_security (codex/security, done, 2m37s) | Total: 11m58s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (228c6d8)

Summary verdict: One medium issue remains; no high or critical findings were reported.

Medium

  • internal/db/analytics.go:2383; internal/postgres/analytics.go:1930; internal/duckdb/analytics_usage.go:1592
    The new model filter reaches the session-shape endpoint, but length distribution still buckets by full-session message_count, and autonomy still queries all messages in the session. Mixed-model sessions can therefore show off-model message volume and tool/user ratios for the selected model. The same unfiltered count is also used for velocity complexity buckets.

    Fix: When f.Model is set, derive session-shape length/autonomy and velocity complexity from the same filtered message/tool stats used by the other analytics views, and add mixed-model tests across SQLite, PostgreSQL, and DuckDB.


Panel: ci_default_security | Synthesis: codex, 8s | Members: codex_default (codex/default, done, 9m27s), codex_security (codex/security, done, 3m10s) | Total: 12m45s

@rodboev

rodboev commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Fixed on 8a23705.

Session-shape now derives length buckets and autonomy from the same filtered per-session message stats that the model-scoped analytics views already use in internal/db/analytics.go, internal/postgres/analytics.go, and internal/duckdb/analytics_usage.go, so mixed-model sessions stop leaking off-model message volume and off-model user turns into the selected model's shape view.

Velocity complexity now uses those filtered message counts as well, which keeps model-filtered sessions in the correct 1-15 or 16-60 bucket instead of classifying them by the full session totals.

I also added mixed-model regressions for SQLite and DuckDB, plus matching PostgreSQL pgtests for the same shape and complexity cases.

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (8a23705)

Medium issue found: model-filtered analytics can still return off-model signal evidence.

Medium

  • internal/db/analytics.go:4026: GetAnalyticsSignalSessions receives the model filter, but signalMessages is called without it and loads every message for each candidate session. Mixed-model sessions can return off-model evidence excerpts even when filtered to one model. The same issue exists in PostgreSQL and DuckDB stores.

    Fix: Pass the model filter into the signal-message loaders and apply the same CSV model predicate there, with backend coverage for mixed-model sessions.


Panel: ci_default_security | Synthesis: codex, 6s | Members: codex_default (codex/default, done, 7m32s), codex_security (codex/security, done, 3m1s) | Total: 10m39s

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (f571938)

Summary verdict: one medium-severity analytics correctness issue should be addressed before merge.

Medium

  • internal/db/analytics.go:304
    Model-filtered message stats only keep rows whose own messages.model matches. Several parsers store the model only on assistant messages and leave user turns empty, so filtered analytics can drop all user messages and skew user counts, autonomy, velocity, and trends for normal sessions.
    Fix: Attribute user turns to the selected-model exchange, or define model filtering around assistant/model turns and add parser-realistic tests where user messages have an empty model.

Panel: ci_default_security | Synthesis: codex, 7s | Members: codex_default (codex/default, done, 10m14s), codex_security (codex/security, done, 2m19s) | Total: 12m40s

@rodboev

rodboev commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

I traced the last roborev report back to the model-scoped message loaders rather than any single panel. The shared filtered-message helpers, the velocity loaders, and the trend scans were all treating messages.model as a row-local gate, which drops parser-realistic user turns when only the assistant row carries the selected model.

This rework keeps the model filter anchored on the selected assistant turn, but it now buffers empty-model user rows and attributes them to that assistant exchange when the reply matches the active model. I’m applying that rule in SQLite, PostgreSQL, and DuckDB, and I switched the existing mixed-model regressions over to empty-model user rows so the tests exercise the real parser shape that roborev flagged.

@roborev-ci

roborev-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (b074d05)

Summary verdict: model-filter analytics mostly look safe, but two Medium correctness issues remain.

Medium

  • Location: internal/db/analytics.go:1504, internal/postgres/analytics.go:724
    Problem: Model-filtered activity always calls filteredSessionIDs, even when no hour/day filter is active. That helper requires a non-empty/non-NULL parseable message timestamp, so sessions whose selected-model messages lack timestamps are dropped from SQLite/PostgreSQL activity, while other model-filtered panels still count them.
    Fix: Only call and apply filteredSessionIDs when f.HasTimeFilter() is true; otherwise rely on the model EXISTS predicate already in where.

  • Location: internal/db/analytics.go:4270, internal/postgres/analytics.go:3361, internal/duckdb/analytics_usage.go:2664
    Problem: Model-filtered signal evidence filters messages strictly by model, which drops parser-style user turns that usually have an empty model. User-prompt signals such as short prompts, duplicate prompts, and frustration markers can then lose their actual evidence and fall back to unfiltered FirstMessage with no ordinal.
    Fix: Reuse the model-scoped message selection logic that preserves pending model-less user turns before selected-model assistant messages, or otherwise include those scoped user evidence rows for signal examples.


Panel: ci_default_security | Synthesis: codex, 10s | Members: codex_default (codex/default, done, 8m35s), codex_security (codex/security, done, 2m48s) | Total: 11m33s

@rodboev

rodboev commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up pushed in

@roborev-ci

roborev-ci Bot commented Jun 26, 2026

Copy link
Copy Markdown

roborev: Combined Review (043dcc7)

Summary verdict: one medium correctness issue remains; no high or critical findings were reported.

Medium

  • internal/db/analytics.go:2931, internal/postgres/analytics.go:2226, internal/duckdb/analytics_usage.go:1878
    Model-filtered tool analytics ignore the active day/hour filter when counting tool calls. The session prefilter can require a selected-model message in the active time window, but the aggregation query then counts every tool call for that model in those sessions, including calls outside the selected hour/day.

    Fix: Apply the same message timestamp filter used by getAnalyticsFilteredToolCallCounts when aggregating tool categories, and add SQLite/PostgreSQL/DuckDB tests for Model + Hour or Model + DayOfWeek tool counts.


Panel: ci_default_security | Synthesis: codex, 6s | Members: codex_default (codex/default, done, 10m12s), codex_security (codex/security, done, 2m44s) | Total: 13m2s

@rodboev

rodboev commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Applied the tool analytics fix. Model-scoped tool counts now resolve each call against its own message timestamp, with session fallback only when the tool row has no message timestamp, so Model + Hour and Model + DayOfWeek stop pulling in off-window tool calls from the same session. I added regression coverage for SQLite, PostgreSQL, and DuckDB, and reran CGO_ENABLED=1 go test -tags "fts5,kit_posthog_disabled" ./internal/db ./internal/duckdb ./internal/postgres -count=1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Model filter on the main dashboard

1 participant