feat(llma): add tools tab for LLM Analytics#48423
Open
andrewm4894 wants to merge 13 commits intomasterfrom
Open
feat(llma): add tools tab for LLM Analytics#48423andrewm4894 wants to merge 13 commits intomasterfrom
andrewm4894 wants to merge 13 commits intomasterfrom
Conversation
Extract tool call names from $ai_output_choices at ingestion time (OpenAI tool_calls and Anthropic tool_use formats) into new $ai_tools_called and $ai_tool_call_count properties. Add a new Tools tab (behind LLM_ANALYTICS_TOOLS_TAB feature flag) that aggregates tool usage with traces, users, sessions, and time metrics.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
IContains on the JSON array string $ai_tools_called will match tool names as substrings, causing false positives. If a user clicks "search" from the tools table, it will match any tool containing "search" (e.g., "web_search", "search_docs", "vector_search"). Consider using JSONHas or exact matching after parsing the array in HogQL.
Prompt To Fix With AI
This is a comment left during a code review.
Path: products/llm_analytics/frontend/LLMAnalyticsTools.tsx
Line: 62
Comment:
`IContains` on the JSON array string `$ai_tools_called` will match tool names as substrings, causing false positives. If a user clicks "search" from the tools table, it will match any tool containing "search" (e.g., "web_search", "search_docs", "vector_search"). Consider using `JSONHas` or exact matching after parsing the array in HogQL.
How can I resolve this? If you propose a fix, please make it concise.
Member
Author
There was a problem hiding this comment.
Fixed in c1aadc9 — switched from IContains to Regex with (^|,)toolname(,|$) so it matches exact comma-delimited tokens. Since we use comma-separated storage, this gives precise matching without substring false positives.
Contributor
|
Size Change: +5.55 kB (+0.01%) Total Size: 100 MB
ℹ️ View Unchanged
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a3004c7aee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Add support for normalized format (content blocks with type=function) - Add Python repr fallback for OpenAI Agents SDK format (ResponseFunctionToolCall(name='...')) - Fix nullable ARRAY JOIN by wrapping property in ifNull(..., '[]') - Use parseJSON instead of JSON.parse per eslint rule
Contributor
|
🎭 Playwright report · View test results →
These issues are not necessarily caused by your changes. |
- OpenAI Responses API: flat items with type=function_call and name - Unwrapped choices: tool_calls directly on choice without message wrapper
HogQL's property access via JSONExtractRaw + regex stripping leaves
escaped quotes in JSON array values, breaking JSONExtractArrayRaw.
Switch to comma-separated format (get_weather,search_docs) and use
splitByChar(',', ...) in the query instead.
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
…, and caps Add fast keyword pre-check and 500KB size limit to skip unnecessary JSON parsing for text-only events. Sanitize tool names (trim, replace commas, truncate to 200 chars). Cap at 100 tools per event. Use else-if between message-wrapped and top-level paths. Track new metric outcomes (skipped_no_indicators, skipped_too_large).
Remove errors column (always 0 since tool calls imply successful generation). Add solo_pct column showing percentage of calls where the tool was the only one called, placed after days_seen.
Some SDKs store the full API response in $ai_output_choices as
{"choices":[...]} instead of just the choices array. Real prod data
shows 33% of tool-calling events use this format. Unwrap the choices
key before processing.
- Use regex (^|,)toolname(,|$) instead of IContains for exact tool matching in drilldown filter, avoiding false positives - Set $ai_tool_call_count from user-provided $ai_tools_called so events aren't excluded by the tools query filter
| */ | ||
|
|
||
| SELECT | ||
| arrayJoin(splitByChar(',', assumeNotNull(properties.$ai_tools_called))) as tool, |
Contributor
There was a problem hiding this comment.
Potential null/empty string handling issue. The query uses assumeNotNull(properties.$ai_tools_called) but if $ai_tools_called is an empty string, splitByChar(',', '') returns [''], causing arrayJoin to create a row with an empty tool name. While the WHERE clause filters for $ai_tool_call_count > 0, there's no guarantee these two properties are always in sync (e.g., if manually set by users or due to edge cases in extraction logic).
Fix: Add validation to filter out empty tool names:
SELECT
arrayJoin(splitByChar(',', assumeNotNull(properties.$ai_tools_called))) as tool,
...
WHERE event = '$ai_generation'
AND properties.$ai_tool_call_count > 0
AND properties.$ai_tools_called != ''
AND tool != ''
AND {filters}Spotted by Graphite Agent
Is this helpful? React 👍 or 👎 to let us know.
The posthog-js SDK uses type "tool-call" (not "function") for Vercel AI SDK and OTel integrations. Our extraction logic only matched "function" and "tool_use", silently dropping these tool calls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
$ai_output_choicesat ingestion time into$ai_tools_called(comma-separated) and$ai_tool_call_count(integer) properties{"choices":[...]})LLM_ANALYTICS_TOOLS_TABfeature flag) showing per-tool aggregation with solo % indicating how often a tool is called alone vs in a batchChanges
Ingestion (Node.js)
nodejs/src/ingestion/ai/tools/extract-tool-calls.ts— Core extraction for 7 formats, sanitization (trim, comma→underscore, 200-char truncate), 100-tool capnodejs/src/ingestion/ai/tools/index.ts— Pipeline step with fast keyword pre-check (tool_call,tool_use,function_call,"function"), 500KB size limit, new metric labels (skipped_no_indicators,skipped_too_large)nodejs/src/ingestion/ai/tools/extract-tool-calls.test.ts— 66 parameterized testsnodejs/src/ingestion/ai/process-ai-event.ts— Wired as pipeline step 5nodejs/src/ingestion/ai/metrics.ts—aiToolCallExtractionCounterPrometheus counterFrontend
products/llm_analytics/frontend/tabs/llmAnalyticsToolsLogic.ts— Kea logic (mirrors errors tab pattern)products/llm_analytics/frontend/LLMAnalyticsTools.tsx— DataTable with tool name links to filtered generations, solo % rendererproducts/llm_analytics/frontend/LLMAnalyticsScene.tsx— Tab entry behind feature flagproducts/llm_analytics/frontend/llmAnalyticsSharedLogic.ts— Addedtoolstab ID and URL routingproducts/llm_analytics/manifest.tsx— Route and URLfrontend/src/lib/constants.tsx—LLM_ANALYTICS_TOOLS_TABfeature flagBackend
products/llm_analytics/backend/queries/tools.sql— HogQL query usingarrayJoin(splitByChar(',', ...))with columns: tool, total_calls, solo_pct, traces, users, sessions, days_seen, first_seen, last_seenposthog/taxonomy/taxonomy.py— Property definitions for$ai_tools_calledand$ai_tool_call_countStorage format
Comma-separated string (not JSON array). HogQL's
JSONExtractRaw+ regex stripping leaves escaped quotes that breakJSONExtractArrayRaw. Comma-separated works cleanly withsplitByChar.Test plan
LLM_ANALYTICS_TOOLS_TABfeature flag and verify the tab renders at/llm-analytics/tools$ai_generationevents with tool calls and verify extraction works for all formats