Add analytics tracking and thumbs feedback to AI sidekick (V2 editor)#2441
Open
morgan-wowk wants to merge 1 commit into
Open
Add analytics tracking and thumbs feedback to AI sidekick (V2 editor)#2441morgan-wowk wants to merge 1 commit into
morgan-wowk wants to merge 1 commit into
Conversation
🎩 PreviewA preview build has been created at: |
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
morgan-wowk
force-pushed
the
ai-sidekick-analytics
branch
from
edc0ef3 to
4d1dbb5
Compare
Tracks three events on the editor AI assistant: - ai_assistant.message.submitted — fires on send with prompt + thread_message_count - ai_assistant.response.received — fires after response completes with prompt, response, and thread_message_count - ai_assistant.response.feedback — fires when user rates a response via thumbs up/down with rating, prompt, and response Adds a ResponseFeedback component below each assistant message: outline thumbs icons that highlight to `text-info` on hover, then on click record the feedback event and fade (200ms out → 300ms in) to "Thank you for sharing your feedback".
morgan-wowk
force-pushed
the
ai-sidekick-analytics
branch
from
4d1dbb5 to
be6217a
Compare
maxy-shpfy
reviewed
Jun 19, 2026
| if (lastMessage?.role === "assistant") { | ||
| track("ai_assistant.response.received", { | ||
| prompt, | ||
| response: lastMessage.content, |
Collaborator
There was a problem hiding this comment.
Response can be pretty beefy... not sure if we need to track it
maxy-shpfy
reviewed
Jun 19, 2026
Comment on lines
+27
to
+61
| <div className="relative flex items-center h-5 mt-1.5"> | ||
| <div | ||
| className={cn( | ||
| "flex gap-0.5 transition-opacity duration-200", | ||
| state === "submitted" ? "opacity-0" : "opacity-100", | ||
| )} | ||
| aria-hidden={state === "submitted"} | ||
| > | ||
| <button | ||
| onClick={() => handleFeedback("positive")} | ||
| disabled={state !== "idle"} | ||
| className="text-muted-foreground/30 hover:text-info transition-colors duration-150 p-0.5 rounded cursor-pointer" | ||
| aria-label="Helpful response" | ||
| > | ||
| <Icon name="ThumbsUp" size="xs" /> | ||
| </button> | ||
| <button | ||
| onClick={() => handleFeedback("negative")} | ||
| disabled={state !== "idle"} | ||
| className="text-muted-foreground/30 hover:text-info transition-colors duration-150 p-0.5 rounded cursor-pointer" | ||
| aria-label="Not helpful response" | ||
| > | ||
| <Icon name="ThumbsDown" size="xs" /> | ||
| </button> | ||
| </div> | ||
| <span | ||
| className={cn( | ||
| "absolute left-0 text-xs text-muted-foreground transition-opacity duration-300", | ||
| showThanks ? "opacity-100" : "opacity-0 pointer-events-none", | ||
| )} | ||
| > | ||
| Thank you for sharing your feedback | ||
| </span> | ||
| </div> | ||
| ); |
Collaborator
There was a problem hiding this comment.
We must use UI primitives
maxy-shpfy
reviewed
Jun 19, 2026
Comment on lines
+38
to
+41
| const precedingUserMessage = | ||
| msg.role === "assistant" && messages[i - 1]?.role === "user" | ||
| ? messages[i - 1].content | ||
| : undefined; |
Collaborator
There was a problem hiding this comment.
If AI posted multiple messages (it may happen in theory), we're going to loose user prompt. I'm proposing another schema. That can simplify and solidify entire logic:
1. We can extend ChatMessage to carry prompt for assistant type message:
interface BaseChatMessage {
id: string;
content: string;
componentReferences?: Record<string, ComponentRefData>;
}
export interface UserChatMessage extends BaseChatMessage {
role: "user";
}
export interface AssistantChatMessage extends BaseChatMessage {
role: "assistant";
/** Here: */
prompt: string;
}
export type ChatMessage = UserChatMessage | AssistantChatMessage;
2. Next we need to update agentThread.ts so every answer tracks initial prompt:
this.messages = [
...this.messages,
{
id: generateMessageId(),
role: "assistant",
content: response.answer,
prompt,
},
];
3. Now we automatically have access to all required information in ChatMessage component.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds analytics tracking and thumbs up/down feedback UI to the AI chat assistant.
When a user sends a message, an
ai_assistant.message.submittedevent is tracked with the prompt and current thread message count. Once the assistant responds, anai_assistant.response.receivedevent is tracked with the prompt, response content, and updated message count.A new
ResponseFeedbackcomponent is rendered below each assistant message, showing thumbs up and thumbs down buttons. Clicking either fires anai_assistant.response.feedbackevent with the rating, prompt, and response, then fades out the buttons and displays a "Thank you for sharing your feedback" confirmation. The prompt for each assistant message is resolved by looking at the preceding user message in the thread.Analytics events added
Related Issue and Pull requests
Type of Change
Checklist
Screenshots (if applicable)
Test Instructions
ai_assistant.message.submittedandai_assistant.response.receivedevents are fired in your analytics provider after the assistant replies.ai_assistant.response.feedbackevent is tracked with the correct rating, prompt, and response.Additional Comments