diff --git a/weave/guides/evaluation/scorers.mdx b/weave/guides/evaluation/scorers.mdx index db71b553ce..a729fe9f01 100644 --- a/weave/guides/evaluation/scorers.mdx +++ b/weave/guides/evaluation/scorers.mdx @@ -430,7 +430,7 @@ feedback = list(call.feedback) ![Call Scores Tab](./img/call_scores_tab.png) -Scores for an individual call are displayed in the call details page under the "Scores" tab. +Scores for an individual call are displayed in the Call details panel under the "Scores" tab. ### Analyze multiple Calls' Scores diff --git a/weave/guides/tracking/feedback.mdx b/weave/guides/tracking/feedback.mdx index 1247d4a17f..cec5a217d7 100644 --- a/weave/guides/tracking/feedback.mdx +++ b/weave/guides/tracking/feedback.mdx @@ -3,7 +3,7 @@ title: "Collect feedback and use annotations" description: "Collect and analyze feedback for LLM applications through UI and SDK" --- -Efficiently evaluating LLM applications requires robust tooling to collect and analyze feedback. Weave provides an integrated feedback system, allowing users to provide call feedback directly through the UI or programmatically via the SDK. Various feedback types are supported, including emoji reactions, textual comments, and structured data, enabling teams to: +Efficiently evaluating LLM applications requires robust tooling to collect and analyze feedback. W&B Weave provides an integrated feedback system, allowing users to provide Call feedback directly through the UI or programmatically through the SDK. Various feedback types are supported, including emoji reactions, textual comments, and structured data, enabling teams to: - Build evaluation datasets for performance monitoring. - Identify and resolve LLM content issues effectively. @@ -12,33 +12,33 @@ Efficiently evaluating LLM applications requires robust tooling to collect and a This guide covers how to use Weave’s feedback functionality in both the UI and SDK, query and manage feedback, and use human annotations for detailed evaluations. - [Provide feedback in the UI](#provide-feedback-in-the-ui) -- [Provide feedback via the SDK](#provide-feedback-via-the-sdk) +- [Provide feedback through the SDK](#provide-feedback-through-the-sdk) - [Add human annotations](#add-human-annotations) ## Provide feedback in the UI -In the Weave UI, you can add and view feedback [from the call details page](#from-the-call-details-page) or [using the icons](#use-the-icons). +In the Weave UI, you can add and view feedback [from the Call details panel](#from-the-call-details-panel) or [using the icons](#use-the-icons). -### From the call details page +### From the Call details panel -1. In the sidebar, navigate to **Traces**. -2. Find the row for the call that you want to add feedback to. -3. Open the call details page. -4. Select the **Feedback** column for the call. -5. Add, view, or delete feedback: - - _[Add and view feedback using the icons](#use-the-icons)_ located in the upper right corner of the call details feedback view. - - _View and delete feedback from the call details feedback table._ Delete feedback by clicking the trashcan icon in the rightmost column of the appropriate feedback row. +1. In the Weave project sidebar, navigate to **Traces**. +1. Find the row for the Call that you want to add feedback to. +1. Click the linked Trace name to open the trace tree and Call details panel. +1. In the Call details tab bar, choose **Feedback**. +1. Add, view, or delete feedback: + - _Add and view feedback using the icons_ located in the upper right corner of the Call details feedback view. + - _View and delete feedback from the Call details feedback table._ Delete feedback by clicking the trashcan icon in the rightmost column of the appropriate feedback row. -![Screenshot of Feedback tab in call details](/weave/guides/tracking/imgs/feedback_tab.png) +![Screenshot of Feedback tab in Call details](/weave/guides/tracking/imgs/feedback_tab.png) -### Use the icons +### Use the feedback icons -You can add or remove a reaction, and add a note using the icons that are located in both the call table and individual call details pages. +You can add or remove a reaction, and add a note using the icons that are located in both the Traces table and individual Call details panel. -- _Call table_: Located in **Feedback** column in the appropriate row in the call table. -- _Call details page_: Located in the upper right corner of each call details page. +- _Traces table_: Located in **Feedback** column in the appropriate row in the **Traces** table. +- _Call details panel_: Located in the upper right corner of each Call details panel. To add a reaction: @@ -50,12 +50,12 @@ To remove a reaction: 1. Hover over the emoji reaction you want to remove. 2. Click the reaction to remove it. -> You can also delete feedback from the [**Feedback** column on the call details page.](#from-the-call-details-page). +> You can also delete feedback from the [**Feedback** column on the Call details panel](#from-the-call-details-panel). To add a comment: 1. Click the comment bubble icon. -2. In the text box, add your note. +2. In the text box, add your note. The maximum number of characters in a feedback note is 1024. 3. To save the note, press the **Enter** key. You can add additional notes. @@ -66,9 +66,9 @@ The maximum number of characters in a feedback note is 1024. If a note exceeds t ![Screenshot of calls grid with feedback column](/weave/guides/tracking/imgs/feedback_calls.png) -## Provide feedback via the SDK +## Provide feedback through the SDK -> You can find SDK usage examples for feedback in the UI under the **Use** tab in the call details page. +> You can find SDK usage examples for feedback in the UI under the **Use** tab in the Call details panel. You can use the Weave SDK to programmatically add, remove, and query feedback on calls. @@ -77,7 +77,7 @@ You can use the Weave SDK to programmatically add, remove, and query feedback on You can query the feedback for your Weave project using the SDK. The SDK supports the following feedback query operations: - `client.get_feedback()`: Returns all feedback in a project. -- `client.get_feedback("")`: Return a specific feedback object specified by `` as a collection. +- `client.get_feedback("")`: Returns a specific feedback object specified by `` as a collection. - `client.get_feedback(reaction="")`: Returns all feedback objects for a specific reaction type. You can also get additional information for each feedback object in `client.get_feedback()`: @@ -85,7 +85,7 @@ You can also get additional information for each feedback object in `client.get_ - `id`: The feedback object ID. - `created_at`: The creation time information for the feedback object. - `feedback_type`: The type of feedback (reaction, note, custom). -- `payload`: The feedback payload +- `payload`: The feedback payload. @@ -119,9 +119,9 @@ You can also get additional information for each feedback object in `client.get_ -### Add feedback to a call +### Add feedback to a Call -You can add feedback to a call using the call's UUID. To use the UUID to get a particular call, [retrieve it during or after call execution](#retrieve-the-call-uuid). The SDK supports the following operations for adding feedback to a call: +You can add feedback to a Call using the Call's UUID. To use the UUID to get a particular Call, [retrieve it during or after Call execution](#retrieve-the-call-uuid). The SDK supports the following operations for adding feedback to a Call: - `call.feedback.add_reaction("")`: Add one of the supported `` (emojis), such as 👍. - `call.feedback.add_note("")`: Add a note. @@ -159,16 +159,13 @@ The maximum number of characters in a feedback note is 1024. If a note exceeds t -#### Retrieve the call UUID +#### Retrieve the Call UUID -For scenarios where you need to add feedback immediately after a call, you can retrieve the call UUID programmatically during or after the call execution. +For scenarios where you need to add feedback immediately after a Call, you can retrieve the Call UUID programmatically during or after the Call execution. -- [During call execution](#during-call-execution) -- [After call execution](#after-call-execution) +##### During Call execution -##### During call execution - -To retrieve the UUID during call execution, get the current call, and return the ID. +To retrieve the UUID during Call execution, get the current Call, and return the ID. @@ -195,9 +192,9 @@ To retrieve the UUID during call execution, get the current call, and return the -##### After call execution +##### After Call execution -Alternatively, you can use `call()` method to execute the operation and retrieve the ID after call execution: +Alternatively, you can use `call()` method to execute the operation and retrieve the ID after Call execution: @@ -222,7 +219,7 @@ Alternatively, you can use `call()` method to execute the operation and retrieve -### Delete feedback from a call +### Delete feedback from a Call You can delete feedback from a particular call by specifying a UUID. @@ -241,57 +238,59 @@ You can delete feedback from a particular call by specifying a UUID. ## Add human annotations -Human annotations are supported in the Weave UI. To make human annotations, you must first create a Human Annotation scorer using either the [UI](#create-a-human-annotation-scorer-in-the-ui) or the [API](#create-a-human-annotation-scorer-using-the-api). Then, you can [use the scorer in the UI to make annotations](#use-the-human-annotation-scorer-in-the-ui), and [modify your annotation scorers using the API](#modify-a-human-annotation-scorer-using-the-api). +Human annotations are supported in the Weave UI. This functionality allows you to create custom fields to add additional, human-entered data to your Traces as feedback. To make human annotations, you must first create a Human Annotation scorer using either the [UI](#create-a-human-annotation-scorer-in-the-ui) or the [API](#create-a-human-annotation-scorer-using-the-api). Then, you can [use the scorer in the UI to make annotations](#use-the-human-annotation-scorer-in-the-ui), and [modify your annotation scorers using the API](#modify-a-human-annotation-scorer-using-the-api). ### Create a human annotation scorer in the UI To create a human annotation scorer in the UI, do the following: -1. In the sidebar, navigate to **Scorers**. -2. In the upper right corner, click **+ Create scorer**. -3. In the configuration page, set: +1. In the project sidebar, navigate to **Assets**. +1. In the Assets navigation panel, click **Scorers**. +1. In the **Scorers** panel header, click **New scorer**. +1. In the **Create Scorer** modal dialog, set: - `Scorer type` to `Human annotation` - `Name` - `Description` - `Type`, which determines the type of feedback that will be collected, such as `boolean` or `integer`. -4. Click **Create scorer**. Now, you can [use your scorer to make annotations](#use-the-human-annotation-scorer-in-the-ui). +1. Click **Create scorer**. Now, you can use your scorer to make annotations. In the following example, a human annotator is asked to select which type of document the LLM ingested. As such, the `Type` selected for the score configuration is an `enum` containing the possible document types. -![Human Annotation scorer form](/weave/guides/tracking/imgs/human-annotation-scorer-form.png) +![Create Scorer modal dialog](/weave/guides/tracking/imgs/human-annotation-scorer-form.png) ### Use the human annotation scorer in the UI -Once you [create a human annotation scorer](#create-a-human-annotation-scorer-in-the-ui), it will automatically display in the **Feedback** sidebar of the call details page with the configured options. To use the scorer, do the following: +Once you create a human annotation scorer, it becomes available to use on the Traces page. -1. In the sidebar, navigate to **Traces** -2. Find the row for the call that you want to add a human annotation to. -3. Open the call details page. -4. In the upper right corner, click the **Show feedback** button. +To use the scorer, do the following: +1. In the project sidebar, navigate to **Traces**. +2. Find the row for the Call that you want to add a human annotation to. +3. Click the linked Trace name to open the trace tree and Call details panel. +4. In the upper right corner of the Call details tab bar, click the **Show feedback** button. - ![Marker icon in call header](/weave/guides/tracking/imgs/marker-icon.png) + ![Marker icon in Call header](/weave/guides/tracking/imgs/marker-icon.png) - Your available human annotation scorers display in the sidebar. + Your available human annotation scorers display in an additional **Annotate** panel. - ![Human Annotation scorer feedback sidebar](/weave/guides/tracking/imgs/full-feedback-sidebar.png) + ![Human Annotation scorer feedback panel](/weave/guides/tracking/imgs/full-feedback-sidebar.png) 5. Make an annotation. 6. Click **Save**. -7. In the call details page, click **Feedback** to view the calls table. The new annotation displays in the table. You can also view the annotations in the **Annotations** column in the call table in **Traces**. +7. In the Call details panel tab bar, click the **Feedback** tab to view the Feedback table. The new annotation displays in the table. You can also view the annotations in the **Annotations** column in the main Traces table. - > Refresh the call table to view the most up-to-date information. + > Refresh the Traces table to view the most up-to-date information. -![Human Annotation scorer feedback in calls table](/weave/guides/tracking/imgs/feedback-in-the-table.png) +![Human Annotation scorer feedback in Traces table](/weave/guides/tracking/imgs/feedback-in-the-table.png) ### Create a human annotation scorer using the API Human annotation scorers can also be created through the API. Each scorer is its own object, which is created and updated independently. To create a human annotation scorer programmatically, do the following: -1. Import the `AnnotationSpec` class from `weave.flow.annotation_spec` +1. Import the `AnnotationSpec` class from `weave.flow.annotation_spec`. 2. Use the `publish` method from `weave` to create the scorer. In the following example, two scorers are created. The first scorer, `Temperature`, is used to score the perceived temperature of the LLM call. The second scorer, `Tone`, is used to score the tone of the LLM response. Each scorer is created using `save` with an associated object ID (`temperature-scorer` and `tone-scorer`).