Skip to content

instruction correction to no-longer-existing Scorers tab DOCS-1912 #2196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anastasiaguspan wants to merge 2 commits into
base: main
Choose a base branch
from
Open

instruction correction to no-longer-existing Scorers tab DOCS-1912 #2196

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9013a00
instruction correction to no-longer-existing Scorers tab
anastasiaguspan Feb 25, 2026
a21a4e3
adding explanatory sentence
anastasiaguspan Feb 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion weave/guides/evaluation/scorers.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ feedback = list(call.feedback)
![Call Scores Tab](./img/call_scores_tab.png)
</Frame>

Scores for an individual call are displayed in the call details page under the "Scores" tab.
Scores for an individual call are displayed in the Call details panel under the "Scores" tab.

### Analyze multiple Calls' Scores

Expand Down
103 changes: 51 additions & 52 deletions weave/guides/tracking/feedback.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Collect feedback and use annotations"
description: "Collect and analyze feedback for LLM applications through UI and SDK"
---

Efficiently evaluating LLM applications requires robust tooling to collect and analyze feedback. Weave provides an integrated feedback system, allowing users to provide call feedback directly through the UI or programmatically via the SDK. Various feedback types are supported, including emoji reactions, textual comments, and structured data, enabling teams to:
Efficiently evaluating LLM applications requires robust tooling to collect and analyze feedback. W&B Weave provides an integrated feedback system, allowing users to provide Call feedback directly through the UI or programmatically through the SDK. Various feedback types are supported, including emoji reactions, textual comments, and structured data, enabling teams to:

- Build evaluation datasets for performance monitoring.
- Identify and resolve LLM content issues effectively.
Expand All @@ -12,33 +12,33 @@ Efficiently evaluating LLM applications requires robust tooling to collect and a
This guide covers how to use Weave’s feedback functionality in both the UI and SDK, query and manage feedback, and use human annotations for detailed evaluations.

- [Provide feedback in the UI](#provide-feedback-in-the-ui)
- [Provide feedback via the SDK](#provide-feedback-via-the-sdk)
- [Provide feedback through the SDK](#provide-feedback-through-the-sdk)
- [Add human annotations](#add-human-annotations)

## Provide feedback in the UI

In the Weave UI, you can add and view feedback [from the call details page](#from-the-call-details-page) or [using the icons](#use-the-icons).
In the Weave UI, you can add and view feedback [from the Call details panel](#from-the-call-details-panel) or [using the icons](#use-the-icons).

### From the call details page
### From the Call details panel

1. In the sidebar, navigate to **Traces**.
2. Find the row for the call that you want to add feedback to.
3. Open the call details page.
4. Select the **Feedback** column for the call.
5. Add, view, or delete feedback:
- _[Add and view feedback using the icons](#use-the-icons)_ located in the upper right corner of the call details feedback view.
- _View and delete feedback from the call details feedback table._ Delete feedback by clicking the trashcan icon in the rightmost column of the appropriate feedback row.
1. In the Weave project sidebar, navigate to **Traces**.
1. Find the row for the Call that you want to add feedback to.
1. Click the linked Trace name to open the trace tree and Call details panel.
1. In the Call details tab bar, choose **Feedback**.
1. Add, view, or delete feedback:
- _Add and view feedback using the icons_ located in the upper right corner of the Call details feedback view.
- _View and delete feedback from the Call details feedback table._ Delete feedback by clicking the trashcan icon in the rightmost column of the appropriate feedback row.

<Frame>
![Screenshot of Feedback tab in call details](/weave/guides/tracking/imgs/feedback_tab.png)
![Screenshot of Feedback tab in Call details](/weave/guides/tracking/imgs/feedback_tab.png)
</Frame>

### Use the icons
### Use the feedback icons

You can add or remove a reaction, and add a note using the icons that are located in both the call table and individual call details pages.
You can add or remove a reaction, and add a note using the icons that are located in both the Traces table and individual Call details panel.

- _Call table_: Located in **Feedback** column in the appropriate row in the call table.
- _Call details page_: Located in the upper right corner of each call details page.
- _Traces table_: Located in **Feedback** column in the appropriate row in the **Traces** table.
- _Call details panel_: Located in the upper right corner of each Call details panel.

To add a reaction:

Expand All @@ -50,12 +50,12 @@ To remove a reaction:
1. Hover over the emoji reaction you want to remove.
2. Click the reaction to remove it.

> You can also delete feedback from the [**Feedback** column on the call details page.](#from-the-call-details-page).
> You can also delete feedback from the [**Feedback** column on the Call details panel](#from-the-call-details-panel).

To add a comment:

1. Click the comment bubble icon.
2. In the text box, add your note.
2. In the text box, add your note. The maximum number of characters in a feedback note is 1024.
3. To save the note, press the **Enter** key. You can add additional notes.

<Warning>
Expand All @@ -66,9 +66,9 @@ The maximum number of characters in a feedback note is 1024. If a note exceeds t
![Screenshot of calls grid with feedback column](/weave/guides/tracking/imgs/feedback_calls.png)
</Frame>

## Provide feedback via the SDK
## Provide feedback through the SDK

> You can find SDK usage examples for feedback in the UI under the **Use** tab in the call details page.
> You can find SDK usage examples for feedback in the UI under the **Use** tab in the Call details panel.
Copy link
Contributor

@dbrian57 dbrian57 Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> You can find SDK usage examples for feedback in the UI under the **Use** tab in the Call details panel.
You can find SDK usage examples for feedback in the UI under the **Use** tab in the Call details panel.

pretty sure we don't need to indent this.


You can use the Weave SDK to programmatically add, remove, and query feedback on calls.
Copy link
Contributor

@dbrian57 dbrian57 Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can use the Weave SDK to programmatically add, remove, and query feedback on calls.
You can use the Weave Python SDK to programmatically [add](/weave/reference/python-sdk/trace/feedback#method-add), [remove](/weave/reference/python-sdk/trace/feedback#method-purge), and [query feedback](/weave/reference/python-sdk/trace/weave_client#method-get_feedback) on calls. The TypeScript SDK does not currently support feedback functionality.

I know this was outside the scope of this PR, but this definitely needed some links 😅


Expand All @@ -77,15 +77,15 @@ You can use the Weave SDK to programmatically add, remove, and query feedback on
You can query the feedback for your Weave project using the SDK. The SDK supports the following feedback query operations:

- `client.get_feedback()`: Returns all feedback in a project.
- `client.get_feedback("<feedback_uuid>")`: Return a specific feedback object specified by `<feedback_uuid>` as a collection.
- `client.get_feedback("<feedback_uuid>")`: Returns a specific feedback object specified by `<feedback_uuid>` as a collection.
- `client.get_feedback(reaction="<reaction_type>")`: Returns all feedback objects for a specific reaction type.

You can also get additional information for each feedback object in `client.get_feedback()`:

- `id`: The feedback object ID.
- `created_at`: The creation time information for the feedback object.
- `feedback_type`: The type of feedback (reaction, note, custom).
- `payload`: The feedback payload
- `payload`: The feedback payload.

<Tabs>
<Tab title="Python">
Expand Down Expand Up @@ -119,9 +119,9 @@ You can also get additional information for each feedback object in `client.get_
</Tab>
</Tabs>

### Add feedback to a call
### Add feedback to a Call

You can add feedback to a call using the call's UUID. To use the UUID to get a particular call, [retrieve it during or after call execution](#retrieve-the-call-uuid). The SDK supports the following operations for adding feedback to a call:
You can add feedback to a Call using the Call's UUID. To use the UUID to get a particular Call, [retrieve it during or after Call execution](#retrieve-the-call-uuid). The SDK supports the following operations for adding feedback to a Call:

- `call.feedback.add_reaction("<reaction_type>")`: Add one of the supported `<reaction_types>` (emojis), such as 👍.
- `call.feedback.add_note("<note>")`: Add a note.
Expand Down Expand Up @@ -159,16 +159,13 @@ The maximum number of characters in a feedback note is 1024. If a note exceeds t
</Tab>
</Tabs>

#### Retrieve the call UUID
#### Retrieve the Call UUID

For scenarios where you need to add feedback immediately after a call, you can retrieve the call UUID programmatically during or after the call execution.
For scenarios where you need to add feedback immediately after a Call, you can retrieve the Call UUID programmatically during or after the Call execution.

- [During call execution](#during-call-execution)
- [After call execution](#after-call-execution)
##### During Call execution

##### During call execution

To retrieve the UUID during call execution, get the current call, and return the ID.
To retrieve the UUID during Call execution, get the current Call, and return the ID.

<Tabs>
<Tab title="Python">
Expand All @@ -195,9 +192,9 @@ To retrieve the UUID during call execution, get the current call, and return the
</Tab>
</Tabs>

##### After call execution
##### After Call execution

Alternatively, you can use `call()` method to execute the operation and retrieve the ID after call execution:
Alternatively, you can use `call()` method to execute the operation and retrieve the ID after Call execution:

<Tabs>
<Tab title="Python">
Expand All @@ -222,7 +219,7 @@ Alternatively, you can use `call()` method to execute the operation and retrieve
</Tab>
</Tabs>

### Delete feedback from a call
### Delete feedback from a Call

You can delete feedback from a particular call by specifying a UUID.

Expand All @@ -241,57 +238,59 @@ You can delete feedback from a particular call by specifying a UUID.

## Add human annotations

Human annotations are supported in the Weave UI. To make human annotations, you must first create a Human Annotation scorer using either the [UI](#create-a-human-annotation-scorer-in-the-ui) or the [API](#create-a-human-annotation-scorer-using-the-api). Then, you can [use the scorer in the UI to make annotations](#use-the-human-annotation-scorer-in-the-ui), and [modify your annotation scorers using the API](#modify-a-human-annotation-scorer-using-the-api).
Human annotations are supported in the Weave UI. This functionality allows you to create custom fields to add additional, human-entered data to your Traces as feedback. To make human annotations, you must first create a Human Annotation scorer using either the [UI](#create-a-human-annotation-scorer-in-the-ui) or the [API](#create-a-human-annotation-scorer-using-the-api). Then, you can [use the scorer in the UI to make annotations](#use-the-human-annotation-scorer-in-the-ui), and [modify your annotation scorers using the API](#modify-a-human-annotation-scorer-using-the-api).

### Create a human annotation scorer in the UI

To create a human annotation scorer in the UI, do the following:

1. In the sidebar, navigate to **Scorers**.
2. In the upper right corner, click **+ Create scorer**.
3. In the configuration page, set:
1. In the project sidebar, navigate to **Assets**.
1. In the Assets navigation panel, click **Scorers**.
1. In the **Scorers** panel header, click **New scorer**.
1. In the **Create Scorer** modal dialog, set:
- `Scorer type` to `Human annotation`
- `Name`
- `Description`
- `Type`, which determines the type of feedback that will be collected, such as `boolean` or `integer`.
4. Click **Create scorer**. Now, you can [use your scorer to make annotations](#use-the-human-annotation-scorer-in-the-ui).
1. Click **Create scorer**. Now, you can use your scorer to make annotations.

In the following example, a human annotator is asked to select which type of document the LLM ingested. As such, the `Type` selected for the score configuration is an `enum` containing the possible document types.

<Frame>
![Human Annotation scorer form](/weave/guides/tracking/imgs/human-annotation-scorer-form.png)
![Create Scorer modal dialog](/weave/guides/tracking/imgs/human-annotation-scorer-form.png)
</Frame>

### Use the human annotation scorer in the UI

Once you [create a human annotation scorer](#create-a-human-annotation-scorer-in-the-ui), it will automatically display in the **Feedback** sidebar of the call details page with the configured options. To use the scorer, do the following:
Once you create a human annotation scorer, it becomes available to use on the Traces page.

1. In the sidebar, navigate to **Traces**
2. Find the row for the call that you want to add a human annotation to.
3. Open the call details page.
4. In the upper right corner, click the **Show feedback** button.
To use the scorer, do the following:
1. In the project sidebar, navigate to **Traces**.
2. Find the row for the Call that you want to add a human annotation to.
3. Click the linked Trace name to open the trace tree and Call details panel.
4. In the upper right corner of the Call details tab bar, click the **Show feedback** button.

![Marker icon in call header](/weave/guides/tracking/imgs/marker-icon.png)
![Marker icon in Call header](/weave/guides/tracking/imgs/marker-icon.png)

Your available human annotation scorers display in the sidebar.
Your available human annotation scorers display in an additional **Annotate** panel.

![Human Annotation scorer feedback sidebar](/weave/guides/tracking/imgs/full-feedback-sidebar.png)
![Human Annotation scorer feedback panel](/weave/guides/tracking/imgs/full-feedback-sidebar.png)

5. Make an annotation.
6. Click **Save**.
7. In the call details page, click **Feedback** to view the calls table. The new annotation displays in the table. You can also view the annotations in the **Annotations** column in the call table in **Traces**.
7. In the Call details panel tab bar, click the **Feedback** tab to view the Feedback table. The new annotation displays in the table. You can also view the annotations in the **Annotations** column in the main Traces table.

> Refresh the call table to view the most up-to-date information.
> Refresh the Traces table to view the most up-to-date information.

<Frame>
![Human Annotation scorer feedback in calls table](/weave/guides/tracking/imgs/feedback-in-the-table.png)
![Human Annotation scorer feedback in Traces table](/weave/guides/tracking/imgs/feedback-in-the-table.png)
</Frame>

### Create a human annotation scorer using the API

Human annotation scorers can also be created through the API. Each scorer is its own object, which is created and updated independently. To create a human annotation scorer programmatically, do the following:

1. Import the `AnnotationSpec` class from `weave.flow.annotation_spec`
1. Import the `AnnotationSpec` class from `weave.flow.annotation_spec`.
2. Use the `publish` method from `weave` to create the scorer.

In the following example, two scorers are created. The first scorer, `Temperature`, is used to score the perceived temperature of the LLM call. The second scorer, `Tone`, is used to score the tone of the LLM response. Each scorer is created using `save` with an associated object ID (`temperature-scorer` and `tone-scorer`).
Expand Down
Loading