Rebuild Annotation App

[nadahlberg]

This PR includes a from-scratch rebuild of the annotation app. Fixes #144 see issue for the motivation.

An Agent-first Annotation App

Overview

Similar to the old app, the core structure of this app is that there are Projects which include Labels and Documents. Data can be added to projects from a list of texts or a DataFrame. Columns other than text are stored in Documents in a meta dict, which we may add search support for in the future.

from clx.models import Project

project = Project.objects.get(name="My Project")
project.add_docs(texts)

Run the app with clx manage runserver. Projects and labels can be added through the UI.

Texts get a trigram index for making expensive boolean logic on substring comparisons more efficient. A text_prefix field (50 chars) is added with an index for more efficient startswith queries.

Examples can be sampled into a label's training set, these are stored as LabelDocuments.

LabelDocuments can be linked to an Annotation. Annotations are scoped by source although currently only source="agent" is supported.

Agent Tools

The agent has several tools allowing it to manage itself and the label training data:

Memory management tools:

• Consolidate Memory: Generate a compacted summary of the current session replacing the conversation history. Used to keep costs manageable with long running sessions.
• Clear Tool History: Wipes the tool inputs and outputs of earlier tool calls / response messages in the conversation. Search and annotation tools add a ton of tokens. This allows us to drop these inputs, dramatically reducing a session's tokens without losing anything else from the conversation history.
• Update project / label instructions: Project and label instructions are injected in the agent's system prompt for long term memory across sessions. Agents have tools to overwrite or append to these.

Data management tools:

• Search: Allows the agent to search through the project data. It has an expressively powerful text query parameter, allowing it to do substring searches with arbitrarily composed AND, OR, NOT, and STARTSWITH operators. Search can also filter documents by their annotation and prediction values for the current label.
• Add Trainset Examples: Allows the agent to sample examples from a search into the trainset. Examples are added from a prior search tool call.
• Annotate: Allows the agent to annotate examples in bulk. Annotations reference document ids returned by the search tool.

User interaction tools:

• Ask User Question: This tool allows the agent to pause an automated workflow and present a question and proposed answer choices to the user. The workflow will be flagged as requiring user input, so that the user can easily jump around to tasks that require intervention.
• Complete Task: This tool is used for tasks run in autopilot mode, allowing the agent to signal that a task is complete so it can move on to the next.

Workflow Prompts

There are currently three pre-defined steps that can be customized per-project. These are just prompts that instruct the agent to take certain actions in a specific order. Users can overwrite the default instructions for their project, making this fully customizable depending on project needs. Users can also define custom project-specific workflow prompts that can be triggered manually.

• Label Understanding: This is a prompt that instructs the agent to get familiar with the data, come up with an initial hypothesis about the label's annotation boundaries, and then conduct an interview with the user to clarify it's understanding of the label
• Sampling Strategy: This is a prompt that instructs the agent on how to build an initial training set. The default instructions mirror our approach for the old version of this app. We ask the agent to come up with minimal and likely keyword conditions and sample evenly across excluded, neutral, and likely buckets. The agent is then asked to sample from searches anchored on specific tricky language.
• Annotate: This instructs the agent to search for any unannotated examples and annotate them in batches of 100. The agent is also instructed to clear its tool history between batches to free up context.

Patterns for Dynamic Configs with Natural Language

Our sampling strategy is quite similar to our old approach, except things like the creation of minimal / likely condition queries or the number of examples to target for the trainingset are not baked into the application logic. Instead the workflow steps are just described in the workflow prompt.

One pattern that is useful here is the ability to have a prompt that asks the agent to store information in the project or label instructions if it doesn't already exist. For example, in our sampling prompt we tell the agent that the project instructions should mention the number of examples to target for each trainingset, and that if this isn't present to ask the user about it and add it. So for the first label in a project, the agent will prompt the user for this info. Then on subsequent labels it will see the needed info in the project instructions and move forward automatically. Thus we can use project and label instructions as a kind of dynamic configuration file that can be used any way your project needs.

We do something similar with minimal and likely keyword searches -- asking the agent to store it in label memory so it can be referenced in future tasks.

Autopilot Mode

Autopilot mode is a worker that runs in the background and automatically runs the three pre-defined workflows for each label. When a label is initialized, a label understanding task is triggered. Once the user has responded to any open questions, the sampling strategy is triggered. Then the annotation workflow runs on repeat any time there are unannotated examples.

Autopilot runs in a dedicated thread visible on the chat tab. All autopilot tasks run in the same long-running session which is automatically compacted when the session exceeds 25k tokens. The user can see the autopilot tasks running and can inject messages if it gets stuck.

Finetunes and Predictions

Once a label has annotated data, buttons will appear for finetuning a single-label classifier and making predicitons. Finetuning jobs are serverless and can be fired off in the background, a progress bar tracking the current steps. Predictions can then be added to the trainset from the finetuned model. These will display in search alongside the annotated value. We have search filter helpers for filtering on prediction-annotation disagreements and for sorting examples by confidence scores close to 0.5.

Multi-label Training

There is also a CLI script that makes a project-level trainset that combines the unique documents from all labels' trainsets. It will compute predictions across all examples for all labels, updating automatically when a new model has been finetuned and the predictions are stale. These are then used as a training set for training a single, multi-label model that can make predictions across all labels.

Screenshots and examples

Chat with an agent directly to manage your label

Agent pauses automated workflows and requests input from users

Search through project data, review annotations, predictions, confidence scores

Customize and add new workflow prompts for your project

Trigger finetuning and prediction Jobs

See autopilot progress and what needs human input

Get an overview of label progress, trainset distribution, and finetune scores