Skip to content

Add agent documentation and API references #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
raghul-velt wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Add agent documentation and API references #168

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5786301
Add agent documentation and API references
raghul-velt May 15, 2026
10dde07
Merge remote-tracking branch 'origin/main' into raghul/agent-docs
raghul-velt Jun 4, 2026
9d996ee
docs: add new API endpoints for activity, notification, and permissio…
raghul-velt Jun 4, 2026
e672656
docs: enhance comment annotations and agent configurations in API doc…
raghul-velt Jun 4, 2026
00d5f15
docs: update comment annotations and webhook configurations in API do…
raghul-velt Jun 4, 2026
866eca0
docs: clarify agent annotations structure in get-comment-annotations-…
raghul-velt Jun 4, 2026
09f967f
docs: update activity, email, notification, and webhook configuration…
raghul-velt Jun 4, 2026
c2db9b4
docs: clarify agent name requirements in comment annotations
raghul-velt Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 167 additions & 0 deletions ai/agents/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
---
title: "Agents"
sidebarTitle: "Overview"
description: "Run AI agents that analyze web pages, surface findings, and create comment annotations — without writing the analysis pipeline yourself."
---

## What are Agents?

Agents are AI-driven QA tools that analyze a web page (or set of pages) and surface findings as comment annotations. You configure what an agent looks for; the platform handles context gathering, LLM invocation, post-processing, deduplication, and annotation creation.

Build any check you want — brand consistency, accessibility, SEO, broken links, spelling, custom domain rules — by writing a prompt and selecting a few configuration options. No need to run your own browser pool, prompt your own model, or wire up annotation creation.

You define an **agent configuration** once, then call **Run Execution** with a URL. Results are persisted as a versioned execution document plus per-URL findings, and annotations are automatically posted to the document via the Comments API.

## What you get

- **Built-in and custom agents.** Use ready-made agents (spell check, broken links, grammar, PII detection, profanity, sensitive-data) or define your own custom agents from a prompt.

- **Pluggable context gathering.** Stack one or more strategies — page text, screenshots, HTML, CSS, links, accessibility tree, computed styles, robots.txt, sitemap, Lighthouse — and the engine extracts everything in parallel.

- **Versioned configurations.** Every behavioral edit creates a new version. In-flight executions stay pinned to the version they started on. One-click restore rolls back the most recent change.

- **Async execution with polling.** `Run Execution` returns an `executionId` immediately and dispatches a Cloud Task. Poll `Get Execution` until `status !== "running"`.

- **Cross-page execution.** Set `crossPageExecute: true` and the engine crawls the seed URL, processes up to `maxUrlsToProcess` pages, and aggregates findings across the site.

- **Match-and-merge dedup.** Re-running an agent against the same document only creates new annotations for new findings; pre-existing matches are skipped, and resolved findings are auto-resolved.

- **Token usage analytics.** Per-agent, per-model, per-month token consumption tracked automatically in Firestore and exposed via the Analytics endpoint.

- **Agent groups.** Organize related agents into named groups (e.g. "Brand QA") and filter list responses by group.

## Use cases

<CardGroup cols={2}>
<Card title="Brand consistency" icon="palette">
A custom agent verifies brand colors, typography, and logo placement across your marketing pages. Re-run on every deploy.
</Card>

<Card title="Pre-launch QA" icon="rocket">
Spell-check, broken-links, and accessibility agents run in parallel before a release. Findings appear as comment annotations on the staging document.
</Card>

<Card title="Content moderation" icon="shield-halved">
PII detection and profanity filter agents flag sensitive content on user-generated pages. Pair with Approval Engine for human review.
</Card>

<Card title="Cross-page audit" icon="sitemap">
Run a brand consistency agent in `crossPageExecute: true` mode against a marketing site. The crawler discovers internal links, then the agent checks each one.
</Card>

<Card title="Custom QA tasks" icon="wand-magic-sparkles">
Describe a check in plain English (e.g. "every CTA must use the primary brand color"). The Validate Prompt endpoint expands it into a structured QA task you can ship as an agent.
</Card>

<Card title="Workflow integration" icon="diagram-project">
Trigger an agent execution from an Approval Engine workflow node. The engine handles parking the workflow on the agent's findings.
</Card>
</CardGroup>

## How it works

1. **Define the agent.** Call [Create Agent](/api-reference/rest-apis/v2/agents/create) with a name, instructions, and a configuration block (context gathering strategies, execution strategy, post-processing options). The engine creates version 1 in the agent's versions subcollection.
2. **Run an execution.** Call [Run Execution](/api-reference/rest-apis/v2/agents/execution/run) with `agentId` and `url`. The engine creates an execution document, dispatches a Cloud Task, and returns an `executionId`.
3. **The pipeline runs.** Context gathering → LLM execution → post-processing (guardrails, match-and-merge, annotation creation, analytics) → response shaping. Every phase is bounded by `phaseTimeoutMs`.
4. **Poll for results.** Call [Get Execution](/api-reference/rest-apis/v2/agents/execution/get) until `status !== "running"`. Set `includeResults: true` to fetch per-URL findings.
5. **Findings appear as annotations.** When `postProcess.annotations.enabled` is true (default), each finding becomes a comment annotation on the document referenced by `organizationId` / `documentId`.

## Mental model

### Agent

An **agent** is a reusable QA configuration. Each agent has an identity (`name`, `description`, `enabled`) plus a behavioral configuration (instructions, context gathering, execution, post-processing). Identity edits update the root document; behavioral edits create a new version.

### Execution

An **execution** is one run of an agent against a seed URL. It pins the agent's `version` at dispatch time, so in-flight runs are immune to mid-run config changes.

**Lifecycle:**

```
running → passed | failed | error | skipped
```

### Finding

A **finding** is one issue the agent surfaced on one URL. Findings carry severity, category, target text, suggestion, HTML selector, and confidence. Findings below the guardrails confidence floor are suppressed unless explicitly disabled.

### Pipeline phases

Each agent run executes 5 sequential phases:

| Phase | What happens |
| ----------------- | ---------------------------------------------------------------------------------- |
| `validate` | Zod check on payload + input requirements + variable keys |
| `extract` | All configured context gathering strategies run (cached by tenant + content hash) |
| `execute` | LLM call (or registered service); results normalized to a finding array |
| `postProcess` | Guardrails → match-and-merge → annotation creation → analytics |
| `structureResponse` | Final response assembly with optional response adapter |

## Built-in agents

| Agent ID | Description | Internal |
| ------------------- | ---------------------------------------------------- | -------- |
| `spell-check` | Spelling and typo detection | no |
| `grammar-check` | Grammar checking | no |
| `broken-links` | Broken link validation | no |
| `pii-detection` | PII (personal info) detection | no |
| `profanity-filter` | Profanity detection | no |
| `sensitive-data` | Sensitive data detection | no |
| `screenshot` | Page screenshot capture | yes |
| `crawler` | Site crawling | yes |

Internal agents (`metadata.internal: true`) power the execution pipeline and are excluded from list responses.

## Context gathering strategies

Stack one or more strategies in the order you want them to run. Strategy results are cached by tenant + content hash, so re-running the same agent against the same page is cheap.

| Strategy | What it returns |
| ------------------------- | -------------------------------------------------------------- |
| `web-page-text` | Visible text extracted via Puppeteer |
| `web-page-screenshot` | Full-page screenshot as image |
| `web-page-html` | Cleaned HTML content |
| `web-page-css` | Cleaned CSS content from stylesheets |
| `web-page-links` | All hyperlinks on the page |
| `web-page-accessibility` | Accessibility tree (ARIA roles, landmarks, headings) |
| `computed-styles` | Computed CSS for every element |
| `robots-txt` | Fetched `robots.txt` |
| `sitemap-data` | Discovered and parsed XML sitemaps |
| `lighthouse` | Google Lighthouse audit (perf / a11y / SEO) |
| `none` | No context gathering (for service-only agents) |

## Execution strategies

| Strategy | Purpose |
| ------------------ | ------------------------------------------------------------- |
| `ai` | LLM-driven analysis (default) |
| `service` | Delegate to a registered built-in service (`serviceId` required) |
| `service+ai` | Service context gathering + AI analysis |
| `stagehand-agent` | Autonomous browser agent via Stagehand AI |

## Scope

| Field | Bound to |
| ---------------- | ---------------------------------------------------------- |
| `apiKey` | Workspace-wide (default; agents are per-workspace) |
| `organizationId` | Annotations created on the named organization |
| `documentId` | Annotations created on the named document |

`organizationId` and `documentId` on Run Execution control where annotations land. They are not required for the agent to execute.

## Async execution

Run Execution returns immediately with `{ executionId }` and dispatches a Cloud Task. The engine processes the URL(s) asynchronously and writes results to Firestore. Poll Get Execution to track progress.

## Get started

<CardGroup cols={2}>
<Card title="Setup" icon="gear" href="/ai/agents/setup">
Create an agent, run it against a URL, and read the findings end-to-end.
</Card>

<Card title="API Reference" icon="code" href="/api-reference/rest-apis/v2/agents/create">
All endpoints organized into Agents, Execution, Versioning, Prompt Tools, Analytics, and Groups.
</Card>
</CardGroup>
188 changes: 188 additions & 0 deletions ai/agents/setup.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
---
title: "Setup"
description: "Build an agent end to end: create, run, poll, and inspect findings."
---

This guide walks through the five steps to get an agent running and surfacing findings as comment annotations. Each step links to the full API reference for request and response details.

## Step 1: (Optional) Refine your prompt

If you have a one-line QA instruction and you're not sure it's specific enough to ship, run it through the prompt tools first.

**Check completeness:** [Enhance Prompt](/api-reference/rest-apis/v2/agents/prompt/enhance) returns either "your prompt is sufficient" or a specific clarification request (with suggested options for the user to pick from).

**Expand into a structured task:** [Validate Prompt](/api-reference/rest-apis/v2/agents/prompt/validate) takes a simple instruction and expands it into a comprehensive QA task with an analysis prompt, response descriptions, demo examples, and tool requirements.

**Iterate on demo failures:** [Refine Prompt](/api-reference/rest-apis/v2/agents/prompt/refine) takes an existing analysis prompt and a list of demo failures (with feedback) and produces a refined prompt.

**Resolve optimal config:** [Resolve Config](/api-reference/rest-apis/v2/agents/config/resolve) takes raw + processed instructions and recommends extraction strategies + execution strategy.

These tools are optional. Skip straight to Step 2 if you already know what your agent should do.

## Step 2: Create the agent

Call [Create Agent](/api-reference/rest-apis/v2/agents/create) with a name, instructions, and a configuration block.

The minimum viable agent is `name` + `instructions` + `contextGathering.strategies`:

```JSON
{
"data": {
"name": "Brand Color Checker",
"instructions": "Verify all CTAs use the primary brand color #1A73E8.",
"contextGathering": {
"strategies": ["web-page-text", "web-page-screenshot"]
}
}
}
```

The engine creates the agent's main document and writes version 1 to its versions subcollection. The response returns `{ agentId }`.

For a complete walkthrough of every config block (`execution`, `response`, `postProcess`, `input`, `scope`, `setup`, `metadata`), see the [Create Agent](/api-reference/rest-apis/v2/agents/create) reference.

<Info>
Built-in agents (`spell-check`, `broken-links`, etc.) are pre-registered for every workspace. You don't need to create them — skip straight to Step 3 with one of the [built-in agent IDs](/ai/agents/overview#built-in-agents).
</Info>

## Step 3: Run an execution

Call [Run Execution](/api-reference/rest-apis/v2/agents/execution/run) with `agentId` and `url`. Pass `organizationId` and `documentId` if you want findings to land as annotations on a specific document.

```JSON
{
"data": {
"agentId": "abc123def456",
"url": "https://example.com/pricing",
"organizationId": "org_001",
"documentId": "doc_001",
"ranBy": {
"userId": "user_123",
"name": "Jane Doe",
"email": "jane@example.com"
}
}
}
```

The response returns immediately with `{ executionId }`. The engine creates an execution document, dispatches a Cloud Task, and starts processing asynchronously.

**Cross-page execution:** Set `crossPageExecute: true` and the engine crawls the seed URL and processes up to `maxUrlsToProcess` pages (default 50).

**Workflow trigger:** Set `trigger: "workflow"` and pass `workflowExecutionId` when running an agent as part of an Approval Engine workflow node.

## Step 4: Poll for results

Call [Get Execution](/api-reference/rest-apis/v2/agents/execution/get) with the `executionId` returned from Step 3. Poll until `execution.status !== "running"`.

```JSON
{
"data": {
"executionId": "exec_1711900000000_abc123def456"
}
}
```

Terminal statuses:

| Status | Meaning |
| ----------- | ------------------------------------------------------------------ |
| `passed` | Execution completed successfully |
| `failed` | Execution completed with failures |
| `error` | Unrecoverable error (see `execution.error.code` and `retryable`) |
| `skipped` | Execution was skipped (precondition unmet) |

When the execution completes, `resultsSummary` contains aggregate counts:

```JSON
{
"totalFindings": 7,
"totalAnnotationsCreated": 5,
"urlsProcessed": 12,
"urlsWithFindings": 5,
"matchResult": {
"created": 5,
"skipped": 2,
"resolved": 1
}
}
```

`matchResult` is populated when `postProcess.matchAndMerge.enabled` is true (the default). It tells you how many findings were new (`created`), already existed as annotations (`skipped`), or were resolved because the previous finding is no longer detected (`resolved`).

## Step 5: Fetch per-URL findings

Set `includeResults: true` on Get Execution to fetch the full findings array per URL:

```JSON
{
"data": {
"executionId": "exec_1711900000000_abc123def456",
"includeResults": true
}
}
```

The response includes a `results` array alongside `execution`. Each entry has the URL, its findings, and per-URL counts:

```JSON
{
"results": [
{
"url": "https://example.com/pricing",
"urlHash": "a1b2c3d4e5f6",
"findings": [
{
"id": "finding-1",
"title": "CTA uses non-brand color",
"description": "The 'Buy now' button uses #FF5722 instead of #1A73E8",
"severity": "high",
"category": "color",
"targetText": "Buy now",
"suggestion": "Change background-color to #1A73E8",
"htmlSelector": "a.cta-primary",
"isPageLevel": false,
"issueType": "brand-color",
"confidence": 95
}
],
"findingCount": 1,
"annotationsCreated": 1,
"processedAt": 1711900050000
}
]
}
```

Findings are also written to the document as comment annotations (when `postProcess.annotations.enabled` is true and `organizationId` + `documentId` were provided on dispatch). Authors of these annotations are flagged with the system `AGENT_USER`.

## Iterating on the agent

Behavioral edits create a new version. Identity edits do not.

- **Edit identity (`name`, `description`, `enabled`):** [Update Agent](/api-reference/rest-apis/v2/agents/update). No new version; the root doc is updated in place.
- **Edit behavior (`instructions`, `contextGathering`, `execution`, `postProcess`, `input`, `scope`, `setup`):** [Update Agent Version](/api-reference/rest-apis/v2/agents/version/update). Creates version `N+1` and bumps the agent's `version` pointer. In-flight executions stay pinned to whatever version they started on.
- **List versions:** [List Versions](/api-reference/rest-apis/v2/agents/versions/list). Each version contains the full behavioral snapshot.
- **Roll back the most recent change:** [Restore Version](/api-reference/rest-apis/v2/agents/versions/restore). Single-step undo; call repeatedly to walk backward. Cannot go below version 1.

## Organizing agents

Use [Agent Groups](/api-reference/rest-apis/v2/agents/groups/create) to bundle related agents together (e.g. "Brand QA"). Agents stay independent documents — groups just store an `agentIds` array. An agent can belong to multiple groups; deleting an agent removes it from every group automatically.

When listing agents with [Get Agent(s)](/api-reference/rest-apis/v2/agents/get), pass `groupId` to filter to a single group's members.

## Track usage

[Get Agent Analytics](/api-reference/rest-apis/v2/agents/analytics/get) returns per-agent, per-model, per-month token consumption plus execution counts. Filter by `agentId`, `year`, `month`, or `model`.

## Next steps

<CardGroup cols={2}>
<Card title="API Reference" icon="code" href="/api-reference/rest-apis/v2/agents/create">
All endpoints organized into Agents, Execution, Versioning, Prompt Tools, Analytics, and Groups, with full request and response schemas.
</Card>

<Card title="Overview" icon="book-open" href="/ai/agents/overview">
What agents are, the 5-phase pipeline, built-in vs custom agents, and the full enum vocabulary.
</Card>
</CardGroup>
Loading