experiment-bud

An autonomous conversion-rate-optimization (CRO) experiment agent, implemented as a set of Warp agent skills. It runs the full A/B-testing lifecycle for a marketing site — research pages, propose hypotheses, build experiments, monitor them, analyze results, and chain to the next experiment — with humans in the loop for prioritization and merges.

This is a reference implementation. It was open-sourced from an internal project, so all infrastructure identifiers, credentials, and team-specific data have been replaced with placeholders (e.g. your-gcp-project, NOTION_*_ID, SLACK_*, your-org/your-marketing-site). It will not run as-is — you must supply your own integrations and values. See How to deploy.

How it works

The system is a pipeline of eight skills that move each experiment hypothesis through a status lifecycle in Notion. See .warp/skills/cro/README.md for the full design: the loop diagram, statistical methodology (frequentist + Bayesian), and guardrails.

Skill	Trigger	What it does
page_registry_sync	daily cron	Syncs page traffic/CVR and computes each page's research status
research	status webhook	Researches a page and writes A/B hypotheses to Notion
backlog_monitor	2×/day cron	Dispatches builds, flags idle pages, triggers research
build	status webhook	Builds an experiment at a 50/50 split and opens a PR
experiment_monitor	2×/day cron	Manages the post-merge lifecycle: validate → run → conclude
analyze	status webhook	Full analysis, report, ship/kill PR
weekly_recap	Friday cron	Posts a weekly recap to Slack
owner_sync	field webhook	Propagates page ownership across Notion databases

Integrations

• Notion — system of record (Page Registry, Roadmap & Backlog, Knowledge Base, Analysis Log, Reports).
• BigQuery — traffic and conversion analytics (queried via the bq CLI).
• Metabase — experiment dashboards and a fallback query path.
• Slack — experiment notifications and the weekly recap.
• Google Ads API — keyword and campaign research.
• Sanity + a Next.js marketing site — where experiment variants are actually built and served (a separate repository).
• Warp / Oz — runs the skills on schedules and webhooks.

How to deploy

Prerequisites

• Warp with agent skills enabled
• Python 3.10+
• Accounts and credentials for the integrations listed above

Fastest path: clone this repo, open it in Warp, and ask the agent to run the setup skill. It interactively collects your values, replaces every placeholder, writes .env, and verifies each integration.

Alternatively, follow the manual steps below.

1. Secrets and environment variables

Copy .env.example to .env (or configure the values as Oz team secrets) and fill in real values. See that file for the full list.

2. Replace placeholders

This repo ships with placeholder identifiers instead of real ones. Search-and-replace the following with your own values before running:

• your-gcp-project — your GCP project id
• ADS_CUSTOMER_ID — your Google Ads customer id (used as a suffix on Google Ads tables)
• BigQuery dataset/table names — the SQL files use illustrative names (analytics.website_conversion, events.pages, etc.); replace with your actual dataset and table names
• BigQuery column names — fields like did_download_same_day, is_excluded_page_type_1 are illustrative; replace with your actual conversion event columns
• your-rudderstack-source / your-legacy-rudderstack-source — your Rudderstack source names as they appear in BigQuery
• your-rudderstack-property — your Rudderstack property name used in property IN (...) filters
• NOTION_*_ID — your Notion database and page ids (Page Registry, Roadmap, Knowledge Base, Analysis Log, Reports, etc.)
• notion.so/your-workspace/ — your Notion workspace slug
• SLACK_*_CHANNEL_ID, SLACK_BOT_USER_ID, SLACK_APP_ID, SLACK_BOT_ID, #your-*-channel, @your-bot
• your-sanity-project — your Sanity project id
• your-org/your-marketing-site — your marketing-site repository
• Host placeholders: your-webhook-host.example.com, your-oz-host.example.com, metabase.example.com, www.example.com
• SCHEDULE_ID_*, CRO_SILENT_ENV_ID, CRO_MAIN_ENV_ID — your Oz schedule and environment ids
• .warp/skills/cro/references/owners.py — replace the placeholder Owner → Slack/GitHub mappings with your team's

3. Notion schema

The skills expect specific Notion databases and fields. See .warp/skills/cro/references/config.md and the per-skill SKILL.md files for the required schema.

Repository layout

AGENTS.md                     # top-level agent instructions
.env.example                  # required environment variables
.warp/skills/setup/SKILL.md   # interactive setup skill (run with Warp)
.warp/skills/cro/
├── README.md                 # detailed system design
├── <skill>/SKILL.md          # one directory per skill
├── references/               # shared config, priors, owner mapping
└── */scripts/                # SQL templates + Python stats/dashboard helpers

License

MIT.