diff --git a/CLAUDE.md b/CLAUDE.md index 431105d..15467b3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,12 +7,14 @@ This plugin integrates Confidence with Claude Code, providing tools for feature - `/confidence:migrate-posthog >` — Migrate feature flags from PostHog to Confidence SDK - `/confidence:migrate-eppo >` — Migrate feature flags from Eppo to Confidence SDK - `/confidence:migrate-statsig >` — Migrate feature flag definitions from Statsig to Confidence (Phase 1; code transformation ships separately) +- `/confidence:onboard-confidence ` — Create accounts, onboard users, set up SDK clients, configure warehouses, and learn experimentation concepts ## Skills - **migrate-posthog** — Auto-triggers when the user asks to migrate PostHog flags or transform SDK code to Confidence - **migrate-eppo** — Auto-triggers when the user asks to migrate Eppo flags or transform SDK code to Confidence - **migrate-statsig** — Auto-triggers when the user asks to migrate Statsig gates/configs/experiments to Confidence +- **onboard-confidence** — Auto-triggers when the user asks to create a Confidence account, invite users, set up SDK clients, configure warehouses, run the setup wizard, or learn about experimentation ## MCP Servers diff --git a/commands/onboard-confidence.md b/commands/onboard-confidence.md new file mode 100644 index 0000000..ba937a7 --- /dev/null +++ b/commands/onboard-confidence.md @@ -0,0 +1,9 @@ +--- +name: onboard-confidence +description: Create Confidence accounts and onboard users +argument-hint: [create-account | invite-user | create-client | setup-wizard | setup-warehouse | learn | status] +--- + +All onboarding instructions are maintained in `skills/onboard-confidence/SKILL.md` to prevent divergence. + +**Before doing anything else**, use the Read tool to read `skills/onboard-confidence/SKILL.md` and follow those instructions to handle this command. diff --git a/skills/onboard-confidence-dry-run/SKILL.md b/skills/onboard-confidence-dry-run/SKILL.md new file mode 100644 index 0000000..6b958ae --- /dev/null +++ b/skills/onboard-confidence-dry-run/SKILL.md @@ -0,0 +1,1381 @@ +--- +description: Dry-run the Confidence onboarding flow to test UX without real API calls. Use when the user says "dry run", "test onboarding", "demo onboarding", or wants to preview the onboarding experience. +--- + +# Confidence Onboarding — Dry Run + +This skill runs the full onboarding experience with simulated API responses. No real accounts, flags, or warehouses are created. Use it to test the UX flow, demo to stakeholders, or train new users. + +## How it works + +- Every API call is simulated with realistic mock responses +- The step trackers, questions, confirmations, and explanations are identical to the real skill +- Browser login is skipped — a mock token is used +- All warehouse types can be tested without needing actual AWS/GCP/Snowflake/Databricks accounts + +## Commands + +| Command | What it simulates | +|---------|-------------------| +| `/onboard-confidence-dry-run create-account` | Account creation flow | +| `/onboard-confidence-dry-run invite-user` | User invitation flow | +| `/onboard-confidence-dry-run create-client` | SDK client creation flow | +| `/onboard-confidence-dry-run setup-wizard` | Full setup wizard (client → flag → variants → targeting → resolve) | +| `/onboard-confidence-dry-run setup-warehouse` | Warehouse setup dispatcher | +| `/onboard-confidence-dry-run setup-warehouse-bigquery` | BigQuery warehouse setup | +| `/onboard-confidence-dry-run setup-warehouse-snowflake` | Snowflake warehouse setup | +| `/onboard-confidence-dry-run setup-warehouse-databricks` | Databricks warehouse setup | +| `/onboard-confidence-dry-run setup-warehouse-redshift` | Redshift warehouse setup | + +--- + +## Dry Run Rules + +1. **Show the exact same UX** as the real skill — same step trackers, same questions, same confirmations, same tone +2. **Display `[DRY RUN]` prefix** on every status update so the user knows it's simulated +3. **Simulate API responses** — don't make real HTTP calls. Instead, print what would happen and show mock response data +4. **Still ask the user for input** at every step (workspace name, flag name, warehouse type, etc.) — the point is to test the interaction flow +5. **Skip browser login** — use a mock token with mock claims: + ``` + Mock token claims: + - account_name: accounts/dry-run-demo + - region: EU + - org_id: org_DryRunDemo123 + - identity: identities/udryrun123 + ``` +6. **Use realistic mock data** for API responses. Examples are listed in the Mock Data Reference section below. +7. **For warehouse-specific dry runs**, simulate the full flow including: + - Snowflake: mock crypto key creation, show the ALTER USER SQL that would be generated + - Databricks: mock S3 bucket creation, IAM role, show the trust policy that would be created + - Redshift: mock cluster creation, show the dual trust policy, GRANT statements + - BigQuery: mock gcloud commands that would run +8. **At the end of each dry run**, show the dry-run summary banner (see Dry Run Summary section) +9. **No sandbox overrides** — since no real network calls are made, the skill never needs `dangerouslyDisableSandbox: true` +10. **No token persistence** — never write anything to `~/.confidence/` or `$TMPDIR` + +--- + +## Mock Data Reference + +Use these mock responses when simulating API calls. Substitute user-provided values (workspace name, flag name, etc.) where indicated with ``. + +### Authentication mock + +Skip all browser-based Auth0 login. Instead, tell the user: + +> [DRY RUN] Skipping browser login — using mock credentials. + +Mock token claims to use throughout the session: + +``` +account_name: accounts/dry-run-demo +region: EU +org_id: org_DryRunDemo123 +identity: identities/udryrun123 +email: dryrun@example.com +``` + +Region derived from mock token: `eu` (lowercase). All mock API URLs use `eu` prefix (e.g., `iam.eu.confidence.dev`). + +### Create account response + +```json +{ + "name": "accounts/dry-run-demo", + "externalId": "org_DryRunDemo123", + "loginId": "", + "displayName": "" +} +``` + +### Check login ID availability + +```json +{"available": true, "message": ""} +``` + +If the user enters `taken-name` (for testing), return: +```json +{"available": false, "message": "This workspace name is already in use."} +``` + +### Create client response + +```json +{ + "name": "clients/dry-run-client", + "displayName": "" +} +``` + +### Client secret (mock) + +``` +dryrn_sk_mock1234567890abcdef +``` + +### Create flag response + +```json +{ + "name": "flags/", + "schema": {} +} +``` + +### Update flag schema response + +```json +{ + "name": "flags/", + "schema": { + "schema": { + "enabled": {"boolSchema": {}} + } + } +} +``` + +### Create variant response + +```json +{ + "name": "flags//variants/", + "value": {"enabled": true} +} +``` + +### Add flag to client response + +```json +{ + "name": "flags/", + "clients": ["clients/dry-run-client"] +} +``` + +### Create segment response + +```json +{ + "name": "segments/everyone", + "displayName": "Everyone", + "allocation": {"proportion": {"value": "1"}} +} +``` + +### Create rule response + +```json +{ + "name": "flags//rules/rule1", + "segment": "segments/everyone", + "enabled": true +} +``` + +### Resolve response + +```json +{ + "resolvedFlags": [ + { + "flag": "flags/", + "variant": "flags//variants/", + "value": {"enabled": true}, + "reason": "RESOLVE_REASON_MATCH" + } + ] +} +``` + +### User invitation response + +```json +{ + "name": "userInvitations/dry-run-inv-001", + "invitedEmail": "", + "inviter": "Dry Run Admin", + "expirationTime": "2026-06-17T10:00:00Z", + "invitationUri": "https://confidence.spotify.com/invite/mock-token", + "invitationToken": "mock-invitation-token" +} +``` + +### Current user response + +```json +{ + "user": { + "name": "users/dry-run-user", + "fullName": "Dry Run User", + "email": "dryrun@example.com" + }, + "accountMemberships": [ + { + "account": "accounts/dry-run-demo", + "displayName": "Dry Run Demo", + "loginId": "dry-run-demo", + "region": "EU" + } + ], + "account": "accounts/dry-run-demo", + "identity": { + "name": "identities/udryrun123", + "displayName": "Dry Run User" + } +} +``` + +### Validate warehouse (BigQuery) + +```json +{ + "validation": [ + {"key": "SERVICE_ACCOUNT", "description": "Service account access", "success": true}, + {"key": "PERMISSIONS", "description": "BigQuery permissions", "success": true}, + {"key": "DATASET", "description": "Dataset access", "success": true} + ], + "successful": true +} +``` + +### Validate warehouse (Snowflake) + +```json +{ + "validation": [ + {"key": "AUTHENTICATION", "description": "Key-pair authentication", "success": true}, + {"key": "ROLE", "description": "Role access", "success": true}, + {"key": "WAREHOUSE", "description": "Warehouse access", "success": true}, + {"key": "DATABASE", "description": "Database access", "success": true}, + {"key": "SCHEMA", "description": "Schema access", "success": true} + ], + "successful": true +} +``` + +### Validate warehouse (Redshift) + +```json +{ + "validation": [ + {"key": "CLUSTER", "description": "Cluster connectivity", "success": true}, + {"key": "IAM_ROLE", "description": "IAM role assumption", "success": true}, + {"key": "SCHEMA", "description": "Schema access", "success": true} + ], + "successful": true +} +``` + +### Create warehouse + +```json +{"name": "dataWarehouses/dry-run-wh-123"} +``` + +### Create crypto key (Snowflake) + +```json +{ + "name": "cryptoKeys/snowflake-key", + "kind": "SNOWFLAKE", + "publicKey": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0mock1234567890abcd\nefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ01234567\n89mockkeydatafordryruntestingpurposes0123456789abcdefghijklmnopqrst\nuvwxyz==\n-----END PUBLIC KEY-----" +} +``` + +### Create flag applied connection + +```json +{"name": "flagAppliedConnections/dry-run-connector", "state": "STATE_RUNNING"} +``` + +### Create event connection + +```json +{"name": "eventConnections/dry-run-events", "state": "STATE_RUNNING"} +``` + +### Create assignment table + +```json +{"name": "assignmentTables/dry-run-assignments", "displayName": "Assignments"} +``` + +### Verify pipeline — assignments + +``` +targeting_key | rule | assignment_id | assignment_time +dry-run-user | flags/my-test-flag/rules/rule1 | on | 2026-06-10T12:00:00Z +``` + +### Verify pipeline — events + +``` +_event_time | user_action | page +2026-06-10T12:00:00Z | clicked_button | homepage +``` + +### Publish events response + +```json +{"errors": []} +``` + +### Learning progress response + +```json +{ + "courseProgresses": [ + {"course": "courses/STATS", "completedLessons": 0, "totalLessons": 5}, + {"course": "courses/DESIGN", "completedLessons": 0, "totalLessons": 5}, + {"course": "courses/FLAGS", "completedLessons": 0, "totalLessons": 5}, + {"course": "courses/METRICS", "completedLessons": 0, "totalLessons": 5}, + {"course": "courses/COORDINATION", "completedLessons": 0, "totalLessons": 5} + ], + "completedCourses": 0 +} +``` + +--- + +## Dry Run Summary + +At the end of every sub-command dry run, display this banner: + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Complete +═══════════════════════════════════════════════════════════════ + + This was a simulated run. No real resources were created. + + To run for real: + • /onboard-confidence + • /onboard-confidence:setup-warehouse- + +═══════════════════════════════════════════════════════════════ +``` + +--- + +## Sub-command: create-account + +### Step Tracker + +Display at START and after EACH step completes (updating status). Prefix the title with `[DRY RUN]`. + +``` +───── [DRY RUN] Create Account ─────────────────────────────── + [1] Log in ○ pending + [2] Workspace name ○ pending + [3] Account details ○ pending + [4] Create account ○ pending + [5] Connect tools ○ pending + [6] Done ○ pending +────────────────────────────────────────────────────────────── +``` + +Use `●` for completed, `▶` for in-progress, `○` for pending. + +### Step 1: Log in + +**Skip browser login entirely.** Display: + +> [DRY RUN] Skipping browser login — using mock credentials. +> [DRY RUN] Authenticated as dryrun@example.com + +Mark step 1 as `●`. + +### Step 2: Workspace name + +Same UX as the real skill. EDUCATE then ASK: + +> Your workspace name is the unique identifier for your Confidence account. +> It appears in URLs and is used to log in. +> +> **Rules:** 3-21 characters, lowercase letters, digits, and hyphens. Must start with a letter and end with a letter or digit. + +**Wait for user input.** Validate locally against regex `^[a-z][a-z0-9-]{1,19}[a-z0-9]$`. If invalid, explain and re-ask — exactly like the real skill. + +Then simulate the availability check: + +> [DRY RUN] Checking availability... `` is available! + +If the user enters `taken-name`, simulate: + +> [DRY RUN] Checking availability... `taken-name` is already taken. Try another name. + +### Step 3: Account details + +Same UX as the real skill. Collect interactively, one field at a time: + +1. **Display name** — ask, validate (3-21 chars, starts with letter). + +2. **Region** — present as a choice: + > Where should your data be stored? This **cannot be changed later**. + > 1. EU (Europe) + > 2. US (United States) + +3. **Authentication method** — present as a choice: + > How should users log in to your workspace? + > 1. Google + > 2. Email + password + > 3. Both + +4. **Admin email** — ask. Validate work email. If free email (gmail, yahoo, etc.), reject: + > Confidence requires a work email address. Free providers like Gmail aren't allowed. + +5. **Allowed login email domains** — optional. Ask if they want to restrict. + +### Step 4: Create account + +Display what would happen: + +> [DRY RUN] Would call `POST https://onboarding.confidence.dev/v1/accounts` +> [DRY RUN] Creating workspace ****... + +Then show mock success: + +> [DRY RUN] Your workspace **** has been created! +> Workspace ID: `` +> Region: +> +> You can access it at: https://confidence.spotify.com + +### Step 5: Connect tools + +> [DRY RUN] Would re-authenticate with org-scoped token (browser auto-redirect). +> [DRY RUN] Skipping — mock token already has org scope. + +Then suggest MCP: + +> To connect Confidence tools for flag management, type `/mcp` and authenticate **confidence-flags**. + +### Step 6: Done + +Show the same summary as the real skill, but with `[DRY RUN]` in the banner: + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Welcome to Confidence! +═══════════════════════════════════════════════════════════════ + + Workspace: () + Region: + Admin: + URL: https://confidence.spotify.com + + Next steps: + • Invite team members: /onboard-confidence invite-user + • Create a feature flag: Ask me to create a flag, or use + the Confidence UI + • Integrate your app: Ask me for SDK setup instructions + +═══════════════════════════════════════════════════════════════ +``` + +Then show the Dry Run Summary banner. + +--- + +## Sub-command: invite-user + +### Step Tracker + +``` +───── [DRY RUN] Invite User ────────────────────────────────── + [1] Authenticate ○ pending + [2] Target account ○ pending + [3] Invitation details ○ pending + [4] Send invitation ○ pending +────────────────────────────────────────────────────────────── +``` + +### Step 1: Authenticate + +> [DRY RUN] Skipping browser login — using mock credentials. +> [DRY RUN] Authenticated as dryrun@example.com + +> [DRY RUN] Would call `GET https://iam.eu.confidence.dev/v1/currentUser` +> [DRY RUN] Current user: Dry Run User (dryrun@example.com) + +### Step 2: Target account + +> [DRY RUN] Account: **Dry Run Demo** (dry-run-demo) + +### Step 3: Invitation details + +Same UX as the real skill. Ask for: + +1. **Email address(es)** — required. Accept single or comma-separated. Validate format locally. +2. **Send invitation email?** — default yes. + +### Step 4: Send invitation + +For each email: + +> [DRY RUN] Would call `POST https://iam.eu.confidence.dev/v1/userInvitations` + +For single invite: +> [DRY RUN] Invitation sent to ****! +> They'll receive an email with instructions to join. +> The invitation expires on Jun 17, 2026. + +For batch invites, show a summary table: +``` +[DRY RUN] Invitations sent: + ✓ alice@example.com — expires Jun 17 + ✓ bob@example.com — expires Jun 17 +``` + +If any email fails local validation: +``` + ✗ charlie@invalid — invalid email address +``` + +Then show the Dry Run Summary banner. + +--- + +## Sub-command: create-client + +### Step Tracker + +``` +───── [DRY RUN] Create Client ──────────────────────────────── + [1] Client name ○ pending + [2] Create client ○ pending + [3] Get credentials ○ pending +────────────────────────────────────────────────────────────── +``` + +### Step 1: Client name + +Same UX as the real skill: + +> What should we call this client? (e.g., "iOS App", "Web Frontend", "Backend Service") + +Wait for user input. + +### Step 2: Create client + +> [DRY RUN] Would call `POST https://iam.eu.confidence.dev/v1/clients` +> [DRY RUN] Client **** created. + +### Step 3: Get credentials + +> [DRY RUN] Would call `POST https://iam.eu.confidence.dev/v1/clients/dry-run-client/credentials` + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Client Created +═══════════════════════════════════════════════════════════════ + + Name: + Secret: dryrn_sk_mock1234567890abcdef + + Use this secret in your SDK configuration to resolve flags. + Keep it safe — you can regenerate it, but the old one will + stop working. + + Next: Ask me for SDK integration instructions, or run + /onboard-confidence setup-wizard + +═══════════════════════════════════════════════════════════════ +``` + +Then show the Dry Run Summary banner. + +--- + +## Sub-command: setup-wizard + +### Step Tracker + +``` +───── [DRY RUN] Setup Wizard ───────────────────────────────── + [1] Create client ○ pending + [2] Create flag ○ pending + [3] Add variants ○ pending + [4] Add targeting ○ pending + [5] Test resolve ○ pending + [6] Done ○ pending +────────────────────────────────────────────────────────────── +``` + +### Prerequisites + +> [DRY RUN] Skipping browser login — using mock credentials. +> [DRY RUN] Region: EU (from mock token) + +### Step 1: Create client + +> [DRY RUN] Would call `GET https://iam.eu.confidence.dev/v1/clients` +> [DRY RUN] No existing clients found. Creating one now. + +Ask user for client name (same UX as real skill). + +> [DRY RUN] Would call `POST https://iam.eu.confidence.dev/v1/clients` +> [DRY RUN] Client **** created. +> [DRY RUN] Would call `POST https://iam.eu.confidence.dev/v1/clients/dry-run-client/credentials` +> [DRY RUN] Client secret generated. + +### Step 2: Create flag + +Same EDUCATE then ASK flow: + +> A feature flag controls a piece of functionality. Let's create your first one. +> What should it be called? (e.g., "new-checkout-flow", "dark-mode") + +Wait for user input. Validate: 4-63 chars, `[a-z0-9-]`. + +> [DRY RUN] Would call `POST https://flags.eu.confidence.dev/v1/flags?flag_id=` +> [DRY RUN] Flag **** created. + +### Step 3: Add variants + +Same EDUCATE flow: + +> Variants are the different values a flag can have. For a simple on/off flag, you'd have "on" and "off" variants. +> +> What variants should this flag have? +> 1. Simple on/off (boolean) +> 2. Custom variants (I'll name them) + +Wait for user input. + +> [DRY RUN] Would call `PATCH https://flags.eu.confidence.dev/v1/flags/` (set schema) +> [DRY RUN] Schema set. + +For each variant: +> [DRY RUN] Would call `POST https://flags.eu.confidence.dev/v1/flags//variants` +> [DRY RUN] Variant **** created with value ``. + +After all variants: +> [DRY RUN] Would call `POST https://flags.eu.confidence.dev/v1/flags/:addFlagClient` +> [DRY RUN] Flag attached to client. + +### Step 4: Add targeting + +Same EDUCATE flow: + +> Targeting rules control who sees which variant. Let's set a default — you can add more rules later. +> Which variant should be the default? + +Wait for user input. + +> [DRY RUN] Would call `POST https://flags.eu.confidence.dev/v1/segments?segment_id=everyone` +> [DRY RUN] Segment "Everyone" created. +> [DRY RUN] Would call `PATCH https://flags.eu.confidence.dev/v1/segments/everyone` (set allocation) +> [DRY RUN] Would call `POST https://flags.eu.confidence.dev/v1/segments/everyone:allocate` +> [DRY RUN] Segment allocated at 100%. +> [DRY RUN] Would call `POST https://flags.eu.confidence.dev/v1/flags//rules` +> [DRY RUN] Rule created — all users get variant ****. + +### Step 5: Test resolve + +> Let's verify the flag works by resolving it. + +> [DRY RUN] Would call `POST https://resolver.eu.confidence.dev/v1/flags:resolve` +> [DRY RUN] Flag **** resolved to variant **** — it works! + +Show mock resolve response: +``` +[DRY RUN] Mock resolve result: + Flag: + Variant: + Value: {"enabled": true} + Reason: RESOLVE_REASON_MATCH +``` + +### Step 6: Done + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Setup Complete! +═══════════════════════════════════════════════════════════════ + + Client: + Secret: dryrn_sk_mock1234567890abcdef + Flag: + Variants: + Default: + + Your flag is live and resolving. Next steps: + • Integrate the SDK: Ask me for setup instructions + • Create more flags: Ask me or use the Confidence UI + • Set up experiments: /onboard-confidence learn + +═══════════════════════════════════════════════════════════════ +``` + +Then show the Dry Run Summary banner. + +--- + +## Sub-command: setup-warehouse + +### Flow + +Show the 4 warehouse options (same as real skill): + +> Which data warehouse do you use? +> 1. BigQuery +> 2. Snowflake +> 3. Databricks +> 4. Redshift + +After the user picks, run the corresponding warehouse-specific dry run below. + +--- + +## Sub-command: setup-warehouse-bigquery + +### Step Tracker + +``` +───── [DRY RUN] Setup Warehouse (BigQuery) ─────────────────── + [1] Choose warehouse ● done + [2] GCP project ID ○ pending + [3] Dataset name ○ pending + [4] Service account ○ pending + [5] Validate & fix ○ pending + [6] Create warehouse ○ pending + [7] Create connectors ○ pending + [8] Assignment table ○ pending + [9] Verify pipeline ○ pending + [10] Done ○ pending +────────────────────────────────────────────────────────────── +``` + +### Step 1: Choose warehouse (already done) + +Mark as `●`. + +### Step 2: GCP Project ID + +Same UX as real skill: + +> What's your GCP project ID? Go to **Google Cloud Console** (console.cloud.google.com). Your project ID is shown in the top bar next to "Google Cloud". It looks like `my-company-prod` or `project-12345`. + +Wait for user input. + +### Step 3: Dataset name + +Same UX: + +> A dataset is like a folder in BigQuery where Confidence stores its tables. The default is `confidence`. +> If you don't have one yet, I can create it for you via `bq mk`. + +Wait for user input (or accept default). + +### Step 4: Service account + +Same UX: + +> A service account is a robot account that Confidence uses to write data to your BigQuery dataset. +> Go to **Google Cloud Console -> IAM & Admin -> Service Accounts**. Create one (e.g., `confidence-connector@.iam.gserviceaccount.com`) or pick an existing one. +> It needs **BigQuery Data Editor** and **BigQuery Job User** roles. + +Wait for user input. + +### Step 5: Validate & fix + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/dataWarehouseConfig:validate` +> [DRY RUN] Validation passed! All checks succeeded: +> - SERVICE_ACCOUNT: Service account access ✓ +> - PERMISSIONS: BigQuery permissions ✓ +> - DATASET: Dataset access ✓ + +Then show what gcloud commands would have been run if validation had failed: + +> [DRY RUN] If validation had failed, these commands would fix it: +> ``` +> # Grant Confidence access to your service account +> gcloud iam service-accounts add-iam-policy-binding \ +> --project= \ +> --member="serviceAccount:account-dry-run-demo@spotify-confidence.iam.gserviceaccount.com" \ +> --role="roles/iam.workloadIdentityUser" +> +> # Grant BigQuery Job User +> gcloud projects add-iam-policy-binding \ +> --member="serviceAccount:" \ +> --role="roles/bigquery.jobUser" +> ``` + +### Step 6: Create warehouse + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/dataWarehouses` +> [DRY RUN] Warehouse created: `dataWarehouses/dry-run-wh-123` + +### Step 7: Create connectors + +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/flagAppliedConnections` +> [DRY RUN] Flag assignment connector created: `flagAppliedConnections/dry-run-connector` (STATE_RUNNING) +> +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/eventConnections` +> [DRY RUN] Event connector created: `eventConnections/dry-run-events` (STATE_RUNNING) + +### Step 8: Assignment table + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/assignmentTables` +> [DRY RUN] Assignment table created: `assignmentTables/dry-run-assignments` + +Show the SQL that would be used: +```sql +SELECT targeting_key, rule, assignment_id, assignment_time +FROM `..assignments` +``` + +### Step 9: Verify pipeline + +> [DRY RUN] Would resolve a flag and publish test events to verify data flow. + +Show mock pipeline results: + +``` +[DRY RUN] Pipeline verification: + ● Assignments: 1 row — data flowing + dry-run-user -> on (2026-06-10T12:00:00Z) + ● Events: 1 row — data flowing + clicked_button on homepage (2026-06-10T12:00:00Z) +``` + +### Step 10: Done + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: BigQuery () + Dataset: + Connectors: + ● Flag assignments -> assignments table (verified) + ● Events -> events_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + +═══════════════════════════════════════════════════════════════ +``` + +Then show the Dry Run Summary banner. + +--- + +## Sub-command: setup-warehouse-snowflake + +### Step Tracker + +``` +───── [DRY RUN] Setup Warehouse (Snowflake) ────────────────── + [1] Choose warehouse ● done + [2] Account & user ○ pending + [3] Role & warehouse ○ pending + [4] Database & schema ○ pending + [5] Create crypto key ○ pending + [6] Register key in SF ○ pending + [7] Validate ○ pending + [8] Create warehouse ○ pending + [9] Create connectors ○ pending + [10] Assignment table ○ pending + [11] Verify pipeline ○ pending + [12] Done ○ pending +────────────────────────────────────────────────────────────── +``` + +### Step 2: Account & user + +Same UX as real skill. Ask for: +- **Account** — Snowflake account identifier (e.g., `zlvpqre-wr49874`) +- **User** — Snowflake user for Confidence to connect as + +### Step 3: Role & warehouse + +- **Role** — default `ACCOUNTADMIN` +- **Warehouse** — default `COMPUTE_WH` + +### Step 4: Database & schema + +- **Exposure database** — default `CONFIDENCE` +- **Exposure schema** — default `EXPOSURE` + +Show the SQL that would be needed if the database/schema don't exist: +```sql +CREATE DATABASE IF NOT EXISTS ; +CREATE SCHEMA IF NOT EXISTS .; +GRANT USAGE ON DATABASE TO ROLE ; +GRANT ALL ON SCHEMA . TO ROLE ; +``` + +### Step 5: Create crypto key + +> [DRY RUN] Would call `POST https://iam.eu.confidence.dev/v1/cryptoKeys?crypto_key_id=snowflake-key` +> [DRY RUN] Crypto key created: `cryptoKeys/snowflake-key` +> [DRY RUN] Public key generated (mock RSA 2048-bit) + +### Step 6: Register key in Snowflake + +Show the ALTER USER SQL that would be generated: + +> [DRY RUN] In the real flow, this SQL would be copied to your clipboard: +> ```sql +> ALTER USER SET RSA_PUBLIC_KEY='MIIBIjANBgkqhkiG9w0BAQE...mockkey...'; +> ``` + +Ask: "Does another Confidence account share this Snowflake user?" (same as real skill). If yes, show `RSA_PUBLIC_KEY_2` variant. + +### Step 7: Validate + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/dataWarehouseConfig:validate` +> [DRY RUN] Validation passed! All checks succeeded: +> - AUTHENTICATION: Key-pair authentication ✓ +> - ROLE: Role access ✓ +> - WAREHOUSE: Warehouse access ✓ +> - DATABASE: Database access ✓ +> - SCHEMA: Schema access ✓ + +### Step 8: Create warehouse + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/dataWarehouses` +> [DRY RUN] Warehouse created: `dataWarehouses/dry-run-wh-123` + +### Step 9: Create connectors + +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/flagAppliedConnections` +> [DRY RUN] Flag assignment connector created (Snowflake -> ..ASSIGNMENTS) +> +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/eventConnections` +> [DRY RUN] Event connector created (Snowflake -> ..EVENTS_*) + +### Step 10: Assignment table + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/assignmentTables` +> [DRY RUN] Assignment table created. + +Show the SQL: +```sql +SELECT targeting_key, rule, assignment_id, assignment_time +FROM ..ASSIGNMENTS +``` + +### Step 11: Verify pipeline + +Show mock pipeline results: + +``` +[DRY RUN] Pipeline verification: + ● Assignments: 1 row — data flowing + dry-run-user -> on (2026-06-10T12:00:00Z) + ● Events: 1 row — data flowing + clicked_button on homepage (2026-06-10T12:00:00Z) +``` + +### Step 12: Done + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: Snowflake () + Database: + Schema: + Connectors: + ● Flag assignments -> ASSIGNMENTS table (verified) + ● Events -> EVENTS_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + +═══════════════════════════════════════════════════════════════ +``` + +Then show the Dry Run Summary banner. + +--- + +## Sub-command: setup-warehouse-databricks + +### Step Tracker + +``` +───── [DRY RUN] Setup Warehouse (Databricks) ───────────────── + [1] Choose warehouse ● done + [2] Workspace URL ○ pending + [3] SQL Warehouse ID ○ pending + [4] Service principal ○ pending + [5] AWS account & CLI ○ pending + [6] S3 bucket ○ pending + [7] IAM role ○ pending + [8] Databricks schema ○ pending + [9] Create warehouse ○ pending + [10] Create connectors ○ pending + [11] Assignment table ○ pending + [12] Verify pipeline ○ pending + [13] Done ○ pending +────────────────────────────────────────────────────────────── +``` + +### Overview + +Same overview as real skill: + +> Setting up Databricks with Confidence requires three things: +> +> 1. **A Databricks workspace** — you need admin access to create a service principal (a robot account) +> 2. **An AWS account with an S3 bucket** — Confidence needs this as a staging area for loading data into Databricks. This is required even if your Databricks runs on GCP or Azure +> 3. **A schema in Databricks** — a place for Confidence to create tables (e.g., `confidence`) +> +> **How data flows:** +> Confidence collects your flag assignments and events internally, then writes parquet files to an S3 bucket you provide, and finally loads them into Databricks tables. This happens in batches every ~5 minutes. + +### Step 2: Workspace URL + +Same UX: ask for URL, extract hostname, confirm. + +### Step 3: SQL Warehouse ID + +Same UX: explain how to find it, ask for the ID. + +### Step 4: Service principal + +Same UX: explain how to create one, ask for Client ID and Secret. + +For dry run, accept any values. Display: +> [DRY RUN] Service principal configured (mock credentials accepted). + +### Step 5: AWS account & CLI + +Same choice: +> Do you have the `aws` CLI set up, or would you prefer manual steps? +> 1. Set it up for me (requires `aws` CLI) +> 2. Show me the steps + +> [DRY RUN] Skipping AWS CLI check — mock mode. + +### Step 6: S3 bucket + +Ask for bucket name (suggest `confidence-staging-dry-run-demo`) and region. + +> [DRY RUN] Would run: `aws s3api create-bucket --bucket --region ` +> [DRY RUN] S3 bucket `` created in ``. + +### Step 7: IAM role + +Show the trust policy that would be created: + +> [DRY RUN] Would create IAM role with this trust policy: +> ```json +> { +> "Version": "2012-10-17", +> "Statement": [{ +> "Effect": "Allow", +> "Principal": {"Federated": "accounts.google.com"}, +> "Action": "sts:AssumeRoleWithWebIdentity", +> "Condition": { +> "StringEquals": { +> "accounts.google.com:sub": "123456789012345678901" +> } +> } +> }] +> } +> ``` +> +> [DRY RUN] Would create S3 access policy scoped to ``. +> [DRY RUN] IAM role created: `arn:aws:iam::123456789012:role/confidence-databricks-staging` + +### Step 8: Databricks schema + +Same UX: ask for schema name (default `confidence`). + +Show the SQL that would need to be run: + +> [DRY RUN] In the real flow, this SQL would be copied to your clipboard: +> ```sql +> CREATE SCHEMA IF NOT EXISTS confidence; +> GRANT USE SCHEMA, CREATE TABLE ON SCHEMA confidence TO ``; +> ``` + +### Step 9: Create warehouse + +> [DRY RUN] Note: Pre-validation is not available for Databricks. +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/dataWarehouses` +> [DRY RUN] Warehouse created: `dataWarehouses/dry-run-wh-123` + +### Step 10: Create connectors + +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/flagAppliedConnections` +> [DRY RUN] Flag assignment connector created (Databricks -> .assignments, S3 staging: ) +> +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/eventConnections` +> [DRY RUN] Event connector created (Databricks -> .events_*, S3 staging: ) + +### Step 11: Assignment table + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/assignmentTables` +> [DRY RUN] Assignment table created. + +Show the SQL: +```sql +SELECT targeting_key, rule, assignment_id, assignment_time +FROM .assignments +``` + +### Step 12: Verify pipeline + +``` +[DRY RUN] Pipeline verification: + ● Assignments: 1 row — data flowing + dry-run-user -> on (2026-06-10T12:00:00Z) + ● Events: 1 row — data flowing + clicked_button on homepage (2026-06-10T12:00:00Z) +``` + +### Step 13: Done + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: Databricks () + Schema: + S3 Bucket: () + Connectors: + ● Flag assignments -> assignments table (verified) + ● Events -> events_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + + Note: Data is delivered in ~5 minute batches. + +═══════════════════════════════════════════════════════════════ +``` + +Then show the Dry Run Summary banner. + +--- + +## Sub-command: setup-warehouse-redshift + +### Step Tracker + +``` +───── [DRY RUN] Setup Warehouse (Redshift) ─────────────────── + [1] Choose warehouse ● done + [2] AWS account & CLI ○ pending + [3] Redshift cluster ○ pending + [4] S3 bucket ○ pending + [5] IAM role ○ pending + [6] Attach role ○ pending + [7] Schema & grants ○ pending + [8] Validate ○ pending + [9] Create warehouse ○ pending + [10] Create connectors ○ pending + [11] Assignment table ○ pending + [12] Verify pipeline ○ pending + [13] Done ○ pending +────────────────────────────────────────────────────────────── +``` + +### Overview + +Same overview as real skill: + +> Setting up Redshift with Confidence requires an **AWS account**. Here's what we'll set up: +> +> 1. **A Redshift cluster** — a data warehouse that stores your experiment data +> 2. **An S3 bucket** — a staging area where Confidence drops data files before loading them into Redshift +> 3. **An IAM role** — permissions that let Confidence write to S3 and load into Redshift +> 4. **A schema** — a folder inside Redshift where Confidence creates its tables +> +> **Important: Redshift Serverless won't work** — Confidence needs a provisioned cluster. + +### Step 2: AWS account & CLI + +Same choice as real skill. In dry run: + +> [DRY RUN] Skipping AWS CLI check — mock mode. + +### Step 3: Redshift cluster + +Ask if they have one or want to create one. Same UX as real skill. + +If creating: +> [DRY RUN] Would run: +> ``` +> aws redshift create-cluster \ +> --cluster-identifier confidence-redshift-dry-run-demo \ +> --cluster-type single-node \ +> --node-type ra3.large \ +> --master-username admin \ +> --master-user-password \ +> --db-name dev \ +> --region eu-west-1 \ +> --publicly-accessible +> ``` +> [DRY RUN] Cluster `confidence-redshift-dry-run-demo` is running. + +If using existing, ask for cluster name. + +### Step 4: S3 bucket + +Ask for bucket name and region (same UX). + +> [DRY RUN] Would run: `aws s3api create-bucket --bucket --region ` +> [DRY RUN] S3 bucket `` created in ``. + +### Step 5: IAM role + +Show the dual trust policy that would be created: + +> [DRY RUN] Would create IAM role with dual trust policy: +> ```json +> { +> "Version": "2012-10-17", +> "Statement": [ +> { +> "Effect": "Allow", +> "Principal": {"Federated": "accounts.google.com"}, +> "Action": "sts:AssumeRoleWithWebIdentity", +> "Condition": { +> "StringEquals": { +> "accounts.google.com:sub": "123456789012345678901" +> } +> } +> }, +> { +> "Effect": "Allow", +> "Principal": {"Service": "redshift.amazonaws.com"}, +> "Action": "sts:AssumeRole" +> } +> ] +> } +> ``` +> +> [DRY RUN] Would create S3 access policy scoped to ``. +> [DRY RUN] Would create Redshift Data API policy. +> [DRY RUN] IAM role created: `arn:aws:iam::123456789012:role/confidence-redshift` + +### Step 6: Attach role + +> [DRY RUN] Would run: +> ``` +> aws redshift modify-cluster-iam-roles \ +> --cluster-identifier \ +> --add-iam-roles arn:aws:iam::123456789012:role/confidence-redshift +> ``` +> [DRY RUN] IAM role attached to cluster. + +### Step 7: Schema & grants + +Ask for schema name (default `confidence`). + +Show the GRANT statements: + +> [DRY RUN] In the real flow, these would be run against Redshift: +> ```sql +> CREATE SCHEMA IF NOT EXISTS ; +> GRANT USAGE ON SCHEMA TO PUBLIC; +> GRANT CREATE ON SCHEMA TO PUBLIC; +> ``` + +### Step 8: Validate + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/dataWarehouseConfig:validate` +> [DRY RUN] Validation passed! All checks succeeded: +> - CLUSTER: Cluster connectivity ✓ +> - IAM_ROLE: IAM role assumption ✓ +> - SCHEMA: Schema access ✓ + +### Step 9: Create warehouse + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/dataWarehouses` +> [DRY RUN] Warehouse created: `dataWarehouses/dry-run-wh-123` + +### Step 10: Create connectors + +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/flagAppliedConnections` +> [DRY RUN] Flag assignment connector created (Redshift -> .assignments, S3 staging: ) +> +> [DRY RUN] Would call `POST https://connectors.eu.confidence.dev/v1/eventConnections` +> [DRY RUN] Event connector created (Redshift -> .events_*, S3 staging: ) + +### Step 11: Assignment table + +> [DRY RUN] Would call `POST https://metrics.eu.confidence.dev/v1/assignmentTables` +> [DRY RUN] Assignment table created. + +Show the SQL: +```sql +SELECT targeting_key, rule, assignment_id, assignment_time +FROM .assignments +``` + +### Step 12: Verify pipeline + +``` +[DRY RUN] Pipeline verification: + ● Assignments: 1 row — data flowing + dry-run-user -> on (2026-06-10T12:00:00Z) + ● Events: 1 row — data flowing + clicked_button on homepage (2026-06-10T12:00:00Z) +``` + +### Step 13: Done + +``` +═══════════════════════════════════════════════════════════════ + [DRY RUN] Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: Redshift () + Database: + Schema: + S3 Bucket: () + Connectors: + ● Flag assignments -> assignments table (verified) + ● Events -> events_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + +═══════════════════════════════════════════════════════════════ +``` + +Then show the Dry Run Summary banner. + +--- + +## User-Facing Communication Rules + +Follow the same rules as the real onboarding skill: + +- **NEVER expose internal technical details** — but since this is a dry run, you DO show the mock API endpoints being "called" and mock response data. This is the point of the dry run. +- **DO show `[DRY RUN]` prefix** on every simulated action +- **DO show human-readable status updates** alongside the mock data +- **Step Tracker:** Display the step tracker at every phase transition, with `[DRY RUN]` in the title. Update it after each step completes. +- **Be conversational** — same tone as the real skill +- **Ask for real input** — workspace names, flag names, warehouse config values, etc. The user should experience the full interaction flow. + +--- + +## Important: What NOT to do + +- **Do NOT make any real HTTP calls** — no `curl`, no `open`, no `python3` auth scripts +- **Do NOT write files to disk** — no `$TMPDIR/confidence_auth.py`, no `~/.confidence/.auth_token` +- **Do NOT require `dangerouslyDisableSandbox: true`** — there are no external network calls +- **Do NOT use any MCP tools** — everything is simulated +- **Do NOT skip user input steps** — the entire point is to test the interaction flow diff --git a/skills/onboard-confidence/SKILL.md b/skills/onboard-confidence/SKILL.md new file mode 100644 index 0000000..12844f7 --- /dev/null +++ b/skills/onboard-confidence/SKILL.md @@ -0,0 +1,1307 @@ +--- +description: Create Confidence accounts and onboard users. Use when the user asks to create an account, invite users, onboard to Confidence, or check account status. +--- + +# Confidence Onboarding + +Create accounts, invite users, and get started with Confidence — all from the CLI. + +## Default behavior (no sub-command) + +When the user says "onboard me", "get started with Confidence", or triggers this skill without a specific sub-command, go **straight to the setup wizard**. The first question is always: + +> 1. **Create a new account** — I'll walk you through signup +> 2. **Sign in to an existing account** — I already have one + +Do NOT show a menu of sub-commands. Do NOT offer "Setup Wizard" as a choice — it IS the default flow. The only decision the user needs to make upfront is whether they have an account. + +## Commands + +| Command | Description | +|---------|-------------| +| `/onboard-confidence create-account` | Create a new Confidence account | +| `/onboard-confidence invite-user` | Invite a user to an account | +| `/onboard-confidence create-client` | Create an SDK client and generate credentials | +| `/onboard-confidence setup-wizard` | Guided walkthrough: client → flag → targeting → resolve | +| `/onboard-confidence setup-warehouse` | Configure data warehouse, connectors, and assignment tables | +| `/onboard-confidence learn` | Interactive learning about experimentation concepts | +| `/onboard-confidence status` | Check current user/account status | + +--- + +## Authentication + +**Browser-based Auth0 login.** The skill opens a browser for Auth0 login (Google, email/password, SSO) and captures the token automatically. The user never touches a token. + +### Auth0 Configuration (agent-internal) + +| Parameter | Signup (create-account) | Existing account (all other commands) | +|-----------|-------------------------|---------------------------------------| +| Domain | `auth.confidence.dev` | `auth.confidence.dev` | +| Client ID | `82qMvwZvqd3t3S0gRDvs8R53TehQXSJY` | `2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w` | +| Audience | `https://confidence.dev/` | `https://confidence.dev/` | +| Scope | `openid profile email offline_access` | `openid profile email offline_access` | + +### Auth script + +The auth script is **bundled in the plugin** as `auth.py` next to this SKILL.md. The path is shown in the "Base directory for this skill" header at the top of the loaded skill context. Do NOT write the script — just run it. + +**Usage — single Bash tool call** with `dangerouslyDisableSandbox: true` and `timeout: 130000`: +```bash +lsof -ti:8084 | xargs kill -9 2>/dev/null; python3 /auth.py [ORGANIZATION] +``` + +Replace `` with the actual path from the skill header (e.g., `/Users/.../confidence-ai-plugins/.claude/skills/onboard-confidence`). + +**Outputs on stdout** (parse line by line): +- `WAITING_FOR_LOGIN` — browser opened, waiting for callback +- `TOKEN:` — success, extract everything after `TOKEN:` +- `AUTH_ERROR:` — Auth0 returned an error +- `TOKEN_ERROR:` — token exchange failed + +**Examples:** + +Signup (no org): +```bash +lsof -ti:8084 | xargs kill -9 2>/dev/null; python3 /auth.py 82qMvwZvqd3t3S0gRDvs8R53TehQXSJY +``` + +Existing account login: +```bash +lsof -ti:8084 | xargs kill -9 2>/dev/null; python3 /auth.py 2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w org_abc123 +``` + +**Key details:** +- Port is fixed at **8084** (must match Auth0 Allowed Callback URLs) +- For signup (`create-account`): omit ORGANIZATION arg → adds `screen_hint=signup` + `prompt=login` +- For existing account (all other commands): pass `ORGANIZATION=` → auto-completes if browser session exists +- After `create-account`, automatically re-auth with org param to get org-scoped token (browser auto-redirects, no interaction) +- All network commands require `dangerouslyDisableSandbox: true` and `timeout: 130000` + +### Token management + +Tokens are persisted to `$TMPDIR/confidence_token` (and optionally `$TMPDIR/confidence_refresh_token`). This avoids re-exporting the JWT on every Bash tool call. **NEVER write tokens to `~/.confidence/` or anywhere outside `$TMPDIR`.** + +**CRITICAL: TMPDIR differs between sandboxed and non-sandboxed Bash calls.** Sandboxed calls use a path like `/tmp/claude-501/`, while `dangerouslyDisableSandbox: true` calls use the system TMPDIR (e.g., `/var/folders/.../T/`). If tokens are written in a sandboxed call but read in a non-sandboxed curl call, the curl will read a stale or missing token. **ALL token writes and reads MUST use `dangerouslyDisableSandbox: true`** to ensure a consistent TMPDIR path. This includes the auth script call (already non-sandboxed for network), the token save, the token validity check, and all curl calls. + +**After every successful auth**, write the token to file — **in the same `dangerouslyDisableSandbox: true` Bash call** as the auth script or curl that produced it: +```bash +# Parse TOKEN from auth.py stdout and persist (same Bash call, same TMPDIR) +echo "" > "$TMPDIR/confidence_token" +``` + +**On every sub-command start**, check if the token file exists and is not expired. **This Bash call MUST use `dangerouslyDisableSandbox: true`** so it reads from the same TMPDIR that curl will use: + +```bash +# dangerouslyDisableSandbox: true +python3 -c " +import json, base64, time, os +p = os.path.join(os.environ.get('TMPDIR', '/tmp'), 'confidence_token') +try: + t = open(p).read().strip() +except FileNotFoundError: + print('MISSING'); exit(0) +if not t: + print('MISSING'); exit(0) +parts = t.split('.')[1] +parts += '=' * (4 - len(parts) % 4) +d = json.loads(base64.b64decode(parts)) +if d.get('exp', 0) < time.time(): + print('EXPIRED'); exit(0) +print('VALID') +print('REGION=' + d.get('https://confidence.dev/region', 'EU')) +print('ORG=' + d.get('org_id', '')) +print('ACCOUNT=' + d.get('https://confidence.dev/account_name', '')) +" +``` + +Output is multi-line: first line is `VALID`/`EXPIRED`/`MISSING`, followed by `REGION=EU`, `ORG=...`, `ACCOUNT=...` if valid. + +If expired or missing, run the browser auth flow and write the new token to the file. + +**In curl calls**, read from the file instead of a shell variable: +```bash +curl -s ... -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" +``` + +Use the `REGION` value (lowercased) for URL prefixes: `iam.eu.confidence.dev`, `flags.eu.confidence.dev`, etc. + +### Important: gRPC-REST transcoding rules + +The Confidence APIs use gRPC with REST transcoding. The `body` field in the proto HTTP binding determines the JSON structure: + +- **`body: "client"`** → send the client object directly: `{"display_name": "iOS App"}` +- **`body: "flag"`** → send the flag object directly: `{}` +- **`body: "*"`** → send the full request message: `{"account": {...}, "billingDetails": {...}}` + +Fields NOT in the body (like `flag_id`, `parent`) become **query parameters**. + +**Field names are `snake_case`** in requests. Responses may use `camelCase`. + +### Speed: minimize tool calls + +**Every Bash tool call adds latency.** Optimize by combining commands: + +- **Prefer MCP over REST** for flag/client operations — one MCP tool call replaces 3-5 chained curls +- **Chain independent curls** with `&&` or `;` in a single Bash call when the results don't depend on each other +- **Token is in a file** — no need to export; just use `$(cat $TMPDIR/confidence_token)` in curl headers +- **Port kill + auth run**: Always combine: `lsof -ti:8084 | xargs kill -9 2>/dev/null; python3 ...` +- **Never use Write/Read tools** for temporary files — use Bash heredocs or bundled scripts + +### Common notes + +- All network commands require `dangerouslyDisableSandbox: true` +- Never show the token value to the user +- Always use region-specific URLs (e.g., `iam.eu.confidence.dev` not `iam.confidence.dev`) + +--- + +## User-Facing Communication Rules + +**NEVER expose internal technical details to the user.** + +- Do NOT show raw JSON request/response bodies in conversation +- Do NOT show Auth0 configuration details, token values, or OAuth internals +- Do NOT mention error codes, org IDs, JWT claims, token scoping, or API error details +- Do NOT ask the user for organization IDs, external IDs, or any auth-internal identifiers +- DO show human-readable status updates: "Opening browser for login...", "Creating your workspace...", "Invitation sent!" +- DO describe results in plain English +- DO handle all token re-issuance, org-scoping, and retry logic transparently — if something needs to happen behind the scenes (re-auth, polling, retry), just do it and show a friendly progress message +- The agent handles all auth/API complexity silently + +**Step Tracker:** Display a visual step tracker at every phase transition. Update and re-display it each time you move to a new step. + +**Use AskUserQuestion for all choices.** Present options as selectable items (up/down/enter) — never numbered lists in plain text. Only ask the user to type when collecting free-text input like names or emails. + +--- + +## Sub-command: create-account + +### Step Tracker + +Display at START and after EACH step completes (updating status): + +``` +───── Create Account ────────────────────────────────────── + [1] Log in ○ pending + [2] Workspace name ○ pending + [3] Account details ○ pending + [4] Create account ○ pending + [5] Connect tools ○ pending + [6] Done ○ pending +──────────────────────────────────────────────────────────── +``` + +Use `●` for completed, `▶` for in-progress, `○` for pending. + +### Step 1: Log in + +Run the bundled auth script with the **signup client ID** (`82qMvwZvqd3t3S0gRDvs8R53TehQXSJY`) and no organization arg. Parse the TOKEN and REFRESH_TOKEN from stdout. + +Tell the user: +> Opening your browser to log in. Sign up with Google or create an account with email and password. + +Write `TOKEN` to `$TMPDIR/confidence_token` and `REFRESH_TOKEN` to `$TMPDIR/confidence_refresh_token`. **The token save and all subsequent reads MUST use `dangerouslyDisableSandbox: true`** to ensure consistent TMPDIR paths (see Token management section). + +If login fails, show the error in plain English and offer to retry. + +**After successful login**, immediately extract the user's email by calling the Auth0 userinfo endpoint — **combine the token save and userinfo curl in a single `dangerouslyDisableSandbox: true` Bash call**: +```bash +echo "" > "$TMPDIR/confidence_token" && echo "" > "$TMPDIR/confidence_refresh_token" && curl -s "https://konfidens.eu.auth0.com/userinfo" -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" +``` +Response: `{ "email": "user@company.com", "name": "...", ... }` + +Store the `email` value as `SIGNUP_EMAIL`. This is used to: +- Derive workspace name suggestions in Step 2 +- Pre-fill the admin email in Step 3 + +### Step 2: Workspace name + +EDUCATE then ASK: + +> Your workspace name is the unique identifier for your Confidence account. +> It appears in URLs and is used to log in. +> +> **Rules:** 3-21 characters, lowercase letters, digits, and hyphens. Must start with a letter and end with a letter or digit. + +**Suggest names derived from `SIGNUP_EMAIL`.** Extract the local part (before `@`), strip `+` suffixes, and generate 2-3 suggestions. For example, if `SIGNUP_EMAIL` is `jane+test@acme.com`, suggest `jane`, `jane-acme`, `acme-jane`. + +Wait for user input. Then: + +1. **Validate locally** against regex `^[a-z][a-z0-9-]{1,19}[a-z0-9]$` +2. **Check availability:** +```bash +curl -s "https://onboarding.confidence.dev/v1/loginIdAvailability:check?login_id=${LOGIN_ID}" +``` +Response: `{ "available": true/false }` + +If taken, inform the user and suggest alternatives (append numbers, abbreviations). Re-ask. + +### Step 3: Account details + +Collect interactively, one field at a time: + +1. **Display name** — the human-readable name for the workspace (company name). + Validate: 3-32 characters, starts with a letter/digit, alphanumeric + Unicode letters + spaces + hyphens. + +2. **Region** — present as a choice: + > Where should your data be stored? This **cannot be changed later**. + > 1. EU (Europe) + > 2. US (United States) + +3. **Authentication method** — present as a choice: + > How should users log in to your workspace? + > 1. Google + > 2. Email + password + > 3. Both + +4. **Admin email** — the email of the first admin user. Must be a **work email** — free email providers (Gmail, Yahoo, etc.) are rejected by the API. + **Default to `SIGNUP_EMAIL`** (the email from Step 1). Present it as the pre-filled suggestion. Only ask the user to change it if they want a different admin email. + +5. **Allowed login email domains** — optional. Ask if they want to restrict login to a specific email domain (e.g., `@company.com`). + +### Step 4: Create account + +Build and send the request. Use `--max-time 120` to allow for slow gRPC provisioning: + +```bash +curl -s -w "\n%{http_code}" --max-time 120 -X POST "https://onboarding.confidence.dev/v1/accounts" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{ + "account": { + "displayName": "", + "loginId": "", + "region": "", + "authConnections": [], + "adminEmail": "", + "allowedLoginEmailDomains": [] + } + }' +``` + +**Auth connections format:** +- Google: `[{"googleAuthConnection": {}}]` +- Password: `[{"passwordAuthConnection": {}}]` +- Both: `[{"googleAuthConnection": {}}, {"passwordAuthConnection": {}}]` + +**Success response (HTTP 200):** +```json +{ "name": "accounts/...", "externalId": "...", "loginId": "my-workspace", "displayName": "My Workspace" } +``` + +Tell the user: +> Your workspace **** has been created! +> Workspace ID: `` +> Region: +> +> You can access it at: https://confidence.spotify.com + +**Error handling:** + +| HTTP Status | Meaning | User message | +|---|---|---| +| 400 + "work email" | Free email rejected | "Confidence requires a work email address. Free providers like Gmail aren't allowed." | +| 400 + "already have an account" | Logged-in Auth0 user already has account | "This login already has a Confidence account. Log in with a different email to create a new workspace." → re-run Step 1 | +| 400 + code 9 | Account under review | see "Under review handling" below — **do NOT assume email verification** | +| 400 | Other validation error | Parse `.message`, show in plain English, re-collect the invalid field | +| 401 | Token expired/invalid | "Session expired. Let me log you in again." → re-run Step 1 | +| 409 | Name already taken | "That workspace name was just taken. Let's pick another." → re-run Step 2 | +| 504 / timeout | gRPC deadline exceeded | Retry up to 3 times with 3-second delays. If it still fails, tell the user: "The server is taking longer than usual. Let me try once more." | +| 500+ | Server error | "Something went wrong on our end. Let me try again in a moment." | + +**Under review handling (code 9):** + +Code 9 means the account is "under review" — but the **reason** varies. Parse the `.message` field to determine the cause: + +1. **Email not verified** (message contains "verify" or "email"): Tell the user: "Please check your email for a verification link from Confidence and confirm your address. Let me know once you've done that!" + +2. **Account flagged/blocked** (message contains "fraud", "flagged", "blocked", "suspicious", or doesn't match #1): Tell the user: "Your account has been flagged for review. This usually resolves quickly. If it persists, contact support at confidence-support@spotify.com." + +3. **Generic "under review"** with no clear cause: Tell the user: "Your account is under review. This can happen for a few reasons — please check your email for any messages from Confidence. If you need help, contact confidence-support@spotify.com." + +**For case #1 only (email verification)**, after the user confirms, retry 4 times with 2-second delays in a single Bash command: + +```bash +for i in 1 2 3 4; do + RESP=$(curl -s -w "\n%{http_code}" --max-time 120 -X POST "https://onboarding.confidence.dev/v1/accounts" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '') + HTTP=$(echo "$RESP" | tail -1) + BODY=$(echo "$RESP" | sed '$d') + echo "ATTEMPT $i: HTTP=$HTTP" + echo "$BODY" + if [ "$HTTP" = "200" ]; then echo "SUCCESS"; break; fi + if [ "$HTTP" != "400" ] || ! echo "$BODY" | grep -q "under review"; then echo "DIFFERENT_ERROR"; break; fi + if [ "$i" -lt 4 ]; then sleep 2; fi +done +``` + +For cases #2 and #3, do NOT auto-retry — the issue won't resolve by retrying. Wait for the user to indicate they want to try again. + +If all 4 retry attempts still return "under review", tell the user: "Verification hasn't propagated yet. Please wait a moment and let me know when you'd like to try again." + +### Step 5: Get account-scoped token + +The token from Step 1 has no `org_id` (it was issued before the account existed). The signup client's refresh token **cannot** be exchanged for an org-scoped token — Auth0 rejects cross-client refresh, and the signup client doesn't support org-scoping. A browser auth with the regular client is required. + +**Use the browser auth script** with the **regular client ID** and the new org. The browser session from Step 1 is still active, so Auth0 auto-completes — the user sees no extra login prompt: +```bash +lsof -ti:8084 | xargs kill -9 2>/dev/null; python3 /auth.py 2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w +``` + +The response token will contain `org_id`, `account_name`, and `region` claims. Parse the TOKEN and REFRESH_TOKEN from stdout, then **save them in a separate `dangerouslyDisableSandbox: true` Bash call**: + +```bash +echo "" > "$TMPDIR/confidence_token" && echo "" > "$TMPDIR/confidence_refresh_token" +``` + +**This save call MUST use `dangerouslyDisableSandbox: true`** — even though it doesn't need network access — so that `$TMPDIR` resolves to the same path that future curl calls will use. A sandboxed save writes to a different TMPDIR and the token will be invisible to non-sandboxed curl calls. + +Tell the user: +> Connecting to your new workspace... (your browser will briefly open and close automatically — no action needed) + +### Step 6: Done + +Show a summary and next steps: + +``` +═══════════════════════════════════════════════════════════════ + Welcome to Confidence! +═══════════════════════════════════════════════════════════════ + + Workspace: () + Region: + Admin: + URL: https://confidence.spotify.com + + Next steps: + • Run the setup wizard: /onboard-confidence setup-wizard + • Invite team members: /onboard-confidence invite-user + • Set up data warehouse: /onboard-confidence setup-warehouse + • Create a feature flag: Ask me or use the Confidence UI + • Integrate your app: Ask me for SDK setup instructions + • Learn experimentation: /onboard-confidence learn + +═══════════════════════════════════════════════════════════════ +``` + +--- + +## Sub-command: invite-user + +### Step Tracker + +``` +───── Invite User ───────────────────────────────────────── + [1] Authenticate ○ pending + [2] Target account ○ pending + [3] Invitation details ○ pending + [4] Send invitation ○ pending +──────────────────────────────────────────────────────────── +``` + +### Step 1: Authenticate + +Check if a token is available from a prior `create-account` run in this session. + +If not, run the bundled auth script with the **regular client ID** (`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w`) — this user already has an account. + +Validate the token works by calling: +```bash +curl -s "https://iam.confidence.dev/v1/currentUser" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" +``` + +### Step 2: Target account + +Try to identify the account automatically: + +1. If MCP is connected, call `mcp__confidence-flags__getIdentityInfo` (no args) — returns current user's identity and account +2. If MCP isn't connected, use the `/v1/currentUser` REST response +3. If the user has multiple account memberships, ask which one + +Tell the user which account will receive the invitation. + +### Step 3: Invitation details + +Ask for: + +1. **Email address(es)** — required. Accept a single email or a comma-separated list for batch invites. + Validate email format locally. + +2. **Send invitation email?** — default yes. + > Should Confidence send an invitation email? (yes/no, default: yes) + +### Step 4: Send invitation + +For each email address: + +```bash +curl -s -w "\n%{http_code}" -X POST "https://iam.confidence.dev/v1/userInvitations" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{ + "userInvitation": { + "invitedEmail": "", + "disableInvitationEmail": + } + }' +``` + +**Success response:** +```json +{ + "name": "userInvitations/abc123", + "invitedEmail": "user@example.com", + "inviter": "Admin Name", + "expirationTime": "2026-06-03T10:00:00Z", + "invitationUri": "https://confidence.spotify.com/...", + "invitationToken": "..." +} +``` + +For single invite, tell the user: +> Invitation sent to **user@example.com**! +> They'll receive an email with instructions to join. +> The invitation expires on . + +For batch invites, show a summary table: +``` +Invitations sent: + ✓ alice@example.com — expires Jun 3 + ✓ bob@example.com — expires Jun 3 + ✗ charlie@invalid — invalid email address +``` + +**Error handling:** + +| HTTP Status | Meaning | User message | +|---|---|---| +| 400 | Invalid email | "That email address doesn't look right. Can you check it?" | +| 401 | Token expired | Re-authenticate (Step 1) | +| 403 | No permission | "You don't have permission to invite users. You need the admin role." | +| 409 | Already invited | "That user has already been invited." | + +--- + +## Sub-command: create-client + +Create an SDK client for flag resolution and generate its credentials. Uses REST APIs — no MCP needed. + +### Step Tracker + +``` +───── Create Client ─────────────────────────────────────── + [1] Client name ○ pending + [2] Create client ○ pending + [3] Get credentials ○ pending +──────────────────────────────────────────────────────────── +``` + +### Step 1: Client name + +Ask the user what to name the client. Suggest based on platform: + +> What should we call this client? (e.g., "iOS App", "Web Frontend", "Backend Service") + +### Step 2: Create client + +Body is the client object directly (proto `body: "client"`): +```bash +curl -s -w "\n%{http_code}" -X POST "https://iam.${REGION}.confidence.dev/v1/clients" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"display_name": ""}' +``` + +Response includes `name` (e.g., `clients/kqr3nc9dh70cwt5e2vws`). Save this for Step 3. + +### Step 3: Get credentials + +Body is the credential object directly (proto `body: "client_credential"`): +```bash +curl -s -w "\n%{http_code}" -X POST "https://iam.${REGION}.confidence.dev/v1/${CLIENT_NAME}/credentials" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"display_name": "Default Secret"}' +``` + +The `clientSecret.secret` is only returned once on creation — show it to the user. + +``` +═══════════════════════════════════════════════════════════════ + Client Created +═══════════════════════════════════════════════════════════════ + + Name: + Secret: + + Use this secret in your SDK configuration to resolve flags. + Keep it safe — you can regenerate it, but the old one will + stop working. + + Next: Ask me for SDK integration instructions, or run + /onboard-confidence setup-wizard + +═══════════════════════════════════════════════════════════════ +``` + +--- + +## Sub-command: setup-wizard + +Guided walkthrough of the full onboarding checklist. Uses MCP tools for flag/client operations when available, REST for everything else. + +### User input style + +**Always use AskUserQuestion** with selectable options for choices (up/down/enter). Only ask the user to type free-text when collecting names, emails, or other open-ended input. Never present numbered lists in plain text when AskUserQuestion can be used instead. + +### Step Tracker + +``` +───── Setup Wizard ──────────────────────────────────────── + [1] Get started ○ pending + [2] Connect tools ○ pending + [3] Create client ○ pending + [4] Create flag ○ pending + [5] Add targeting ○ pending + [6] Test resolve ○ pending + [7] Done ○ pending +──────────────────────────────────────────────────────────── +``` + +### Step 1: Get started + +If the user already answered "create account" vs "sign in" (e.g., from the default onboarding flow), use that answer — do NOT re-ask. + +Otherwise (when entered directly via `/onboard-confidence setup-wizard`), use AskUserQuestion: +- **Create a new account** — I'll walk you through signup +- **Sign in to an existing account** — I already have one + +**If "Create a new account":** +Run the full `create-account` sub-command flow (Steps 1–6 from that section). This handles signup, workspace creation, and re-auth with an org-scoped token. Once complete, proceed to Step 2 of setup-wizard with the token and region already set. + +**If "Sign in to existing account":** +Check if a token file exists at `$TMPDIR/confidence_token` and is valid. If not, run the bundled auth script with the **regular client ID** (`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w`). Validate the token, extract the region, and proceed to Step 2. + +Determine the region from the token — this sets the API base URLs: +- EU: `flags.eu.confidence.dev`, `resolver.eu.confidence.dev`, `iam.eu.confidence.dev` +- US: `flags.us.confidence.dev`, `resolver.us.confidence.dev`, `iam.us.confidence.dev` + +### Step 2: Connect tools + +**This step is critical for onboarding success.** The Confidence MCP tools provide a richer, more reliable experience for managing flags and clients. Nudge the user to connect them now — it only takes a few seconds since their browser session from login will auto-complete. + +Tell the user: +> Before we create your first flag, let's connect the Confidence tools. This gives you richer flag management right inside Claude Code. +> +> Type **`/mcp`** in the prompt, then click **Authenticate** next to **confidence-flags**. Your browser session from login will auto-complete — no extra password needed. +> +> Let me know once you've done that! + +**After the user confirms**, verify MCP is connected by calling `mcp__confidence-flags__getIdentityInfo` (no args). If it succeeds, MCP is connected — set an internal flag `MCP_CONNECTED=true` and proceed. + +**If the user skips** or MCP call fails, proceed with REST fallback — set `MCP_CONNECTED=false`. Tell the user: +> No problem! I'll use the REST API instead. You can always connect the tools later with `/mcp`. + +### Step 3: Create client + +**MCP path** (when `MCP_CONNECTED=true`): + +Check if the user already has a client by calling `mcp__confidence-flags__listClients`. + +If clients exist, use AskUserQuestion to let the user pick one or create a new one. If none exist, ask for a client name and type: + +> What should we call this client? (e.g., "iOS App", "Web Frontend", "Backend Service") + +Then use AskUserQuestion for client type: +- **Frontend** — browser/mobile apps +- **Backend** — server-side services + +Call `mcp__confidence-flags__createClient` with `displayName` and `clientType`. +Then call `mcp__confidence-flags__getClientSecret` with the `clientName` to get the secret. + +**REST fallback** (when `MCP_CONNECTED=false`): + +Check existing clients: +```bash +curl -s "https://iam.${REGION}.confidence.dev/v1/clients" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" +``` + +If clients exist, use AskUserQuestion to pick one. If none, create via REST: +```bash +curl -s -w "\n%{http_code}" -X POST "https://iam.${REGION}.confidence.dev/v1/clients" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"display_name": ""}' +``` + +Then fetch credentials: +```bash +curl -s "https://iam.${REGION}.confidence.dev/v1/${CLIENT_NAME}/credentials" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" +``` + +Save the client `name` and `clientSecret` for later steps. + +### Step 4: Create flag + +EDUCATE then ASK: +> A feature flag controls a piece of functionality. Let's create your first one. +> What should it be called? (e.g., "new-checkout-flow", "dark-mode") + +Validate: 4-63 chars, `[a-z0-9-]`. + +Use AskUserQuestion for variant type: +- **Simple on/off (boolean)** — two variants: on and off +- **Custom variants** — I'll name my own + +**MCP path** (when `MCP_CONNECTED=true`): + +The MCP `createFlag` tool handles schema, variants, AND client attachment in one call: + +For on/off: +``` +mcp__confidence-flags__createFlag({ + flagName: "", + clientName: "", + schemaObject: '{"enabled": "boolean"}', + variants: '[{"name": "on", "value": {"enabled": true}}, {"name": "off", "value": {"enabled": false}}]' +}) +``` + +For custom variants, infer the schema from what the user describes and pass it similarly. + +**REST fallback** (when `MCP_CONNECTED=false`): + +Create flag, set schema, add variants, then attach to client — all in a single chained Bash call: + +```bash +curl -s -X POST "https://flags.${REGION}.confidence.dev/v1/flags?flag_id=" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{}' && \ +curl -s -X PATCH "https://flags.${REGION}.confidence.dev/v1/flags/" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"schema": {"enabled": {"boolSchema": {}}}}}' && \ +curl -s -X POST "https://flags.${REGION}.confidence.dev/v1/flags//variants" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"name": "flags//variants/on", "value": {"enabled": true}}' && \ +curl -s -X POST "https://flags.${REGION}.confidence.dev/v1/flags//variants" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"name": "flags//variants/off", "value": {"enabled": false}}' && \ +curl -s -X POST "https://flags.${REGION}.confidence.dev/v1/flags/:addFlagClient" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"client": "", "flag": "flags/"}' +``` + +### Step 5: Add targeting + +EDUCATE: +> Targeting rules control who sees which variant. Let's set a default — you can add more rules later. + +Use AskUserQuestion to pick the default variant (list the variants created in Step 4). + +**MCP path** (when `MCP_CONNECTED=true`): + +The MCP `addTargetingRule` tool handles segment creation internally: +``` +mcp__confidence-flags__addTargetingRule({ + flagName: "", + variantAllocations: '{"": 100}' +}) +``` + +**REST fallback** (when `MCP_CONNECTED=false`): + +Create a catch-all segment (if one doesn't exist), allocate it, then create a rule — all in one Bash call: +```bash +curl -s -X POST "https://flags.${REGION}.confidence.dev/v1/segments?segment_id=everyone" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"display_name": "Everyone"}' && \ +curl -s -X PATCH "https://flags.${REGION}.confidence.dev/v1/segments/everyone" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{"allocation": {"proportion": {"value": "1"}}}' && \ +curl -s -X POST "https://flags.${REGION}.confidence.dev/v1/segments/everyone:allocate" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{}' && \ +curl -s -w "\n%{http_code}" -X POST "https://flags.${REGION}.confidence.dev/v1/flags//rules" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{ + "segment": "segments/everyone", + "assignment_spec": { + "bucket_count": 100, + "assignments": [{ + "assignment_id": "", + "variant": {"variant": "flags//variants/"}, + "bucket_ranges": [{"lower": 0, "upper": 100}] + }] + }, + "targeting_key_selector": "targeting_key", + "enabled": true + }' +``` + +**IMPORTANT (REST only):** Segment proportion must be > 0 and `:allocate` must be called, otherwise resolve returns empty. + +### Step 6: Test resolve + +EDUCATE: +> Let's verify the flag works by resolving it for different contexts. + +**Test all targeting cases.** If the flag has targeting rules that depend on context fields (e.g., `country`), resolve with context values that exercise EACH rule — both matching and non-matching cases. For example, if the rule is "on when country is not US", test with `country: "SE"` (should match → on) AND `country: "US"` (should not match → off/default). Show results for all cases in a summary table. + +**MCP path** (when `MCP_CONNECTED=true`): + +Make parallel resolve calls for each test case: +``` +mcp__confidence-flags__resolveFlag({ + flagName: "", + clientName: "", + entity: "targeting_key", + entityValue: "test-user-1", + context: '{"": ""}' +}) + +mcp__confidence-flags__resolveFlag({ + flagName: "", + clientName: "", + entity: "targeting_key", + entityValue: "test-user-1", + context: '{"": ""}' +}) +``` + +**REST fallback** (when `MCP_CONNECTED=false`): + +```bash +# Test matching case +curl -s -w "\n%{http_code}" -X POST "https://resolver.${REGION}.confidence.dev/v1/flags:resolve" \ + -H "Content-Type: application/json" \ + -d '{ + "flags": ["flags/"], + "evaluationContext": { + "targeting_key": "test-user-1", + "": "" + }, + "clientSecret": "", + "apply": true + }' && echo "---" && \ +# Test non-matching case +curl -s -w "\n%{http_code}" -X POST "https://resolver.${REGION}.confidence.dev/v1/flags:resolve" \ + -H "Content-Type: application/json" \ + -d '{ + "flags": ["flags/"], + "evaluationContext": { + "targeting_key": "test-user-1", + "": "" + }, + "clientSecret": "", + "apply": true + }' +``` + +Show results in a summary: +``` + Test Results: + country = SE → variant "on" (enabled: true) ✓ + country = US → variant "off" (enabled: false) ✓ +``` + +If resolve fails or returns no match, check that: +1. The flag is attached to the client +2. Rules are enabled +3. Context fields required by targeting rules are included in the resolve call +4. A catch-all rule exists for non-matching contexts (otherwise they fall through to code default) + +### Step 7: Done + +Show a summary, then offer SDK integration using the **confidence-docs MCP**: + +``` +═══════════════════════════════════════════════════════════════ + Setup Complete! +═══════════════════════════════════════════════════════════════ + + Client: + Secret: + Flag: + Variants: + Default: + + Your flag is live and resolving! + +═══════════════════════════════════════════════════════════════ +``` + +Use AskUserQuestion for next steps: +- **Integrate the SDK** — get code snippets for your platform +- **Invite team members** — add collaborators to your workspace +- **Set up data warehouse** — connect analytics pipeline +- **Create more flags** — keep building +- **Learn experimentation** — interactive course on A/B testing + +**If the user picks "Integrate the SDK"**, use `mcp__confidence-docs__getCodeSnippetAndSdkIntegrationTips` with the user's platform (ask via AskUserQuestion: JavaScript, Python, Java, Kotlin, Swift, Go, React) to provide tailored integration code. This gives the user the exact SDK setup they need. + +**For other choices**, direct to the corresponding sub-command. + +--- + +## Sub-command: setup-warehouse + +This command has been split into dedicated skills for each warehouse type. When the user asks to set up a warehouse, use `/onboard-confidence:setup-warehouse` which will guide them to the right one: +- `/onboard-confidence:setup-warehouse-bigquery` +- `/onboard-confidence:setup-warehouse-snowflake` +- `/onboard-confidence:setup-warehouse-databricks` +- `/onboard-confidence:setup-warehouse-redshift` + +--- + +## Sub-command: learn + +Interactive learning about experimentation concepts. The skill teaches, asks questions, and the user answers — like a guided course. + +### Topics + +| Topic | Category | What it covers | +|-------|----------|----------------| +| Statistics | STATS | Statistical significance, p-values, confidence intervals, sample size | +| Experiment Design | DESIGN | Hypothesis formation, control/treatment, randomization, bias | +| Feature Flags | FLAGS | Flag types, targeting rules, rollouts, kill switches | +| Metrics | METRICS | Metric types, guardrails, primary/secondary metrics, SRM | +| Coordination | COORDINATION | Mutual exclusion, layered experiments, interaction effects | + +### Flow + +1. **Pick a topic:** + > What would you like to learn about? + > 1. Statistics fundamentals + > 2. Experiment design + > 3. Feature flags + > 4. Metrics + > 5. Coordination + +2. **Fetch content** — use `mcp__confidence-docs__searchDocumentation` to get relevant Confidence documentation for the chosen topic. + +3. **Teach** — present a concept from the docs in 2-3 clear paragraphs. Use examples relevant to the user's product. + +4. **Ask a question** — pose a comprehension question with multiple-choice answers: + > **Question:** When running an A/B test, why is it important to determine sample size before starting? + > 1. To make the test run faster + > 2. To ensure you have enough statistical power to detect the expected effect + > 3. To reduce server costs + > 4. It's not important — you can stop whenever + +5. **Evaluate the answer** — if correct, explain why. If wrong, explain the right answer and the reasoning. + +6. **Track progress** — call the Learning API to record the user's answer: + ```bash + curl -s -X POST "https://onboarding.confidence.dev/v1/learningProgress:answerQuestions" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" \ + -H "Content-Type: application/json" \ + -d '{ + "course": "courses/", + "questionUpdates": [{ + "lessonIndex": , + "questionIndex": , + "currentAnswerIndex": + }] + }' + ``` + +7. **Continue or finish** — after each question, ask if they want to continue or switch topics. + +8. **Show progress** — at any time, fetch and display progress: + ```bash + curl -s "https://onboarding.confidence.dev/v1/learningProgress" \ + -H "Authorization: Bearer $(cat $TMPDIR/confidence_token)" + ``` + + ``` + ───── Learning Progress ──────────────────────────────────── + Statistics: ██████░░░░ 3/5 lessons + Design: ████████░░ 4/5 lessons + Feature Flags: ██████████ 5/5 complete! + Metrics: ░░░░░░░░░░ not started + Coordination: ░░░░░░░░░░ not started + ──────────────────────────────────────────────────────────── + ``` + +### Key principles + +- **Use AskUserQuestion** for topic selection, quiz answers, and continue/switch decisions — selectable options, not typed numbers +- **Be conversational** — this is a dialogue, not a textbook +- **Use real examples** — tie concepts to the user's product/domain when possible +- **Encourage exploration** — if the user asks follow-up questions, answer them before moving on +- **Track everything** — every answer gets recorded via the Learning API so progress persists across sessions + +--- + +## Sub-command: status + +This is a lightweight command. Try MCP first (no REST auth needed if MCP is connected). + +**If MCP is connected:** + +1. Call `mcp__confidence-flags__getIdentityInfo` (no args) +2. Call `mcp__confidence-flags__listClients` +3. Display: + +``` +═══════════════════════════════════════════════════════════════ + Confidence Account Status +═══════════════════════════════════════════════════════════════ + + Identity: () + Account: + Clients: + + MCP Status: + confidence-flags: ● connected + confidence-docs: ● connected + +═══════════════════════════════════════════════════════════════ +``` + +**If MCP is NOT connected:** + +1. Check if a token is available from a prior command in this session +2. If yes, call `GET https://iam.confidence.dev/v1/currentUser` and display the result +3. If no token, tell the user: + > No active session. Run `/onboard-confidence create-account` to get started, or `/mcp` to authenticate Confidence tools. + +--- + +## API Reference (agent-internal — do NOT show to user) + +### Base URLs + +All APIs except onboarding and Auth0 require **region-specific URLs**. Extract region from the JWT token claim `https://confidence.dev/region` (value: `EU` or `US`), lowercase it, and use as prefix. + +``` +AUTH0_DOMAIN: auth.confidence.dev +ONBOARDING_API: https://onboarding.confidence.dev/v1 (no region prefix) +IAM_API: https://iam.${region}.confidence.dev/v1 (e.g., iam.eu.confidence.dev) +FLAGS_API: https://flags.${region}.confidence.dev/v1 +RESOLVER_API: https://resolver.${region}.confidence.dev/v1 +EVENTS_API: https://events.${region}.confidence.dev/v1 +CONNECTORS_API: https://connectors.${region}.confidence.dev/v1 +METRICS_API: https://metrics.${region}.confidence.dev/v1 +``` + +### Endpoints + +**Check login ID availability (no auth):** +``` +GET ${ONBOARDING_API}/loginIdAvailability:check?login_id={id} +→ { "available": bool } +``` + +**Check region availability (no auth):** +``` +GET ${ONBOARDING_API}/country:validate +→ { "allowed": bool } +``` + +**Create account (Bearer token):** +``` +POST ${ONBOARDING_API}/accounts +Body: { + "account": { + "displayName": string, + "loginId": string, + "region": "REGION_EU" | "REGION_US", + "authConnections": [ {"googleAuthConnection":{}} | {"passwordAuthConnection":{}} ], + "adminEmail": string (must be work email — free providers rejected), + "allowedLoginEmailDomains": [string] (optional) + }, + "marketingOptIn": bool (optional), + "userRole": string (optional), + "userGoals": [string] (optional) +} +→ { "name": string, "externalId": string, "loginId": string, "displayName": string } +``` + +**Create user invitation (Bearer token + admin permission):** +``` +POST ${IAM_API}/userInvitations +Body: { + "userInvitation": { + "invitedEmail": string, + "ttl": { "seconds": int } (optional, default 7 days), + "disableInvitationEmail": bool (optional, default false), + "labels": { string: string } (optional) + } +} +→ { + "name": string, + "invitedEmail": string, + "inviter": string, + "expirationTime": string, + "invitationUri": string, + "invitationToken": string +} +``` + +**List user invitations (Bearer token):** +``` +GET ${IAM_API}/userInvitations +→ { "userInvitations": [...], "nextPageToken": string } +``` + +**Get current user (Bearer token):** +``` +GET ${IAM_API}/currentUser +→ { + "user": { "name", "fullName", "email", ... }, + "accountMemberships": [{ "account", "displayName", "loginId", "region" }], + "account": string, + "identity": { "name", "displayName", ... } +} +``` + +**Create client (Bearer token, body: "client"):** +``` +POST ${IAM_API}/clients +Body (direct client object): { "display_name": string } +→ { "name": "clients/...", "displayName": string, ... } +``` + +**Create client credential (Bearer token, body: "client_credential"):** +``` +POST ${IAM_API}/${clientName}/credentials +Body (direct credential object): { "display_name": string } +→ { "name": "clients/.../clientCredentials/...", "clientSecret": { "secret": string }, ... } + NOTE: secret only returned once on creation +``` + +**List clients (Bearer token):** +``` +GET ${IAM_API}/clients +→ { "clients": [...], "nextPageToken": string } +``` + +**Create flag (Bearer token, body: "flag", flag_id is query param):** +``` +POST ${FLAGS_API}/flags?flag_id= +Body (direct flag object): {} + flag_id: 4-63 chars, [a-z0-9-] +→ Flag object +``` + +**Update flag schema (Bearer token, body: "flag"):** +``` +PATCH ${FLAGS_API}/flags/ +Body: { "schema": { "schema": { "": { "boolSchema": {} | "stringSchema": {} | "intSchema": {} | "doubleSchema": {} } } } } +→ Flag object + NOTE: schema MUST be set before adding variants with values +``` + +**Add flag to client (Bearer token, body: "*"):** +``` +POST ${FLAGS_API}/flags/:addFlagClient +Body: { "client": "clients/", "flag": "flags/" } +→ Flag object +``` + +**Create variant (Bearer token, body: "variant"):** +``` +POST ${FLAGS_API}/flags//variants +Body (direct variant object): { "name": "flags//variants/", "value": { ... } } +→ Variant object + NOTE: value fields must match the flag schema +``` + +**Create rule (Bearer token, body: "rule"):** +``` +POST ${FLAGS_API}/flags//rules +Body (direct rule object): { "assignment_spec": { ... }, "targeting_key_selector": "targeting_key", "enabled": true } +→ Rule object +``` + +**Resolve flags (client secret — NOT Bearer token):** +``` +POST ${RESOLVER_API}/flags:resolve +Body: { + "flags": ["flags/"], + "evaluationContext": { "targeting_key": string, ... }, + "clientSecret": string, + "apply": bool +} +→ { "resolvedFlags": [{ "flag": string, "variant": string, "value": {...}, "reason": string }] } +``` + +**List event definitions (Bearer token):** +``` +GET https://events.${region}.confidence.dev/v1/eventDefinitions +→ { "eventDefinitions": [...], "nextPageToken": string } +``` + +**Create event definition (Bearer token):** +``` +POST https://events.${region}.confidence.dev/v1/eventDefinitions?event_definition_id= +Body (direct object): { "schema": { "": { "stringSchema": {} | "intSchema": {} | "doubleSchema": {} | "boolSchema": {} } } } +→ EventDefinition object +``` + +**Update event definition schema (Bearer token):** +``` +PATCH https://events.${region}.confidence.dev/v1/eventDefinitions/ +Body: { "schema": { "": { "stringSchema": {} } } } +→ EventDefinition object + NOTE: schema fields determine which payload fields appear as columns in warehouse +``` + +**Publish events (client secret — NOT Bearer token):** +``` +POST https://events.${region}.confidence.dev/v1/events:publish +Body: { + "client_secret": string, + "events": [{ "event_definition": "eventDefinitions/", "payload": {...}, "event_time": "ISO8601" }], + "send_time": "ISO8601" +} +→ { "errors": [{ "index": int, "reason": string, "message": string }] } + Empty errors array = success +``` + +**Create data warehouse (Bearer token):** +``` +POST ${METRICS_API}/dataWarehouses +Body: { "dataWarehouse": { "config": { "Config": {...} } } } +→ DataWarehouse object +``` + +**Validate warehouse config (Bearer token):** +``` +POST ${METRICS_API}/dataWarehouseConfig:validate +Body: { "Config": {...} } +→ { "validation": [...], "successful": bool, "configurationResponse": {...} } +``` + +**Check warehouse exists (Bearer token):** +``` +GET ${METRICS_API}/dataWarehouses:exists +→ { "exists": bool } +``` + +**Create flag applied connection (Bearer token):** +``` +POST ${CONNECTORS_API}/flagAppliedConnections +Body: { "flagAppliedConnection": { "": { "Config": {...}, "table": "..." } } } +→ FlagAppliedConnection object +``` + +**Create event connection (Bearer token):** +``` +POST ${CONNECTORS_API}/eventConnections +Body: { "eventConnection": { "": { "Config": {...}, "tablePrefix": "..." } } } +→ EventConnection object +``` + +**Create assignment table (Bearer token):** +``` +POST ${METRICS_API}/assignmentTables +Body: { "assignmentTable": { "displayName": str, "sql": str, "entityColumn": {...}, "timestampColumn": {...}, "exposureKeyColumn": {...}, "variantKeyColumn": {...}, "dataDeliveredUntilUpdateStrategyConfig": {...} } } +→ AssignmentTable object +``` + +**Get learning progress (Bearer token):** +``` +GET https://onboarding.confidence.dev/v1/learningProgress +→ { "courseProgresses": [...], "completedCourses": int } +``` + +**Answer questions (Bearer token):** +``` +POST https://onboarding.confidence.dev/v1/learningProgress:answerQuestions +Body: { "course": "courses/", "questionUpdates": [{ "lessonIndex": int, "questionIndex": int, "currentAnswerIndex": int }] } +→ LearningProgress object +``` + +### Validation Rules + +| Field | Rule | Regex | +|-------|------|-------| +| `loginId` | 3-21 chars, lowercase, digits, hyphens. Starts with letter, ends with letter/digit | `^[a-z][a-z0-9-]{1,19}[a-z0-9]$` | +| `displayName` | 3-32 chars, letters, digits, Unicode letters, spaces, hyphens. Starts/ends with word char/digit/letter | `[\w\d\p{L}][\w\s\d\-\p{L}]{1,30}[\w\d\p{L}]` | +| `region` | Exactly `REGION_EU` or `REGION_US` | — | +| `authConnections` | At least one required | — | +| `adminEmail` | Must be a work email. Free providers (Gmail, Yahoo, Hotmail, etc.) are rejected | — | + +--- + +## Error Handling Reference (agent-internal) + +### Common HTTP errors + +| Status | Meaning | Recovery | +|--------|---------|----------| +| 400 | Validation error | Parse `.message`, show plain English, re-collect invalid field | +| 401 | Invalid/expired token | Re-trigger Auth0 login | +| 403 | Insufficient permissions | Explain needed role/permission | +| 404 | Resource not found | Check account/resource exists | +| 409 | Conflict (already exists) | Name taken or user already invited | +| 429 | Rate limited | Wait briefly and retry | +| 500+ | Server error | Inform user, suggest retry | + +### Sandbox note + +All `curl`, `open`, and `python3` commands that access external hosts (`auth.confidence.dev`, `onboarding.confidence.dev`, `iam.confidence.dev`) require `dangerouslyDisableSandbox: true`. The auth script additionally requires `timeout: 130000` (server timeout is 120s). On first occurrence, briefly explain to the user that network access outside the sandbox is needed for API calls. + +--- + +## MCP Tools Reference + +MCP tools are used for **flag and client operations only** — account creation, invitations, segments, and warehouse config always use REST. + +### confidence-flags MCP (flag/client operations) + +| Tool | Used by | Purpose | +|------|---------|---------| +| `mcp__confidence-flags__getIdentityInfo` | `status`, `setup-wizard` | Verify MCP connection, get identity | +| `mcp__confidence-flags__listClients` | `status`, `setup-wizard` | List available clients | +| `mcp__confidence-flags__createClient` | `setup-wizard` | Create SDK client with name + type | +| `mcp__confidence-flags__getClientSecret` | `setup-wizard` | Retrieve client secret | +| `mcp__confidence-flags__createFlag` | `setup-wizard` | Create flag with schema, variants, and client in one call | +| `mcp__confidence-flags__addTargetingRule` | `setup-wizard` | Add targeting rule with variant allocations (handles segments internally) | +| `mcp__confidence-flags__resolveFlag` | `setup-wizard` | Test flag resolution | + +### confidence-docs MCP (documentation) + +| Tool | Used by | Purpose | +|------|---------|---------| +| `mcp__confidence-docs__searchDocumentation` | `learn` | Fetch educational content | +| `mcp__confidence-docs__getCodeSnippetAndSdkIntegrationTips` | `setup-wizard` (Step 7) | SDK integration guides per platform | + +### What stays on REST (never use MCP) + +- Account creation, email verification, login ID checks → `onboarding.confidence.dev` +- User invitations → `iam.*.confidence.dev` +- Segment creation and allocation → `flags.*.confidence.dev` +- Warehouse config, connectors, assignment tables → `metrics.*.confidence.dev`, `connectors.*.confidence.dev` +- Learning progress tracking → `onboarding.confidence.dev` + +--- + +## Known Limitations + +- **MCP auth cannot be triggered programmatically** — user must run `/mcp` to authenticate MCP servers. The Auth0 browser session from the login step makes this instant (no second login). The setup wizard nudges this at Step 2. +- **MCP is for flag/client operations only** — account creation, invitations, segments, warehouse config, and learning progress always use REST APIs. +- **Port 8084 must be free** — the Auth0 callback server uses a fixed port. The auth script auto-kills any existing process on port 8084. +- **Auth0 Allowed Callback URLs** — both Auth0 clients must have `http://localhost:8084/callback` in their Allowed Callback URLs, Allowed Logout URLs, and Allowed Web Origins. +- **Auth script is bundled** — `auth.py` ships with the plugin in the skill directory. Never write auth scripts to disk; always use the bundled script. +- **Token persistence and TMPDIR** — tokens are written to `$TMPDIR/confidence_token`. `$TMPDIR` resolves to DIFFERENT paths in sandboxed vs non-sandboxed (`dangerouslyDisableSandbox: true`) Bash calls (e.g., `/tmp/claude-501/` vs `/var/folders/.../T/`). ALL token writes and reads MUST use `dangerouslyDisableSandbox: true` to ensure consistency. Never write tokens outside `$TMPDIR`. +- **Learning API** — REST-only (gRPC on epx-onboarding). Course content is generated by the skill using docs MCP; the API only tracks progress indices. +- **`learn` sub-command** — uses docs MCP for content. If MCP not connected, the skill can still teach using its own knowledge but won't have the latest docs. +- **Region-specific API URLs** — flags/resolver APIs use region prefixes (`flags.eu.confidence.dev` vs `flags.us.confidence.dev`). Determine region from the JWT token or from the account creation step. diff --git a/skills/onboard-confidence/auth.py b/skills/onboard-confidence/auth.py new file mode 100644 index 0000000..4d36eb2 --- /dev/null +++ b/skills/onboard-confidence/auth.py @@ -0,0 +1,95 @@ +""" +Confidence Auth0 PKCE login flow. + +Usage: + python3 auth.py [ORGANIZATION] + +Outputs on stdout: + WAITING_FOR_LOGIN — browser opened, waiting for callback + TOKEN: — success, JWT access token + REFRESH_TOKEN: — refresh token (if granted) + AUTH_ERROR: — auth0 returned an error + TOKEN_ERROR: — token exchange failed + +Exit codes: 0 = success, 1 = error +""" +import http.server, urllib.parse, json, sys, subprocess, hashlib, base64, secrets, string, signal + +CLIENT_ID = sys.argv[1] +ORGANIZATION = sys.argv[2] if len(sys.argv) > 2 else '' + +code_verifier = ''.join(secrets.choice(string.ascii_letters + string.digits + '-._~') for _ in range(43)) +code_challenge = base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode()).digest()).rstrip(b'=').decode() + +port = 8084 +REDIRECT_URI = f'http://localhost:{port}/callback' +auth_code = None +error = None + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + global auth_code, error + q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query) + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.end_headers() + if 'code' in q: + auth_code = q['code'][0] + self.wfile.write(b'

Login successful!

You can close this tab.

') + else: + error = q.get('error', ['unknown'])[0] + self.wfile.write(b'

Login failed

Please try again.

') + def log_message(self, format, *args): + pass + +params = { + 'client_id': CLIENT_ID, + 'redirect_uri': REDIRECT_URI, + 'response_type': 'code', + 'scope': 'openid profile email offline_access', + 'audience': 'https://confidence.dev/', + 'code_challenge': code_challenge, + 'code_challenge_method': 'S256', +} +if ORGANIZATION: + params['organization'] = ORGANIZATION +else: + params['screen_hint'] = 'signup' + params['prompt'] = 'login' + +authorize_url = 'https://auth.confidence.dev/authorize?' + urllib.parse.urlencode(params) +subprocess.Popen(['open', authorize_url]) +print('WAITING_FOR_LOGIN', flush=True) + +server = http.server.HTTPServer(('127.0.0.1', port), Handler) +server.timeout = 120 +while auth_code is None and error is None: + server.handle_request() +server.server_close() + +if error: + print(f'AUTH_ERROR:{error}', flush=True) + sys.exit(1) + +import urllib.request +token_data = json.dumps({ + 'grant_type': 'authorization_code', + 'client_id': CLIENT_ID, + 'code': auth_code, + 'redirect_uri': REDIRECT_URI, + 'code_verifier': code_verifier +}).encode() +req = urllib.request.Request( + 'https://auth.confidence.dev/oauth/token', + data=token_data, + headers={'Content-Type': 'application/json'} +) +try: + with urllib.request.urlopen(req) as resp: + token_response = json.loads(resp.read()) + print(f'TOKEN:{token_response["access_token"]}', flush=True) + if 'refresh_token' in token_response: + print(f'REFRESH_TOKEN:{token_response["refresh_token"]}', flush=True) +except Exception as e: + print(f'TOKEN_ERROR:{e}', flush=True) + sys.exit(1) diff --git a/skills/setup-warehouse-bigquery/SKILL.md b/skills/setup-warehouse-bigquery/SKILL.md new file mode 100644 index 0000000..26b1c72 --- /dev/null +++ b/skills/setup-warehouse-bigquery/SKILL.md @@ -0,0 +1,705 @@ +--- +description: Set up BigQuery as a data warehouse for Confidence. Use when the user chose BigQuery for warehouse setup. +--- + +# Setup Warehouse: BigQuery + +Configure BigQuery as the data warehouse for Confidence experimentation analytics. This skill handles the full end-to-end setup: collect GCP config, validate permissions, create the warehouse, set up connectors, create the assignment table, and verify the pipeline. + +--- + +## Authentication + +**Browser-based Auth0 login.** The skill opens a browser for Auth0 login (Google, email/password, SSO) and captures the token automatically. The user never touches a token. + +### Session-only token management + +The token is kept in the current session only and is never saved to disk. If the session ends or the token expires, the skill will open your browser to log in again. + +On every sub-command start, check if the `TOKEN` variable is set and not expired: + +```bash +if [ -n "$TOKEN" ]; then + PAYLOAD=$(echo "$TOKEN" | cut -d. -f2) + EXP=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('exp', 0)) +") + NOW=$(date +%s) + if [ "$EXP" -gt "$NOW" ]; then + echo "VALID" + else + echo "EXPIRED" + unset TOKEN + fi +fi +``` + +If `TOKEN` is unset or expired, run the Auth0 login flow with the **regular client ID** (`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w`) and the user's `organization` parameter. Store the result in the `TOKEN` shell variable only. **NEVER write the token to disk. NEVER reference `~/.confidence/`.** + +### Auth script + +Write the following to `$TMPDIR/confidence_auth.py` with CLIENT_ID=`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w` and ORGANIZATION from the token. Run with `python3 $TMPDIR/confidence_auth.py`. Outputs `TOKEN:` on success. + +```python +import http.server, urllib.parse, json, sys, subprocess, hashlib, base64, secrets, string + +code_verifier = ''.join(secrets.choice(string.ascii_letters + string.digits + '-._~') for _ in range(43)) +code_challenge = base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode()).digest()).rstrip(b'=').decode() + +port = 8084 +CLIENT_ID = '' +ORGANIZATION = '' +REDIRECT_URI = f'http://localhost:{port}/callback' +auth_code = None +error = None + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + global auth_code, error + q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query) + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.end_headers() + if 'code' in q: + auth_code = q['code'][0] + self.wfile.write(b'

Login successful!

You can close this tab.

') + else: + error = q.get('error', ['unknown'])[0] + self.wfile.write(b'

Login failed

Please try again.

') + def log_message(self, format, *args): + pass + +params = { + 'client_id': CLIENT_ID, + 'redirect_uri': REDIRECT_URI, + 'response_type': 'code', + 'scope': 'openid profile email offline_access', + 'audience': 'https://confidence.dev/', + 'code_challenge': code_challenge, + 'code_challenge_method': 'S256', +} +if ORGANIZATION: + params['organization'] = ORGANIZATION + +authorize_url = 'https://auth.confidence.dev/authorize?' + urllib.parse.urlencode(params) +subprocess.Popen(['open', authorize_url]) +print('WAITING_FOR_LOGIN', flush=True) + +server = http.server.HTTPServer(('127.0.0.1', port), Handler) +server.timeout = 120 +while auth_code is None and error is None: + server.handle_request() +server.server_close() + +if error: + print(f'AUTH_ERROR:{error}', flush=True) + sys.exit(1) + +import urllib.request +token_data = json.dumps({ + 'grant_type': 'authorization_code', + 'client_id': CLIENT_ID, + 'code': auth_code, + 'redirect_uri': REDIRECT_URI, + 'code_verifier': code_verifier +}).encode() +req = urllib.request.Request( + 'https://auth.confidence.dev/oauth/token', + data=token_data, + headers={'Content-Type': 'application/json'} +) +try: + with urllib.request.urlopen(req) as resp: + token_response = json.loads(resp.read()) + print(f'TOKEN:{token_response["access_token"]}', flush=True) +except Exception as e: + print(f'TOKEN_ERROR:{e}', flush=True) + sys.exit(1) +``` + +### Extract region from token + +```bash +REGION=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('https://confidence.dev/region', 'EU')) +") +``` + +Then use `${REGION,,}` (lowercase) for URL prefix: `iam.eu.confidence.dev`, `metrics.eu.confidence.dev`, etc. + +### Common notes + +- Port is fixed at **8084** (must match Auth0 Allowed Callback URLs) +- If port 8084 is busy: `lsof -ti:8084 | xargs kill -9 2>/dev/null` +- All network commands require `dangerouslyDisableSandbox: true` +- Never show the token value to the user +- Always use region-specific URLs (e.g., `iam.eu.confidence.dev` not `iam.confidence.dev`) + +### Important: gRPC-REST transcoding rules + +The Confidence APIs use gRPC with REST transcoding. The `body` field in the proto HTTP binding determines the JSON structure: + +- **`body: "data_warehouse"`** -> send the data warehouse object directly: `{"config": {...}}` +- **`body: "flag_applied_connection"`** -> send the connection object directly: `{"bigQuery": {...}}` +- **`body: "event_connection"`** -> send the connection object directly: `{"bigQuery": {...}}` +- **`body: "assignment_table"`** -> send the assignment table object directly: `{"displayName": "...", "sql": "...", ...}` +- **`body: "*"`** -> send the full request message + +The body is the object directly, NOT wrapped in an outer key. + +Fields NOT in the body (like `flag_id`, `parent`) become **query parameters**. + +**Field names are `snake_case`** in requests. Responses may use `camelCase`. + +--- + +## User-Facing Communication Rules + +**NEVER expose internal technical details to the user.** + +- Do NOT show raw JSON request/response bodies in conversation +- Do NOT show Auth0 configuration details, token values, or OAuth internals +- DO show human-readable status updates: "Opening browser for login...", "Creating your warehouse...", "Connectors configured!" +- DO describe results in plain English +- The agent handles all auth/API complexity silently + +**Step Tracker:** Display a visual step tracker at every phase transition. Update and re-display it each time you move to a new step. + +--- + +## Step Tracker + +Display at START and after EACH step completes (updating status): + +``` +───── Setup Warehouse (BigQuery) ────────────────────────── + [1] Choose warehouse ● done + [2] GCP project ID ○ pending + [3] Dataset name ○ pending + [4] Service account ○ pending + [5] Validate & fix ○ pending + [6] Create warehouse ○ pending + [7] Create connectors ○ pending + [8] Assignment table ○ pending + [9] Verify pipeline ○ pending + [10] Done ○ pending +──────────────────────────────────────────────────────────── +``` + +Use `●` for completed, `▶` for in-progress, `○` for pending. Re-display the full tracker after every step transition. + +--- + +## Step 1: Choose warehouse (already done) + +The user has already chosen BigQuery. Mark step 1 as done. + +--- + +## Step 2: GCP Project ID + +Guide the user: + +> What's your GCP project ID? Go to **Google Cloud Console** (console.cloud.google.com). Your project ID is shown in the top bar next to "Google Cloud". It looks like `my-company-prod` or `project-12345`. + +--- + +## Step 3: Dataset name + +> A dataset is like a folder in BigQuery where Confidence stores its tables. The default is `confidence`. +> If you don't have one yet, I can create it for you via `bq mk`. + +Default: `confidence` + +--- + +## Step 4: Service account + +> A service account is a robot account that Confidence uses to write data to your BigQuery dataset. +> Go to **Google Cloud Console -> IAM & Admin -> Service Accounts**. Create one (e.g., `confidence-connector@.iam.gserviceaccount.com`) or pick an existing one. +> It needs **BigQuery Data Editor** and **BigQuery Job User** roles. + +--- + +## Step 5: Validate & fix permissions + +Run the validation endpoint: + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/dataWarehouseConfig:validate" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "bigQueryConfig": { + "serviceAccount": "", + "project": "", + "dataset": "" + } + }' +``` + +**Response:** +```json +{ + "validation": [{ "key": "...", "description": "...", "success": true/false, "error": "..." }], + "successful": true/false, + "configurationResponse": { /* available schemas, etc. */ } +} +``` + +If `successful` is true, move to Step 6. + +**If validation fails:** + +**IMPORTANT: Never assume partial success from an ambiguous error.** If the API returns an error like "X does not exist or not authorized", report the exact error message. Do NOT split it into "connection works but X is missing". Show the user the exact error and let them determine the cause. + +For each validation failure, show: +> Validation failed: `` + +Then offer remediation: + +> Some permissions need to be configured on your GCP project. I can fix this automatically if you have `gcloud` set up, or I can show you the exact commands to run yourself. +> +> 1. Fix it for me (requires gcloud CLI) +> 2. Show me the commands + +### Fix it automatically (gcloud) + +First check gcloud is available: `which gcloud`. If not found, fall back to option 2. + +Extract the account ID from the token claim `https://confidence.dev/account_name` (e.g., `accounts/my-workspace` -> `my-workspace`). The Confidence SA is: `account-${ACCOUNT_ID}@spotify-confidence.iam.gserviceaccount.com` + +For each failure, **confirm before each action:** + +**"Unable to create access token" (SERVICE_ACCOUNT):** +> Confidence needs permission to access your service account. Can I grant that now? +```bash +CONFIDENCE_SA="account-${ACCOUNT_ID}@spotify-confidence.iam.gserviceaccount.com" +gcloud iam service-accounts add-iam-policy-binding ${CUSTOMER_SA} \ + --project=${PROJECT} \ + --member="serviceAccount:${CONFIDENCE_SA}" \ + --role="roles/iam.workloadIdentityUser" --quiet +gcloud iam service-accounts add-iam-policy-binding ${CUSTOMER_SA} \ + --project=${PROJECT} \ + --member="serviceAccount:${CONFIDENCE_SA}" \ + --role="roles/iam.serviceAccountTokenCreator" --quiet +``` + +**"Missing permission 'bigquery.jobs.create'" (PERMISSIONS):** +> Your service account needs BigQuery Job User permissions. Can I grant that? +```bash +gcloud projects add-iam-policy-binding ${PROJECT} \ + --member="serviceAccount:${CUSTOMER_SA}" \ + --role="roles/bigquery.jobUser" --quiet +``` + +**"Could not find dataset" or dataset errors (DATASET):** +> The BigQuery dataset needs to be created or permissions updated. Can I do that? +```bash +bq mk --project_id=${PROJECT} --dataset --location=${REGION} ${DATASET} +bq update --project_id=${PROJECT} --source /dev/stdin ${DATASET} << EOF +{"access": [ + {"role": "WRITER", "userByEmail": "${CUSTOMER_SA}"}, + {"role": "OWNER", "specialGroup": "projectOwners"}, + {"role": "WRITER", "specialGroup": "projectWriters"}, + {"role": "READER", "specialGroup": "projectReaders"} +]} +EOF +``` + +**"free tier" / "Streaming insert is not allowed":** +> BigQuery streaming requires billing enabled on your GCP project. Can I link a billing account? +```bash +gcloud billing accounts list +gcloud billing projects link ${PROJECT} --billing-account=${BILLING_ACCOUNT} +``` +Note: billing propagation to BigQuery can take up to 15 minutes. + +After fixing, re-validate. If still failing (e.g., IAM propagation), inform the user and offer to retry. + +### Show commands (manual) + +Show the exact gcloud/bq commands they need to run, with their specific values filled in: + +``` +Here's what needs to be configured on your GCP project: + +# 1. Grant Confidence access to your service account +gcloud iam service-accounts add-iam-policy-binding \ + \ + --project= \ + --member="serviceAccount:account-@spotify-confidence.iam.gserviceaccount.com" \ + --role="roles/iam.workloadIdentityUser" + +gcloud iam service-accounts add-iam-policy-binding \ + \ + --project= \ + --member="serviceAccount:account-@spotify-confidence.iam.gserviceaccount.com" \ + --role="roles/iam.serviceAccountTokenCreator" + +# 2. Grant BigQuery Job User +gcloud projects add-iam-policy-binding \ + --member="serviceAccount:" \ + --role="roles/bigquery.jobUser" + +# 3. Enable billing (if not already) +gcloud billing projects link --billing-account= + +Run these commands, then let me know and I'll retry validation. +``` + +If `configurationResponse` contains available options (schemas, roles), present these as choices to help the user. + +--- + +## Step 6: Create warehouse + +**IMPORTANT:** The body is the data warehouse object directly (gRPC transcoding `body: "data_warehouse"`), NOT wrapped in a `dataWarehouse` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/dataWarehouses" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "config": { + "bigQueryConfig": { + "serviceAccount": "", + "project": "", + "dataset": "" + } + } + }' +``` + +Save the returned `name` (e.g., `dataWarehouses/...`) for reference. + +--- + +## Step 7: Create connectors + +Create both connectors: + +### Flag Applied Connection (assignment data -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "flag_applied_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/flagAppliedConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "bigQuery": { + "bigQueryConfig": { + "serviceAccount": "", + "project": "", + "dataset": "" + }, + "table": "assignments" + } + }' +``` + +### Event Connection (events -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "event_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/eventConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "bigQuery": { + "bigQueryConfig": { + "serviceAccount": "", + "project": "", + "dataset": "" + }, + "tablePrefix": "events_" + } + }' +``` + +--- + +## Step 8: Assignment table + +Create an assignment table so Confidence can analyze experiment assignments. + +**IMPORTANT:** The body is the assignment table object directly (gRPC transcoding `body: "assignment_table"`), NOT wrapped in an `assignmentTable` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/assignmentTables" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "displayName": "Assignments", + "sql": "SELECT targeting_key, rule, assignment_id, assignment_time FROM `..assignments`", + "entityColumn": { "name": "targeting_key" }, + "timestampColumn": { "name": "assignment_time" }, + "exposureKeyColumn": { "name": "rule" }, + "variantKeyColumn": { "name": "assignment_id" }, + "dataDeliveredUntilUpdateStrategyConfig": { + "strategy": "AUTOMATIC", + "automaticUpdateConfig": { + "commitDelay": "300s" + } + } + }' +``` + +--- + +## Step 9: Verify data pipeline + +Verify both connectors by generating test data and checking it lands in the warehouse. + +### 9a. Get a client secret for testing + +The resolver and events APIs require a **client secret** (not a Bearer token). + +1. **List the user's clients** and show them: + ```bash + curl -s "https://iam.${REGION}.confidence.dev/v1/clients" -H "Authorization: Bearer $TOKEN" + ``` + Display each client with its name and last-seen time. If only one client exists, confirm it with the user. If multiple, let them pick. + +2. **Ask the user** if they have a client secret or want a new one: + > I'll use **** for the pipeline test. Do you have the client secret, or should I create a new credential? + +3. If the user wants a new credential, create one on the chosen client: + ```bash + curl -s -X POST "https://iam.${REGION}.confidence.dev/v1//credentials" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"display_name": "Pipeline Test"}' + ``` + Save the secret to a temp file for pipeline use. **Never print the secret to the user's terminal.** + +### 9b. Verify flag assignments + +Resolve a flag to generate assignment data (use an existing flag + client secret): +```bash +curl -s -X POST "https://resolver.${REGION}.confidence.dev/v1/flags:resolve" \ + -H "Content-Type: application/json" \ + -d '{ + "flags": ["flags/"], + "evaluation_context": {"targeting_key": "warehouse-verify-user"}, + "client_secret": "", + "apply": true + }' +``` + +If no flags exist yet, tell the user: +> No flags to test with. Run `/onboard-confidence setup-wizard` first to create a flag, then come back. + +### 9c. Verify events + +First check for an event definition to use: +```bash +curl -s "https://events.${REGION}.confidence.dev/v1/eventDefinitions" \ + -H "Authorization: Bearer $TOKEN" +``` + +If no event definitions exist, create one with a schema: +```bash +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/eventDefinitions?event_definition_id=test-event" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +If an event definition exists but has an empty schema, update it so payload data flows through: +```bash +curl -s -X PATCH "https://events.${REGION}.confidence.dev/v1/eventDefinitions/" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +Then publish test events (uses client secret, NOT Bearer token): +```bash +NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ") +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/events:publish" \ + -H "Content-Type: application/json" \ + -d '{ + "client_secret": "", + "events": [ + { + "event_definition": "eventDefinitions/", + "payload": {"action": "clicked_button", "page": "homepage"}, + "event_time": "'$NOW'" + } + ], + "send_time": "'$NOW'" + }' +``` + +Check response: `{"errors": []}` means success. If `EVENT_DEFINITION_NOT_FOUND`, the definition doesn't exist. If `EVENT_SCHEMA_VALIDATION_FAILED`, the payload doesn't match the schema. + +### 9d. Check data in BigQuery + +Ask the user: "Want me to check the data, or show you the queries?" + +If user has `bq` CLI: +```bash +echo "=== ASSIGNMENTS ===" && \ +bq query --project_id=${PROJECT} --use_legacy_sql=false \ + 'SELECT targeting_key, rule, assignment_id, assignment_time + FROM `${PROJECT}.${DATASET}.assignments` + ORDER BY assignment_time DESC LIMIT 5' && \ +echo "=== EVENTS ===" && \ +bq query --project_id=${PROJECT} --use_legacy_sql=false \ + 'SELECT * FROM `${PROJECT}.${DATASET}.events_*` + ORDER BY _event_time DESC LIMIT 5' +``` + +If no `bq`, show queries for BigQuery console. + +**Show results:** +``` + ● Assignments: rows -- data flowing + -> () + ● Events: rows -- data flowing + on () +``` + +**If no rows after a few seconds**, tell the user: +> Data delivery can take up to a few minutes depending on your warehouse. Check again shortly, or verify in your BigQuery console. + +--- + +## Step 10: Done + +``` +═══════════════════════════════════════════════════════════════ + Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: BigQuery () + Dataset: + Connectors: + ● Flag assignments -> assignments table (verified) + ● Events -> events_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + +═══════════════════════════════════════════════════════════════ +``` + +--- + +## API Reference (agent-internal -- do NOT show to user) + +### Base URLs + +All APIs require **region-specific URLs**. Extract region from the JWT token claim `https://confidence.dev/region` (value: `EU` or `US`), lowercase it, and use as prefix. + +``` +IAM_API: https://iam.${region}.confidence.dev/v1 +RESOLVER_API: https://resolver.${region}.confidence.dev/v1 +EVENTS_API: https://events.${region}.confidence.dev/v1 +CONNECTORS_API: https://connectors.${region}.confidence.dev/v1 +METRICS_API: https://metrics.${region}.confidence.dev/v1 +``` + +### Endpoints + +**Validate warehouse config (Bearer token):** +``` +POST ${METRICS_API}/dataWarehouseConfig:validate +Body: { "bigQueryConfig": {...} } +-> { "validation": [...], "successful": bool, "configurationResponse": {...} } +``` + +**Check warehouse exists (Bearer token):** +``` +GET ${METRICS_API}/dataWarehouses:exists +-> { "exists": bool } +``` + +**Create data warehouse (Bearer token, body: "data_warehouse"):** +``` +POST ${METRICS_API}/dataWarehouses +Body (direct object): { "config": { "bigQueryConfig": {...} } } +-> DataWarehouse object +``` + +**Create flag applied connection (Bearer token, body: "flag_applied_connection"):** +``` +POST ${CONNECTORS_API}/flagAppliedConnections +Body (direct object): { "bigQuery": { "bigQueryConfig": {...}, "table": "assignments" } } +-> FlagAppliedConnection object +``` + +**Create event connection (Bearer token, body: "event_connection"):** +``` +POST ${CONNECTORS_API}/eventConnections +Body (direct object): { "bigQuery": { "bigQueryConfig": {...}, "tablePrefix": "events_" } } +-> EventConnection object +``` + +**Create assignment table (Bearer token, body: "assignment_table"):** +``` +POST ${METRICS_API}/assignmentTables +Body (direct object): { "displayName": str, "sql": str, "entityColumn": {...}, "timestampColumn": {...}, "exposureKeyColumn": {...}, "variantKeyColumn": {...}, "dataDeliveredUntilUpdateStrategyConfig": {...} } +-> AssignmentTable object +``` + +**List clients (Bearer token):** +``` +GET ${IAM_API}/clients +-> { "clients": [...], "nextPageToken": string } +``` + +**Create client credential (Bearer token, body: "client_credential"):** +``` +POST ${IAM_API}/${clientName}/credentials +Body (direct object): { "display_name": string } +-> { "name": "...", "clientSecret": { "secret": string }, ... } + NOTE: secret only returned once on creation +``` + +**Resolve flags (client secret -- NOT Bearer token):** +``` +POST ${RESOLVER_API}/flags:resolve +Body: { "flags": ["flags/"], "evaluationContext": {...}, "clientSecret": string, "apply": bool } +-> { "resolvedFlags": [...] } +``` + +**Publish events (client secret -- NOT Bearer token):** +``` +POST ${EVENTS_API}/events:publish +Body: { "client_secret": string, "events": [...], "send_time": "ISO8601" } +-> { "errors": [...] } +``` + +--- + +## Error Handling Reference (agent-internal) + +### Common HTTP errors + +| Status | Meaning | Recovery | +|--------|---------|----------| +| 400 | Validation error | Parse `.message`, show plain English, re-collect invalid field | +| 401 | Invalid/expired token | Re-trigger Auth0 login | +| 403 | Insufficient permissions | Explain needed role/permission | +| 404 | Resource not found | Check account/resource exists | +| 409 | Conflict (already exists) | Resource already created | +| 429 | Rate limited | Wait briefly and retry | +| 500+ | Server error | Inform user, suggest retry | + +### Sandbox note + +All `curl`, `open`, and `python3` commands that access external hosts (`auth.confidence.dev`, `metrics.confidence.dev`, `connectors.confidence.dev`, etc.) require `dangerouslyDisableSandbox: true`. On first occurrence, briefly explain to the user that network access outside the sandbox is needed for API calls. diff --git a/skills/setup-warehouse-databricks/SKILL.md b/skills/setup-warehouse-databricks/SKILL.md new file mode 100644 index 0000000..49a6e15 --- /dev/null +++ b/skills/setup-warehouse-databricks/SKILL.md @@ -0,0 +1,860 @@ +--- +description: Set up Databricks as a data warehouse for Confidence. Use when the user chose Databricks for warehouse setup. +--- + +# Setup Warehouse: Databricks + +Configure Databricks as the data warehouse for Confidence experimentation analytics. This skill handles the full end-to-end setup: collect Databricks connection details, set up an S3 staging bucket with IAM, configure the schema, create the warehouse, set up connectors, create the assignment table, and verify the pipeline. + +--- + +## Authentication + +**Browser-based Auth0 login.** The skill opens a browser for Auth0 login (Google, email/password, SSO) and captures the token automatically. The user never touches a token. + +### Session-only token management + +The token is kept in the current session only and is never saved to disk. If the session ends or the token expires, the skill will open your browser to log in again. + +On every sub-command start, check if the `TOKEN` variable is set and not expired: + +```bash +if [ -n "$TOKEN" ]; then + PAYLOAD=$(echo "$TOKEN" | cut -d. -f2) + EXP=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('exp', 0)) +") + NOW=$(date +%s) + if [ "$EXP" -gt "$NOW" ]; then + echo "VALID" + else + echo "EXPIRED" + unset TOKEN + fi +fi +``` + +If `TOKEN` is unset or expired, run the Auth0 login flow with the **regular client ID** (`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w`) and the user's `organization` parameter. Store the result in the `TOKEN` shell variable only. **NEVER write the token to disk. NEVER reference `~/.confidence/`.** + +### Auth script + +Write the following to `$TMPDIR/confidence_auth.py` with CLIENT_ID=`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w` and ORGANIZATION from the token. Run with `python3 $TMPDIR/confidence_auth.py`. Outputs `TOKEN:` on success. + +```python +import http.server, urllib.parse, json, sys, subprocess, hashlib, base64, secrets, string + +code_verifier = ''.join(secrets.choice(string.ascii_letters + string.digits + '-._~') for _ in range(43)) +code_challenge = base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode()).digest()).rstrip(b'=').decode() + +port = 8084 +CLIENT_ID = '' +ORGANIZATION = '' +REDIRECT_URI = f'http://localhost:{port}/callback' +auth_code = None +error = None + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + global auth_code, error + q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query) + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.end_headers() + if 'code' in q: + auth_code = q['code'][0] + self.wfile.write(b'

Login successful!

You can close this tab.

') + else: + error = q.get('error', ['unknown'])[0] + self.wfile.write(b'

Login failed

Please try again.

') + def log_message(self, format, *args): + pass + +params = { + 'client_id': CLIENT_ID, + 'redirect_uri': REDIRECT_URI, + 'response_type': 'code', + 'scope': 'openid profile email offline_access', + 'audience': 'https://confidence.dev/', + 'code_challenge': code_challenge, + 'code_challenge_method': 'S256', +} +if ORGANIZATION: + params['organization'] = ORGANIZATION + +authorize_url = 'https://auth.confidence.dev/authorize?' + urllib.parse.urlencode(params) +subprocess.Popen(['open', authorize_url]) +print('WAITING_FOR_LOGIN', flush=True) + +server = http.server.HTTPServer(('127.0.0.1', port), Handler) +server.timeout = 120 +while auth_code is None and error is None: + server.handle_request() +server.server_close() + +if error: + print(f'AUTH_ERROR:{error}', flush=True) + sys.exit(1) + +import urllib.request +token_data = json.dumps({ + 'grant_type': 'authorization_code', + 'client_id': CLIENT_ID, + 'code': auth_code, + 'redirect_uri': REDIRECT_URI, + 'code_verifier': code_verifier +}).encode() +req = urllib.request.Request( + 'https://auth.confidence.dev/oauth/token', + data=token_data, + headers={'Content-Type': 'application/json'} +) +try: + with urllib.request.urlopen(req) as resp: + token_response = json.loads(resp.read()) + print(f'TOKEN:{token_response["access_token"]}', flush=True) +except Exception as e: + print(f'TOKEN_ERROR:{e}', flush=True) + sys.exit(1) +``` + +### Extract region from token + +```bash +REGION=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('https://confidence.dev/region', 'EU')) +") +``` + +Then use `${REGION,,}` (lowercase) for URL prefix: `iam.eu.confidence.dev`, `metrics.eu.confidence.dev`, etc. + +### Common notes + +- Port is fixed at **8084** (must match Auth0 Allowed Callback URLs) +- If port 8084 is busy: `lsof -ti:8084 | xargs kill -9 2>/dev/null` +- All network commands require `dangerouslyDisableSandbox: true` +- Never show the token value to the user +- Always use region-specific URLs (e.g., `iam.eu.confidence.dev` not `iam.confidence.dev`) + +### Important: gRPC-REST transcoding rules + +The Confidence APIs use gRPC with REST transcoding. The `body` field in the proto HTTP binding determines the JSON structure: + +- **`body: "data_warehouse"`** -> send the data warehouse object directly: `{"config": {...}}` +- **`body: "flag_applied_connection"`** -> send the connection object directly: `{"databricks": {...}}` +- **`body: "event_connection"`** -> send the connection object directly: `{"databricks": {...}}` +- **`body: "assignment_table"`** -> send the assignment table object directly: `{"displayName": "...", "sql": "...", ...}` +- **`body: "*"`** -> send the full request message + +The body is the object directly, NOT wrapped in an outer key. + +Fields NOT in the body (like `flag_id`, `parent`) become **query parameters**. + +**Field names are `snake_case`** in requests. Responses may use `camelCase`. + +--- + +## User-Facing Communication Rules + +**NEVER expose internal technical details to the user.** + +- Do NOT show raw JSON request/response bodies in conversation +- Do NOT show Auth0 configuration details, token values, or OAuth internals +- DO show human-readable status updates: "Opening browser for login...", "Creating your warehouse...", "Connectors configured!" +- DO describe results in plain English +- The agent handles all auth/API complexity silently + +**Step Tracker:** Display a visual step tracker at every phase transition. Update and re-display it each time you move to a new step. + +--- + +## Step Tracker + +Display at START and after EACH step completes (updating status): + +``` +───── Setup Warehouse (Databricks) ──────────────────────── + [1] Choose warehouse ● done + [2] Workspace URL ○ pending + [3] SQL Warehouse ID ○ pending + [4] Service principal ○ pending + [5] AWS account & CLI ○ pending + [6] S3 bucket ○ pending + [7] IAM role ○ pending + [8] Databricks schema ○ pending + [9] Create warehouse ○ pending + [10] Create connectors ○ pending + [11] Assignment table ○ pending + [12] Verify pipeline ○ pending + [13] Done ○ pending +──────────────────────────────────────────────────────────── +``` + +Use `●` for completed, `▶` for in-progress, `○` for pending. Re-display the full tracker after every step transition. + +--- + +## Step 1: Choose warehouse (already done) + +The user has already chosen Databricks. Mark step 1 as done. + +--- + +## Overview + +Before collecting details, explain the full picture so the user knows what they need: + +> Setting up Databricks with Confidence requires three things: +> +> 1. **A Databricks workspace** -- you need admin access to create a service principal (a robot account) +> 2. **An AWS account with an S3 bucket** -- Confidence needs this as a staging area for loading data into Databricks. This is required even if your Databricks runs on GCP or Azure +> 3. **A schema in Databricks** -- a place for Confidence to create tables (e.g., `confidence`) +> +> **How data flows:** +> Confidence collects your flag assignments and events internally, then writes parquet files to an S3 bucket you provide, and finally loads them into Databricks tables. This happens in batches every ~5 minutes. +> +> ``` +> Confidence (collects data) -> S3 bucket (staging) -> Databricks (tables) +> ``` +> +> **Don't have an AWS account?** You'll need one for the S3 staging bucket. AWS free tier works fine. I can set it up for you if you have the `aws` CLI, or walk you through the AWS Console. + +Then collect the details **one at a time**. After each answer, confirm it before moving to the next. Don't dump all questions at once. + +--- + +## Step 2: Workspace URL (Part 1: Databricks connection) + +Ask the user: +> What's your Databricks workspace URL? Just paste the URL from your browser address bar. + +Extract the hostname from whatever they paste (strip `https://`, trailing paths, query params). Valid examples: +- `dbc-a1b2c3d4-e5f6.cloud.databricks.com` +- `1234567890.7.gcp.databricks.com` +- `adb-1234567890.12.azuredatabricks.net` + +Confirm: "Got it -- your Databricks workspace is at ``." + +--- + +## Step 3: SQL Warehouse ID + +Ask the user: +> I need a SQL Warehouse ID. Here's how to find it: +> 1. In Databricks, click **SQL Warehouses** in the left sidebar +> 2. Click on a warehouse name +> 3. Open the **Connection details** tab +> 4. Copy the **HTTP Path** -- the ID is the last part after `/sql/1.0/warehouses/` +> +> It looks like a hex string, e.g., `ccf7028466008a3c` +> +> **Don't have a SQL Warehouse?** Click **Create SQL Warehouse** -> name it "Confidence" -> pick **Serverless**, size **Small** -> **Create**. Then copy the ID. + +Confirm: "Using warehouse ``." + +--- + +## Step 4: Service principal + +Ask the user: +> I need a service principal -- this is a robot account that Confidence uses to connect to Databricks. +> +> **To create one:** +> 1. Click the **gear icon** at the top of Databricks -> **Settings** +> 2. Under **Identity and access**, click **Service principals** +> 3. Click **Add service principal -> Add new** +> 4. Name it "Confidence" -> **Add** +> 5. Click into the new service principal +> 6. Copy the **Application ID** (a UUID like `85cc292a-c1d2-...`) +> 7. Go to the **Secrets** tab -> **Generate secret** +> 8. Copy both the **Secret** (shown only once!) and the **Client ID** +> +> Paste the **Client ID** and **Secret** here. + +If the user says they can't access Settings or service principals: +> You need workspace admin access for this step. Ask your Databricks admin to: +> 1. Create a service principal named "Confidence" +> 2. Generate a secret for it +> 3. Send you the Client ID and Secret + +Confirm: "Service principal configured." + +--- + +## Step 5: AWS account & CLI (Part 2: S3 staging bucket) + +Explain why: +> Confidence writes parquet files to an S3 bucket, then Databricks loads them via COPY INTO. Think of it as a mailbox -- Confidence drops files there, and Databricks picks them up. **This is required even if your Databricks runs on GCP or Azure.** +> +> You need an AWS account for this. If you don't have one, I can help you set one up. + +Ask the user: +> Do you have the `aws` CLI set up, or would you prefer manual steps? +> 1. Set it up for me (requires `aws` CLI) +> 2. Show me the steps + +**If the user picks 1 (aws CLI):** + +First check: `which aws`. If not found, offer to install: `brew install awscli` (macOS) or guide them to https://aws.amazon.com/cli/. + +Then check they're logged in: `aws sts get-caller-identity`. If not, tell them: +> Run `aws configure` or `aws sso login` to log into your AWS account first. + +If `aws` CLI is not configured, the skill should: +1. Open the AWS console login: `open "https://console.aws.amazon.com"` +2. Guide user to create access key: **click your name top right -> Security credentials -> Access keys -> Create access key** +3. Write the credentials directly to `~/.aws/credentials` and `~/.aws/config` (don't use interactive `aws configure`) + +--- + +## Step 6: S3 bucket + +Extract the Confidence service account and its numeric unique ID (required for AWS trust policy): +```bash +ACCOUNT_ID=$(echo "$TOKEN" | cut -d. -f2 | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d['https://confidence.dev/account_name'].split('/')[-1]) +") +CONFIDENCE_SA="account-${ACCOUNT_ID}@spotify-confidence.iam.gserviceaccount.com" +``` + +Ask the user for a bucket name (suggest `confidence-staging-`) and region (suggest `eu-west-1`). + +**If using aws CLI:** + +```bash +# 1. Create S3 bucket +aws s3api create-bucket --bucket ${BUCKET_NAME} --region ${AWS_REGION} \ + --create-bucket-configuration LocationConstraint=${AWS_REGION} +``` + +**If using manual steps:** + +> Go to **AWS Console** (https://console.aws.amazon.com) -> **S3 -> Create bucket**. +> - Name: something like `confidence-staging-` (must be globally unique) +> - Region: pick the same region as your Databricks workspace (e.g., `eu-west-1` for EU) +> - Leave all other settings as default -> **Create bucket** +> +> If you already have a bucket you want to reuse, that works too -- just give me the name. + +--- + +## Step 7: IAM role + +Get the Confidence service account numeric unique ID: +```bash +# CRITICAL: AWS trust policy needs the NUMERIC unique ID, not the email. +# The email won't work -- AWS requires accounts.google.com:sub which is the numeric ID. +SA_UNIQUE_ID=$(gcloud iam service-accounts describe ${CONFIDENCE_SA} \ + --project=spotify-confidence --format="value(uniqueId)") +``` + +If `gcloud` can't access `spotify-confidence` project, the user needs to contact Confidence support to get the numeric service account ID. + +**If using aws CLI:** + +```bash +# 1. Create the trust policy file +# IMPORTANT: Use accounts.google.com:sub with the NUMERIC service account ID. +# Using :email will fail with "MalformedPolicyDocument". +# Using the email string as :sub will fail at runtime with "Not authorized to perform sts:AssumeRoleWithWebIdentity". +cat > $TMPDIR/trust-policy.json << EOF +{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": {"Federated": "accounts.google.com"}, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "accounts.google.com:sub": "${SA_UNIQUE_ID}" + } + } + }] +} +EOF + +# 2. Create IAM role +aws iam create-role --role-name confidence-databricks-staging \ + --assume-role-policy-document file://$TMPDIR/trust-policy.json + +# 3. Create and attach S3 access policy +cat > $TMPDIR/s3-policy.json << EOF +{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"], + "Resource": [ + "arn:aws:s3:::${BUCKET_NAME}", + "arn:aws:s3:::${BUCKET_NAME}/*" + ] + }] +} +EOF +aws iam put-role-policy --role-name confidence-databricks-staging \ + --policy-name S3Access --policy-document file://$TMPDIR/s3-policy.json + +# 4. Get the role ARN +ROLE_ARN=$(aws iam get-role --role-name confidence-databricks-staging --query 'Role.Arn' --output text) +echo "ROLE_ARN: $ROLE_ARN" +``` + +After completion, show the user: +> AWS setup complete! +> - Bucket: `` in `` +> - Role: `` +> +> Continuing with connector setup... + +**If using manual steps:** + +> Go to **AWS Console -> IAM -> Roles -> Create role**. +> - Trusted entity: **Web identity** +> - Identity provider: select **accounts.google.com** (add it first if not listed under Identity providers) +> - Audience: `account-@spotify-confidence.iam.gserviceaccount.com` +> (the skill should compute the account ID from the JWT token and fill this in for the user) +> - Click **Next** -> **Create policy** -> JSON tab -> paste this: +> ```json +> { +> "Version": "2012-10-17", +> "Statement": [{ +> "Effect": "Allow", +> "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"], +> "Resource": ["arn:aws:s3:::", "arn:aws:s3:::/*"] +> }] +> } +> ``` +> - Attach the policy -> name the role (e.g., `confidence-databricks-staging`) -> **Create role** +> - Copy the **Role ARN** (looks like `arn:aws:iam::123456789012:role/confidence-databricks-staging`) +> +> **If you get "Not authorized to perform sts:AssumeRoleWithWebIdentity" later:** the trust policy is wrong -- the Confidence service account email must exactly match what's in the role's trust policy. + +Collect the **AWS Region** and **IAM Role ARN** from the user. + +--- + +## Step 8: Databricks schema (Part 3: schema) + +Ask the user: +> Last thing -- where should Confidence create its tables in Databricks? I need a schema name. +> The default is `confidence`. If you already have a schema you'd like to use, let me know. + +Then check if the schema exists and the service principal has access. Generate the SQL and **copy to clipboard**: + +> I'll set up the schema and permissions. Here's what I'm running -- copied to your clipboard. Paste it in the **Databricks SQL Editor** (left sidebar -> SQL Editor) and run it. + +For workspaces **without Unity Catalog** (hive_metastore): +```sql +CREATE SCHEMA IF NOT EXISTS confidence; +GRANT USE SCHEMA, CREATE TABLE ON SCHEMA confidence TO ``; +``` + +For workspaces **with Unity Catalog**: +```sql +CREATE CATALOG IF NOT EXISTS confidence; +CREATE SCHEMA IF NOT EXISTS confidence.confidence; +GRANT USE CATALOG ON CATALOG confidence TO ``; +GRANT USE SCHEMA, CREATE TABLE ON SCHEMA confidence.confidence TO ``; +``` + +**How to tell which one:** If the user sees **Catalog** in the Databricks left sidebar, they have Unity Catalog. If they only see **Data**, they're on hive_metastore. + +After the user runs it, confirm: "Schema ready. Moving on to create the warehouse." + +--- + +## Step 9: Create warehouse + +**NOTE:** The validate endpoint does NOT support Databricks (returns "configuration must be set" for any field name variant). Skip validation and proceed directly to create. Tell the user: +> Pre-validation isn't available yet for Databricks. I'll create the warehouse now and we'll verify the connection works end-to-end in the pipeline test step. + +**IMPORTANT:** The body is the data warehouse object directly (gRPC transcoding `body: "data_warehouse"`), NOT wrapped in a `dataWarehouse` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/dataWarehouses" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "config": { + "dataBricksConfig": { + "host": "", + "warehouseId": "", + "clientId": "", + "clientSecret": "", + "schema": "", + "s3BucketConfig": { + "bucket": "", + "region": "", + "roleArn": "" + } + } + } + }' +``` + +Save the returned `name` (e.g., `dataWarehouses/...`) for reference. + +--- + +## Step 10: Create connectors + +Create both connectors. Databricks connectors use a nested `connectionConfig` for auth, require an **S3 staging bucket** for batch writes, and `batchFileConfig`. + +### Flag Applied Connection (assignment data -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "flag_applied_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/flagAppliedConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "databricks": { + "databricksConfig": { + "connectionConfig": { + "host": "", + "warehouseId": "", + "clientId": "", + "clientSecret": "" + }, + "schema": "", + "s3BucketConfig": { + "bucket": "", + "region": "", + "roleArn": "" + }, + "batchFileConfig": { + "maxFileAge": "300s" + } + }, + "table": "assignments" + } + }' +``` + +### Event Connection (events -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "event_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/eventConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "databricks": { + "databricksConfig": { + "connectionConfig": { + "host": "", + "warehouseId": "", + "clientId": "", + "clientSecret": "" + }, + "schema": "", + "s3BucketConfig": { + "bucket": "", + "region": "", + "roleArn": "" + }, + "batchFileConfig": { + "maxFileAge": "300s" + } + }, + "tablePrefix": "events_" + } + }' +``` + +--- + +## Step 11: Assignment table + +Create an assignment table so Confidence can analyze experiment assignments. + +**IMPORTANT:** The body is the assignment table object directly (gRPC transcoding `body: "assignment_table"`), NOT wrapped in an `assignmentTable` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/assignmentTables" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "displayName": "Assignments", + "sql": "SELECT targeting_key, rule, assignment_id, assignment_time FROM .assignments", + "entityColumn": { "name": "targeting_key" }, + "timestampColumn": { "name": "assignment_time" }, + "exposureKeyColumn": { "name": "rule" }, + "variantKeyColumn": { "name": "assignment_id" }, + "dataDeliveredUntilUpdateStrategyConfig": { + "strategy": "AUTOMATIC", + "automaticUpdateConfig": { + "commitDelay": "300s" + } + } + }' +``` + +--- + +## Step 12: Verify data pipeline + +Verify both connectors by generating test data and checking it lands in the warehouse. + +### 12a. Get a client secret for testing + +The resolver and events APIs require a **client secret** (not a Bearer token). + +1. **List the user's clients** and show them: + ```bash + curl -s "https://iam.${REGION}.confidence.dev/v1/clients" -H "Authorization: Bearer $TOKEN" + ``` + Display each client with its name and last-seen time. If only one client exists, confirm it with the user. If multiple, let them pick. + +2. **Ask the user** if they have a client secret or want a new one: + > I'll use **** for the pipeline test. Do you have the client secret, or should I create a new credential? + +3. If the user wants a new credential, create one on the chosen client: + ```bash + curl -s -X POST "https://iam.${REGION}.confidence.dev/v1//credentials" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"display_name": "Pipeline Test"}' + ``` + Save the secret to a temp file for pipeline use. **Never print the secret to the user's terminal.** + +### 12b. Verify flag assignments + +Resolve a flag to generate assignment data (use an existing flag + client secret): +```bash +curl -s -X POST "https://resolver.${REGION}.confidence.dev/v1/flags:resolve" \ + -H "Content-Type: application/json" \ + -d '{ + "flags": ["flags/"], + "evaluation_context": {"targeting_key": "warehouse-verify-user"}, + "client_secret": "", + "apply": true + }' +``` + +If no flags exist yet, tell the user: +> No flags to test with. Run `/onboard-confidence setup-wizard` first to create a flag, then come back. + +### 12c. Verify events + +First check for an event definition to use: +```bash +curl -s "https://events.${REGION}.confidence.dev/v1/eventDefinitions" \ + -H "Authorization: Bearer $TOKEN" +``` + +If no event definitions exist, create one with a schema: +```bash +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/eventDefinitions?event_definition_id=test-event" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +If an event definition exists but has an empty schema, update it so payload data flows through: +```bash +curl -s -X PATCH "https://events.${REGION}.confidence.dev/v1/eventDefinitions/" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +Then publish test events (uses client secret, NOT Bearer token): +```bash +NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ") +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/events:publish" \ + -H "Content-Type: application/json" \ + -d '{ + "client_secret": "", + "events": [ + { + "event_definition": "eventDefinitions/", + "payload": {"action": "clicked_button", "page": "homepage"}, + "event_time": "'$NOW'" + } + ], + "send_time": "'$NOW'" + }' +``` + +Check response: `{"errors": []}` means success. If `EVENT_DEFINITION_NOT_FOUND`, the definition doesn't exist. If `EVENT_SCHEMA_VALIDATION_FAILED`, the payload doesn't match the schema. + +### 12d. Check data in Databricks + +Use the Databricks SQL Statement API to query directly (the skill already has the service principal credentials): +```bash +DB_TOKEN=$(curl -s -X POST "https://${DATABRICKS_HOST}/oidc/v1/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&scope=all-apis" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") + +curl -s -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/statements" \ + -H "Authorization: Bearer $DB_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "warehouse_id": "'${WAREHOUSE_ID}'", + "statement": "SELECT targeting_key, rule, assignment_id, assignment_time FROM '${SCHEMA}'.assignments ORDER BY assignment_time DESC LIMIT 5", + "wait_timeout": "30s" + }' +``` + +**IMPORTANT:** Data is batched every ~5 minutes. If the table doesn't exist yet, wait and retry. Tell the user: +> Data delivery takes about 5 minutes. Let me check again... + +If `TABLE_OR_VIEW_NOT_FOUND` after 10 minutes, check the connector logs for errors. + +**Show results:** +``` + ● Assignments: rows -- data flowing + -> () + ● Events: rows -- data flowing + on () +``` + +**If no rows after a few seconds**, tell the user: +> Data delivery can take up to 5 minutes for Databricks (batch processing). Check again shortly, or verify in the Databricks SQL Editor. + +--- + +## Step 13: Done + +``` +═══════════════════════════════════════════════════════════════ + Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: Databricks () + Schema: + S3 Bucket: () + Connectors: + ● Flag assignments -> assignments table (verified) + ● Events -> events_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + + Note: Data is delivered in ~5 minute batches. + +═══════════════════════════════════════════════════════════════ +``` + +--- + +## API Reference (agent-internal -- do NOT show to user) + +### Base URLs + +All APIs require **region-specific URLs**. Extract region from the JWT token claim `https://confidence.dev/region` (value: `EU` or `US`), lowercase it, and use as prefix. + +``` +IAM_API: https://iam.${region}.confidence.dev/v1 +RESOLVER_API: https://resolver.${region}.confidence.dev/v1 +EVENTS_API: https://events.${region}.confidence.dev/v1 +CONNECTORS_API: https://connectors.${region}.confidence.dev/v1 +METRICS_API: https://metrics.${region}.confidence.dev/v1 +``` + +### Endpoints + +**Validate warehouse config (Bearer token):** +``` +POST ${METRICS_API}/dataWarehouseConfig:validate +NOTE: Databricks is NOT supported by the validate endpoint. Skip validation and proceed to create. +``` + +**Check warehouse exists (Bearer token):** +``` +GET ${METRICS_API}/dataWarehouses:exists +-> { "exists": bool } +``` + +**Create data warehouse (Bearer token, body: "data_warehouse"):** +``` +POST ${METRICS_API}/dataWarehouses +Body (direct object): { "config": { "dataBricksConfig": { "host": str, "warehouseId": str, "clientId": str, "clientSecret": str, "schema": str, "s3BucketConfig": { "bucket": str, "region": str, "roleArn": str } } } } +-> DataWarehouse object +``` + +**Create flag applied connection (Bearer token, body: "flag_applied_connection"):** +``` +POST ${CONNECTORS_API}/flagAppliedConnections +Body (direct object): { "databricks": { "databricksConfig": { "connectionConfig": {...}, "schema": str, "s3BucketConfig": {...}, "batchFileConfig": {...} }, "table": "assignments" } } +-> FlagAppliedConnection object +``` + +**Create event connection (Bearer token, body: "event_connection"):** +``` +POST ${CONNECTORS_API}/eventConnections +Body (direct object): { "databricks": { "databricksConfig": { "connectionConfig": {...}, "schema": str, "s3BucketConfig": {...}, "batchFileConfig": {...} }, "tablePrefix": "events_" } } +-> EventConnection object +``` + +**Create assignment table (Bearer token, body: "assignment_table"):** +``` +POST ${METRICS_API}/assignmentTables +Body (direct object): { "displayName": str, "sql": str, "entityColumn": {...}, "timestampColumn": {...}, "exposureKeyColumn": {...}, "variantKeyColumn": {...}, "dataDeliveredUntilUpdateStrategyConfig": {...} } +-> AssignmentTable object +``` + +**List clients (Bearer token):** +``` +GET ${IAM_API}/clients +-> { "clients": [...], "nextPageToken": string } +``` + +**Create client credential (Bearer token, body: "client_credential"):** +``` +POST ${IAM_API}/${clientName}/credentials +Body (direct object): { "display_name": string } +-> { "name": "...", "clientSecret": { "secret": string }, ... } + NOTE: secret only returned once on creation +``` + +**Resolve flags (client secret -- NOT Bearer token):** +``` +POST ${RESOLVER_API}/flags:resolve +Body: { "flags": ["flags/"], "evaluationContext": {...}, "clientSecret": string, "apply": bool } +-> { "resolvedFlags": [...] } +``` + +**Publish events (client secret -- NOT Bearer token):** +``` +POST ${EVENTS_API}/events:publish +Body: { "client_secret": string, "events": [...], "send_time": "ISO8601" } +-> { "errors": [...] } +``` + +--- + +## Error Handling Reference (agent-internal) + +### Common HTTP errors + +| Status | Meaning | Recovery | +|--------|---------|----------| +| 400 | Validation error | Parse `.message`, show plain English, re-collect invalid field | +| 401 | Invalid/expired token | Re-trigger Auth0 login | +| 403 | Insufficient permissions | Explain needed role/permission | +| 404 | Resource not found | Check account/resource exists | +| 409 | Conflict (already exists) | Resource already created | +| 429 | Rate limited | Wait briefly and retry | +| 500+ | Server error | Inform user, suggest retry | + +### Sandbox note + +All `curl`, `open`, `python3`, `aws`, and `gcloud` commands that access external hosts (`auth.confidence.dev`, `metrics.confidence.dev`, `connectors.confidence.dev`, AWS APIs, etc.) require `dangerouslyDisableSandbox: true`. On first occurrence, briefly explain to the user that network access outside the sandbox is needed for API calls. diff --git a/skills/setup-warehouse-redshift/SKILL.md b/skills/setup-warehouse-redshift/SKILL.md new file mode 100644 index 0000000..53d8647 --- /dev/null +++ b/skills/setup-warehouse-redshift/SKILL.md @@ -0,0 +1,869 @@ +--- +description: Set up Redshift as a data warehouse for Confidence. Use when the user chose Redshift for warehouse setup. +--- + +# Setup Warehouse: Redshift + +Configure Redshift as the data warehouse for Confidence experimentation analytics. This skill handles the full end-to-end setup: set up or connect a Redshift cluster, create an S3 staging bucket with IAM, configure the schema, create the warehouse, set up connectors, create the assignment table, and verify the pipeline. + +--- + +## Authentication + +**Browser-based Auth0 login.** The skill opens a browser for Auth0 login (Google, email/password, SSO) and captures the token automatically. The user never touches a token. + +### Session-only token management + +The token is kept in the current session only and is never saved to disk. If the session ends or the token expires, the skill will open your browser to log in again. + +On every sub-command start, check if the `TOKEN` variable is set and not expired: + +```bash +if [ -n "$TOKEN" ]; then + PAYLOAD=$(echo "$TOKEN" | cut -d. -f2) + EXP=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('exp', 0)) +") + NOW=$(date +%s) + if [ "$EXP" -gt "$NOW" ]; then + echo "VALID" + else + echo "EXPIRED" + unset TOKEN + fi +fi +``` + +If `TOKEN` is unset or expired, run the Auth0 login flow with the **regular client ID** (`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w`) and the user's `organization` parameter. Store the result in the `TOKEN` shell variable only. **NEVER write the token to disk. NEVER reference `~/.confidence/`.** + +### Auth script + +Write the following to `$TMPDIR/confidence_auth.py` with CLIENT_ID=`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w` and ORGANIZATION from the token. Run with `python3 $TMPDIR/confidence_auth.py`. Outputs `TOKEN:` on success. + +```python +import http.server, urllib.parse, json, sys, subprocess, hashlib, base64, secrets, string + +code_verifier = ''.join(secrets.choice(string.ascii_letters + string.digits + '-._~') for _ in range(43)) +code_challenge = base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode()).digest()).rstrip(b'=').decode() + +port = 8084 +CLIENT_ID = '' +ORGANIZATION = '' +REDIRECT_URI = f'http://localhost:{port}/callback' +auth_code = None +error = None + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + global auth_code, error + q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query) + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.end_headers() + if 'code' in q: + auth_code = q['code'][0] + self.wfile.write(b'

Login successful!

You can close this tab.

') + else: + error = q.get('error', ['unknown'])[0] + self.wfile.write(b'

Login failed

Please try again.

') + def log_message(self, format, *args): + pass + +params = { + 'client_id': CLIENT_ID, + 'redirect_uri': REDIRECT_URI, + 'response_type': 'code', + 'scope': 'openid profile email offline_access', + 'audience': 'https://confidence.dev/', + 'code_challenge': code_challenge, + 'code_challenge_method': 'S256', +} +if ORGANIZATION: + params['organization'] = ORGANIZATION + +authorize_url = 'https://auth.confidence.dev/authorize?' + urllib.parse.urlencode(params) +subprocess.Popen(['open', authorize_url]) +print('WAITING_FOR_LOGIN', flush=True) + +server = http.server.HTTPServer(('127.0.0.1', port), Handler) +server.timeout = 120 +while auth_code is None and error is None: + server.handle_request() +server.server_close() + +if error: + print(f'AUTH_ERROR:{error}', flush=True) + sys.exit(1) + +import urllib.request +token_data = json.dumps({ + 'grant_type': 'authorization_code', + 'client_id': CLIENT_ID, + 'code': auth_code, + 'redirect_uri': REDIRECT_URI, + 'code_verifier': code_verifier +}).encode() +req = urllib.request.Request( + 'https://auth.confidence.dev/oauth/token', + data=token_data, + headers={'Content-Type': 'application/json'} +) +try: + with urllib.request.urlopen(req) as resp: + token_response = json.loads(resp.read()) + print(f'TOKEN:{token_response["access_token"]}', flush=True) +except Exception as e: + print(f'TOKEN_ERROR:{e}', flush=True) + sys.exit(1) +``` + +### Extract region from token + +```bash +REGION=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('https://confidence.dev/region', 'EU')) +") +``` + +Then use `${REGION,,}` (lowercase) for URL prefix: `iam.eu.confidence.dev`, `metrics.eu.confidence.dev`, etc. + +### Common notes + +- Port is fixed at **8084** (must match Auth0 Allowed Callback URLs) +- If port 8084 is busy: `lsof -ti:8084 | xargs kill -9 2>/dev/null` +- All network commands require `dangerouslyDisableSandbox: true` +- Never show the token value to the user +- Always use region-specific URLs (e.g., `iam.eu.confidence.dev` not `iam.confidence.dev`) + +### Important: gRPC-REST transcoding rules + +The Confidence APIs use gRPC with REST transcoding. The `body` field in the proto HTTP binding determines the JSON structure: + +- **`body: "data_warehouse"`** -> send the data warehouse object directly: `{"config": {...}}` +- **`body: "flag_applied_connection"`** -> send the connection object directly: `{"redshift": {...}}` +- **`body: "event_connection"`** -> send the connection object directly: `{"redshift": {...}}` +- **`body: "assignment_table"`** -> send the assignment table object directly: `{"displayName": "...", "sql": "...", ...}` +- **`body: "*"`** -> send the full request message + +The body is the object directly, NOT wrapped in an outer key. + +Fields NOT in the body (like `flag_id`, `parent`) become **query parameters**. + +**Field names are `snake_case`** in requests. Responses may use `camelCase`. + +--- + +## User-Facing Communication Rules + +**NEVER expose internal technical details to the user.** + +- Do NOT show raw JSON request/response bodies in conversation +- Do NOT show Auth0 configuration details, token values, or OAuth internals +- DO show human-readable status updates: "Opening browser for login...", "Creating your warehouse...", "Connectors configured!" +- DO describe results in plain English +- The agent handles all auth/API complexity silently + +**Step Tracker:** Display a visual step tracker at every phase transition. Update and re-display it each time you move to a new step. + +--- + +## Step Tracker + +Display at START and after EACH step completes (updating status): + +``` +───── Setup Warehouse (Redshift) ────────────────────────── + [1] Choose warehouse ● done + [2] AWS account & CLI ○ pending + [3] Redshift cluster ○ pending + [4] S3 bucket ○ pending + [5] IAM role ○ pending + [6] Attach role ○ pending + [7] Schema & grants ○ pending + [8] Validate ○ pending + [9] Create warehouse ○ pending + [10] Create connectors ○ pending + [11] Assignment table ○ pending + [12] Verify pipeline ○ pending + [13] Done ○ pending +──────────────────────────────────────────────────────────── +``` + +Use `●` for completed, `▶` for in-progress, `○` for pending. Re-display the full tracker after every step transition. + +--- + +## Step 1: Choose warehouse (already done) + +The user has already chosen Redshift. Mark step 1 as done. + +--- + +## Overview + +Before collecting details, explain the full picture so the user knows what they're signing up for: + +> Setting up Redshift with Confidence requires an **AWS account**. Here's what we'll set up: +> +> 1. **A Redshift cluster** -- a data warehouse that stores your experiment data +> 2. **An S3 bucket** -- a staging area where Confidence drops data files before loading them into Redshift +> 3. **An IAM role** -- permissions that let Confidence write to S3 and load into Redshift +> 4. **A schema** -- a folder inside Redshift where Confidence creates its tables +> +> **How data flows:** +> ``` +> Confidence -> S3 bucket (staging) -> Redshift COPY -> your tables +> ``` +> +> I can set up everything automatically if you have the `aws` CLI, or walk you through the AWS Console step by step. +> +> **Don't have an AWS account?** You'll need one. I can open the signup page for you. AWS free tier covers S3, but Redshift clusters cost ~$0.25/hr while running. You can delete it after testing. +> +> **Important: Redshift Serverless won't work** -- Confidence needs a provisioned cluster. I'll make sure we create the right type. + +Ask the user: +> Do you have the `aws` CLI set up, or would you prefer manual steps? +> 1. Set it up for me (requires `aws` CLI) +> 2. Show me the steps + +--- + +## Step 2: AWS account & CLI + +**If the user picks 1 (aws CLI):** + +Check `which aws`. If not found: `brew install awscli` (macOS). +Check `aws sts get-caller-identity`. If not logged in, open the AWS console login (`open "https://console.aws.amazon.com"`), guide them to create access keys (**click name top right -> Security credentials -> Access keys -> Create**), then write the credentials to `~/.aws/credentials` and `~/.aws/config`. + +**If the user picks 2 (manual steps):** + +Walk them through the AWS Console for each subsequent step. Each step below includes both CLI and manual instructions. + +--- + +## Step 3: Redshift cluster + +Ask the user: +> Do you already have a Redshift cluster, or should I create one? + +If they have one: +> What's the cluster name? Go to **AWS Console -> Amazon Redshift -> Clusters**. The name is in the first column. + +If they need one, explain: +> I'll create a single-node Redshift cluster. This is a data warehouse -- like a powerful database optimized for analytics. +> - **Cost:** ~$0.25/hour while running. Delete it when you're done testing. +> - **Type:** `ra3.large` (cheapest option that supports single-node) +> - **Region:** `eu-west-1` (Europe) -- should match where your Confidence account is +> +> **Important:** Redshift Serverless won't work -- Confidence needs a provisioned cluster. I'll create the right type. + +Extract the account ID from the token: +```bash +ACCOUNT_ID=$(echo "$TOKEN" | cut -d. -f2 | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d['https://confidence.dev/account_name'].split('/')[-1]) +") +``` + +**If using aws CLI:** + +```bash +aws redshift create-cluster \ + --cluster-identifier confidence-redshift-${ACCOUNT_ID} \ + --cluster-type single-node \ + --node-type ra3.large \ + --master-username admin \ + --master-user-password '' \ + --db-name dev \ + --region eu-west-1 \ + --publicly-accessible +``` + +Wait for status `available` (takes ~1-2 minutes): +```bash +aws redshift wait cluster-available --cluster-identifier ${CLUSTER} --region ${AWS_REGION} +``` + +Confirm: "Redshift cluster `` is running." + +**If using manual steps:** + +> Go to **AWS Console -> Amazon Redshift -> Create cluster** -> single-node, ra3.large, database `dev`, publicly accessible. + +--- + +## Step 4: S3 bucket + +Ask the user: +> Do you have an S3 bucket I should use, or should I create one? + +**If using aws CLI:** + +```bash +aws s3api create-bucket --bucket confidence-redshift-${ACCOUNT_ID} \ + --region ${AWS_REGION} \ + --create-bucket-configuration LocationConstraint=${AWS_REGION} +``` + +Confirm: "S3 bucket `` created in ``." + +**If using manual steps:** + +> Go to **AWS Console -> S3 -> Create bucket** -> name it, pick same region as cluster. + +--- + +## Step 5: IAM role + +Get the Confidence service account numeric ID: +```bash +CONFIDENCE_SA="account-${ACCOUNT_ID}@spotify-confidence.iam.gserviceaccount.com" + +# CRITICAL: AWS trust policy needs the NUMERIC unique ID, not the email. +SA_UNIQUE_ID=$(gcloud iam service-accounts describe ${CONFIDENCE_SA} \ + --project=spotify-confidence --format="value(uniqueId)") +``` + +If `gcloud` can't access `spotify-confidence` project, the user needs to contact Confidence support to get the numeric service account ID. + +**If using aws CLI:** + +Create the role with dual trust (Google OIDC + Redshift): +```bash +cat > $TMPDIR/redshift-trust.json << EOF +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": {"Federated": "accounts.google.com"}, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "accounts.google.com:sub": "${SA_UNIQUE_ID}" + } + } + }, + { + "Effect": "Allow", + "Principal": {"Service": "redshift.amazonaws.com"}, + "Action": "sts:AssumeRole" + } + ] +} +EOF +aws iam create-role --role-name confidence-redshift \ + --assume-role-policy-document file://$TMPDIR/redshift-trust.json +``` + +Attach S3 + Redshift Data API permissions: +```bash +# S3 write access +cat > $TMPDIR/s3-policy.json << EOF +{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:PutObject","s3:GetObject","s3:DeleteObject","s3:ListBucket"],"Resource":["arn:aws:s3:::${BUCKET_NAME}","arn:aws:s3:::${BUCKET_NAME}/*"]}]} +EOF +aws iam put-role-policy --role-name confidence-redshift \ + --policy-name S3Access --policy-document file://$TMPDIR/s3-policy.json + +# Redshift Data API access +cat > $TMPDIR/redshift-data-policy.json << EOF +{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["redshift-data:*","redshift:GetClusterCredentials","redshift:GetClusterCredentialsWithIAM","redshift:DescribeClusters"],"Resource":"*"}]} +EOF +aws iam put-role-policy --role-name confidence-redshift \ + --policy-name RedshiftAccess --policy-document file://$TMPDIR/redshift-data-policy.json +``` + +Get the role ARN: +```bash +ROLE_ARN=$(aws iam get-role --role-name confidence-redshift --query 'Role.Arn' --output text) +``` + +**If using manual steps:** + +> Go to **AWS Console -> IAM -> Roles -> Create role** -> two trust steps: +> - Add **Web identity** trust with `accounts.google.com`, sub = `` (compute and display for the user) +> - Add **AWS service** trust for `redshift.amazonaws.com` +> - Attach policies: custom S3 policy scoped to bucket + `AmazonRedshiftDataFullAccess` +> - Copy the **Role ARN** + +--- + +## Step 6: Attach role to cluster + +**CRITICAL:** Attach the role to the Redshift cluster -- without this, the COPY command can't read from S3: + +**If using aws CLI:** + +```bash +aws redshift modify-cluster-iam-roles \ + --cluster-identifier ${CLUSTER} \ + --add-iam-roles ${ROLE_ARN} --region ${AWS_REGION} +``` + +Wait for `in-sync`: +```bash +aws redshift describe-clusters --cluster-identifier ${CLUSTER} --region ${AWS_REGION} \ + --query "Clusters[0].IamRoles[*].{Role:IamRoleArn,Status:ApplyStatus}" --output table +``` + +Confirm: "IAM role created and attached to cluster." + +**If using manual steps:** + +> Go back to **Redshift -> Clusters -> your cluster -> Properties -> Manage IAM roles -> Add the new role** + +--- + +## Step 7: Schema & grants + +Ask the user: +> What should the schema be called? The default is `confidence`. + +Create the schema and grant permissions so Confidence can see it: + +**If using aws CLI:** + +```bash +aws redshift-data execute-statement \ + --cluster-identifier ${CLUSTER} --database ${DATABASE} --db-user admin \ + --sql "CREATE SCHEMA IF NOT EXISTS ${SCHEMA}; GRANT USAGE ON SCHEMA ${SCHEMA} TO PUBLIC; GRANT CREATE ON SCHEMA ${SCHEMA} TO PUBLIC;" \ + --region ${AWS_REGION} +``` + +**IMPORTANT:** `GRANT USAGE ON SCHEMA ... TO PUBLIC` is required -- without it, Confidence's validation returns "Schema not found" even though the schema exists. This is because Confidence connects via IAM, not as the `admin` user. + +Confirm: "Schema `` created with permissions." + +**If using manual steps:** + +> Go to **Redshift -> Query editor v2** -> connect to cluster -> run: +> ```sql +> CREATE SCHEMA IF NOT EXISTS confidence; +> GRANT USAGE ON SCHEMA confidence TO PUBLIC; +> GRANT CREATE ON SCHEMA confidence TO PUBLIC; +> ``` + +Copy the SQL to clipboard for the user. + +--- + +## Step 8: Validate + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/dataWarehouseConfig:validate" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "redshiftConfig": { + "clusterIdentifier": "", + "database": "", + "schema": "", + "region": "", + "roleArn": "" + } + }' +``` + +**Response:** +```json +{ + "validation": [{ "key": "...", "description": "...", "success": true/false, "error": "..." }], + "successful": true/false, + "configurationResponse": { /* type-specific */ } +} +``` + +If `successful` is true, move to Step 9. + +**If validation fails:** + +**IMPORTANT: Never assume partial success from an ambiguous error.** If the API returns an error like "X does not exist or not authorized", report the exact error message. Do NOT split it into "connection works but X is missing". Show the user the exact error and let them determine the cause. + +For each validation failure, show: +> Validation failed: `` + +Then show the relevant remediation steps: + +- **Schema not found** -> Ensure `GRANT USAGE ON SCHEMA ... TO PUBLIC` was run (Step 7) +- **IAM role errors** -> Check the trust policy has both `accounts.google.com` and `redshift.amazonaws.com` principals +- **S3 access errors** -> Check the S3 policy is attached to the role and scoped to the correct bucket +- **Cluster not found** -> Verify the cluster identifier and region + +--- + +## Step 9: Create warehouse + +**IMPORTANT:** The body is the data warehouse object directly (gRPC transcoding `body: "data_warehouse"`), NOT wrapped in a `dataWarehouse` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/dataWarehouses" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "config": { + "redshiftConfig": { + "clusterIdentifier": "", + "database": "", + "schema": "", + "region": "", + "roleArn": "" + } + } + }' +``` + +Save the returned `name` (e.g., `dataWarehouses/...`) for reference. + +--- + +## Step 10: Create connectors + +Create both connectors. Redshift connectors require `redshiftConfig`, `s3Config`, and `batchFileConfig`. + +### Flag Applied Connection (assignment data -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "flag_applied_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/flagAppliedConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "redshift": { + "redshiftConfig": { + "clusterIdentifier": "", + "database": "", + "schema": "", + "region": "", + "roleArn": "" + }, + "s3Config": { + "bucket": "", + "region": "", + "roleArn": "" + }, + "batchFileConfig": { + "maxEventsPerFile": 10000, + "maxFileAge": "300s", + "maxFileSize": 104857600 + }, + "table": "assignments" + } + }' +``` + +### Event Connection (events -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "event_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/eventConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "redshift": { + "redshiftConfig": { + "clusterIdentifier": "", + "database": "", + "schema": "", + "region": "", + "roleArn": "" + }, + "s3Config": { + "bucket": "", + "region": "", + "roleArn": "" + }, + "batchFileConfig": { + "maxEventsPerFile": 10000, + "maxFileAge": "300s", + "maxFileSize": 104857600 + }, + "tablePrefix": "events_" + } + }' +``` + +--- + +## Step 11: Assignment table + +Create an assignment table so Confidence can analyze experiment assignments. + +**IMPORTANT:** The body is the assignment table object directly (gRPC transcoding `body: "assignment_table"`), NOT wrapped in an `assignmentTable` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/assignmentTables" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "displayName": "Assignments", + "sql": "SELECT targeting_key, rule, assignment_id, assignment_time FROM .assignments", + "entityColumn": { "name": "targeting_key" }, + "timestampColumn": { "name": "assignment_time" }, + "exposureKeyColumn": { "name": "rule" }, + "variantKeyColumn": { "name": "assignment_id" }, + "dataDeliveredUntilUpdateStrategyConfig": { + "strategy": "AUTOMATIC", + "automaticUpdateConfig": { + "commitDelay": "300s" + } + } + }' +``` + +--- + +## Step 12: Verify data pipeline + +Verify both connectors by generating test data and checking it lands in the warehouse. + +### 12a. Get a client secret for testing + +The resolver and events APIs require a **client secret** (not a Bearer token). + +1. **List the user's clients** and show them: + ```bash + curl -s "https://iam.${REGION}.confidence.dev/v1/clients" -H "Authorization: Bearer $TOKEN" + ``` + Display each client with its name and last-seen time. If only one client exists, confirm it with the user. If multiple, let them pick. + +2. **Ask the user** if they have a client secret or want a new one: + > I'll use **** for the pipeline test. Do you have the client secret, or should I create a new credential? + +3. If the user wants a new credential, create one on the chosen client: + ```bash + curl -s -X POST "https://iam.${REGION}.confidence.dev/v1//credentials" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"display_name": "Pipeline Test"}' + ``` + Save the secret to a temp file for pipeline use. **Never print the secret to the user's terminal.** + +### 12b. Verify flag assignments + +Resolve a flag to generate assignment data (use an existing flag + client secret): +```bash +curl -s -X POST "https://resolver.${REGION}.confidence.dev/v1/flags:resolve" \ + -H "Content-Type: application/json" \ + -d '{ + "flags": ["flags/"], + "evaluation_context": {"targeting_key": "warehouse-verify-user"}, + "client_secret": "", + "apply": true + }' +``` + +If no flags exist yet, tell the user: +> No flags to test with. Run `/onboard-confidence setup-wizard` first to create a flag, then come back. + +### 12c. Verify events + +First check for an event definition to use: +```bash +curl -s "https://events.${REGION}.confidence.dev/v1/eventDefinitions" \ + -H "Authorization: Bearer $TOKEN" +``` + +If no event definitions exist, create one with a schema: +```bash +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/eventDefinitions?event_definition_id=test-event" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +If an event definition exists but has an empty schema, update it so payload data flows through: +```bash +curl -s -X PATCH "https://events.${REGION}.confidence.dev/v1/eventDefinitions/" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +Then publish test events (uses client secret, NOT Bearer token): +```bash +NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ") +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/events:publish" \ + -H "Content-Type: application/json" \ + -d '{ + "client_secret": "", + "events": [ + { + "event_definition": "eventDefinitions/", + "payload": {"action": "clicked_button", "page": "homepage"}, + "event_time": "'$NOW'" + } + ], + "send_time": "'$NOW'" + }' +``` + +Check response: `{"errors": []}` means success. If `EVENT_DEFINITION_NOT_FOUND`, the definition doesn't exist. If `EVENT_SCHEMA_VALIDATION_FAILED`, the payload doesn't match the schema. + +### 12d. Check data in Redshift + +If user has `aws redshift-data`: +```bash +aws redshift-data execute-statement \ + --cluster-identifier ${CLUSTER} \ + --database ${DATABASE} \ + --db-user ${DB_USER} \ + --sql "SELECT targeting_key, rule, assignment_id, assignment_time FROM ${SCHEMA}.assignments ORDER BY assignment_time DESC LIMIT 5" +``` + +Otherwise, show queries for the Redshift query editor. + +**Show results:** +``` + ● Assignments: rows -- data flowing + -> () + ● Events: rows -- data flowing + on () +``` + +**If no rows after a few seconds**, tell the user: +> Data delivery can take up to a few minutes depending on your warehouse. Check again shortly, or verify in your Redshift query editor. + +--- + +## Step 13: Done + +``` +═══════════════════════════════════════════════════════════════ + Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: Redshift () + Database: + Schema: + S3 Bucket: () + Connectors: + ● Flag assignments -> assignments table (verified) + ● Events -> events_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + +═══════════════════════════════════════════════════════════════ +``` + +--- + +## API Reference (agent-internal -- do NOT show to user) + +### Base URLs + +All APIs require **region-specific URLs**. Extract region from the JWT token claim `https://confidence.dev/region` (value: `EU` or `US`), lowercase it, and use as prefix. + +``` +IAM_API: https://iam.${region}.confidence.dev/v1 +RESOLVER_API: https://resolver.${region}.confidence.dev/v1 +EVENTS_API: https://events.${region}.confidence.dev/v1 +CONNECTORS_API: https://connectors.${region}.confidence.dev/v1 +METRICS_API: https://metrics.${region}.confidence.dev/v1 +``` + +### Endpoints + +**Validate warehouse config (Bearer token):** +``` +POST ${METRICS_API}/dataWarehouseConfig:validate +Body: { "redshiftConfig": { "clusterIdentifier": str, "database": str, "schema": str, "region": str, "roleArn": str } } +-> { "validation": [...], "successful": bool, "configurationResponse": {...} } +``` + +**Check warehouse exists (Bearer token):** +``` +GET ${METRICS_API}/dataWarehouses:exists +-> { "exists": bool } +``` + +**Create data warehouse (Bearer token, body: "data_warehouse"):** +``` +POST ${METRICS_API}/dataWarehouses +Body (direct object): { "config": { "redshiftConfig": { "clusterIdentifier": str, "database": str, "schema": str, "region": str, "roleArn": str } } } +-> DataWarehouse object +``` + +**Create flag applied connection (Bearer token, body: "flag_applied_connection"):** +``` +POST ${CONNECTORS_API}/flagAppliedConnections +Body (direct object): { "redshift": { "redshiftConfig": {...}, "s3Config": {...}, "batchFileConfig": {...}, "table": "assignments" } } +-> FlagAppliedConnection object +``` + +**Create event connection (Bearer token, body: "event_connection"):** +``` +POST ${CONNECTORS_API}/eventConnections +Body (direct object): { "redshift": { "redshiftConfig": {...}, "s3Config": {...}, "batchFileConfig": {...}, "tablePrefix": "events_" } } +-> EventConnection object +``` + +**Create assignment table (Bearer token, body: "assignment_table"):** +``` +POST ${METRICS_API}/assignmentTables +Body (direct object): { "displayName": str, "sql": str, "entityColumn": {...}, "timestampColumn": {...}, "exposureKeyColumn": {...}, "variantKeyColumn": {...}, "dataDeliveredUntilUpdateStrategyConfig": {...} } +-> AssignmentTable object +``` + +**List clients (Bearer token):** +``` +GET ${IAM_API}/clients +-> { "clients": [...], "nextPageToken": string } +``` + +**Create client credential (Bearer token, body: "client_credential"):** +``` +POST ${IAM_API}/${clientName}/credentials +Body (direct object): { "display_name": string } +-> { "name": "...", "clientSecret": { "secret": string }, ... } + NOTE: secret only returned once on creation +``` + +**Resolve flags (client secret -- NOT Bearer token):** +``` +POST ${RESOLVER_API}/flags:resolve +Body: { "flags": ["flags/"], "evaluationContext": {...}, "clientSecret": string, "apply": bool } +-> { "resolvedFlags": [...] } +``` + +**Publish events (client secret -- NOT Bearer token):** +``` +POST ${EVENTS_API}/events:publish +Body: { "client_secret": string, "events": [...], "send_time": "ISO8601" } +-> { "errors": [...] } +``` + +--- + +## Error Handling Reference (agent-internal) + +### Common HTTP errors + +| Status | Meaning | Recovery | +|--------|---------|----------| +| 400 | Validation error | Parse `.message`, show plain English, re-collect invalid field | +| 401 | Invalid/expired token | Re-trigger Auth0 login | +| 403 | Insufficient permissions | Explain needed role/permission | +| 404 | Resource not found | Check account/resource exists | +| 409 | Conflict (already exists) | Resource already created | +| 429 | Rate limited | Wait briefly and retry | +| 500+ | Server error | Inform user, suggest retry | + +### Sandbox note + +All `curl`, `open`, `python3`, `aws`, and `gcloud` commands that access external hosts (`auth.confidence.dev`, `metrics.confidence.dev`, `connectors.confidence.dev`, AWS APIs, etc.) require `dangerouslyDisableSandbox: true`. On first occurrence, briefly explain to the user that network access outside the sandbox is needed for API calls. diff --git a/skills/setup-warehouse-snowflake/SKILL.md b/skills/setup-warehouse-snowflake/SKILL.md new file mode 100644 index 0000000..66eb904 --- /dev/null +++ b/skills/setup-warehouse-snowflake/SKILL.md @@ -0,0 +1,753 @@ +--- +description: Set up Snowflake as a data warehouse for Confidence. Use when the user chose Snowflake for warehouse setup. +--- + +# Setup Warehouse: Snowflake + +Configure Snowflake as the data warehouse for Confidence experimentation analytics. This skill handles the full end-to-end setup: collect Snowflake config, create a crypto key, register the key in Snowflake, validate, create the warehouse, set up connectors, create the assignment table, and verify the pipeline. + +--- + +## Authentication + +**Browser-based Auth0 login.** The skill opens a browser for Auth0 login (Google, email/password, SSO) and captures the token automatically. The user never touches a token. + +### Session-only token management + +The token is kept in the current session only and is never saved to disk. If the session ends or the token expires, the skill will open your browser to log in again. + +On every sub-command start, check if the `TOKEN` variable is set and not expired: + +```bash +if [ -n "$TOKEN" ]; then + PAYLOAD=$(echo "$TOKEN" | cut -d. -f2) + EXP=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('exp', 0)) +") + NOW=$(date +%s) + if [ "$EXP" -gt "$NOW" ]; then + echo "VALID" + else + echo "EXPIRED" + unset TOKEN + fi +fi +``` + +If `TOKEN` is unset or expired, run the Auth0 login flow with the **regular client ID** (`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w`) and the user's `organization` parameter. Store the result in the `TOKEN` shell variable only. **NEVER write the token to disk. NEVER reference `~/.confidence/`.** + +### Auth script + +Write the following to `$TMPDIR/confidence_auth.py` with CLIENT_ID=`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w` and ORGANIZATION from the token. Run with `python3 $TMPDIR/confidence_auth.py`. Outputs `TOKEN:` on success. + +```python +import http.server, urllib.parse, json, sys, subprocess, hashlib, base64, secrets, string + +code_verifier = ''.join(secrets.choice(string.ascii_letters + string.digits + '-._~') for _ in range(43)) +code_challenge = base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode()).digest()).rstrip(b'=').decode() + +port = 8084 +CLIENT_ID = '' +ORGANIZATION = '' +REDIRECT_URI = f'http://localhost:{port}/callback' +auth_code = None +error = None + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + global auth_code, error + q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query) + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.end_headers() + if 'code' in q: + auth_code = q['code'][0] + self.wfile.write(b'

Login successful!

You can close this tab.

') + else: + error = q.get('error', ['unknown'])[0] + self.wfile.write(b'

Login failed

Please try again.

') + def log_message(self, format, *args): + pass + +params = { + 'client_id': CLIENT_ID, + 'redirect_uri': REDIRECT_URI, + 'response_type': 'code', + 'scope': 'openid profile email offline_access', + 'audience': 'https://confidence.dev/', + 'code_challenge': code_challenge, + 'code_challenge_method': 'S256', +} +if ORGANIZATION: + params['organization'] = ORGANIZATION + +authorize_url = 'https://auth.confidence.dev/authorize?' + urllib.parse.urlencode(params) +subprocess.Popen(['open', authorize_url]) +print('WAITING_FOR_LOGIN', flush=True) + +server = http.server.HTTPServer(('127.0.0.1', port), Handler) +server.timeout = 120 +while auth_code is None and error is None: + server.handle_request() +server.server_close() + +if error: + print(f'AUTH_ERROR:{error}', flush=True) + sys.exit(1) + +import urllib.request +token_data = json.dumps({ + 'grant_type': 'authorization_code', + 'client_id': CLIENT_ID, + 'code': auth_code, + 'redirect_uri': REDIRECT_URI, + 'code_verifier': code_verifier +}).encode() +req = urllib.request.Request( + 'https://auth.confidence.dev/oauth/token', + data=token_data, + headers={'Content-Type': 'application/json'} +) +try: + with urllib.request.urlopen(req) as resp: + token_response = json.loads(resp.read()) + print(f'TOKEN:{token_response["access_token"]}', flush=True) +except Exception as e: + print(f'TOKEN_ERROR:{e}', flush=True) + sys.exit(1) +``` + +### Extract region from token + +```bash +REGION=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('https://confidence.dev/region', 'EU')) +") +``` + +Then use `${REGION,,}` (lowercase) for URL prefix: `iam.eu.confidence.dev`, `metrics.eu.confidence.dev`, etc. + +### Common notes + +- Port is fixed at **8084** (must match Auth0 Allowed Callback URLs) +- If port 8084 is busy: `lsof -ti:8084 | xargs kill -9 2>/dev/null` +- All network commands require `dangerouslyDisableSandbox: true` +- Never show the token value to the user +- Always use region-specific URLs (e.g., `iam.eu.confidence.dev` not `iam.confidence.dev`) + +### Important: gRPC-REST transcoding rules + +The Confidence APIs use gRPC with REST transcoding. The `body` field in the proto HTTP binding determines the JSON structure: + +- **`body: "data_warehouse"`** -> send the data warehouse object directly: `{"config": {...}}` +- **`body: "flag_applied_connection"`** -> send the connection object directly: `{"snowflake": {...}}` +- **`body: "event_connection"`** -> send the connection object directly: `{"snowflake": {...}}` +- **`body: "assignment_table"`** -> send the assignment table object directly: `{"displayName": "...", "sql": "...", ...}` +- **`body: "*"`** -> send the full request message + +The body is the object directly, NOT wrapped in an outer key. + +Fields NOT in the body (like `flag_id`, `parent`) become **query parameters**. + +**Field names are `snake_case`** in requests. Responses may use `camelCase`. + +--- + +## User-Facing Communication Rules + +**NEVER expose internal technical details to the user.** + +- Do NOT show raw JSON request/response bodies in conversation +- Do NOT show Auth0 configuration details, token values, or OAuth internals +- DO show human-readable status updates: "Opening browser for login...", "Creating your warehouse...", "Connectors configured!" +- DO describe results in plain English +- The agent handles all auth/API complexity silently + +**Step Tracker:** Display a visual step tracker at every phase transition. Update and re-display it each time you move to a new step. + +--- + +## Step Tracker + +Display at START and after EACH step completes (updating status): + +``` +───── Setup Warehouse (Snowflake) ───────────────────────── + [1] Choose warehouse ● done + [2] Account & user ○ pending + [3] Role & warehouse ○ pending + [4] Database & schema ○ pending + [5] Create crypto key ○ pending + [6] Register key in SF ○ pending + [7] Validate ○ pending + [8] Create warehouse ○ pending + [9] Create connectors ○ pending + [10] Assignment table ○ pending + [11] Verify pipeline ○ pending + [12] Done ○ pending +──────────────────────────────────────────────────────────── +``` + +Use `●` for completed, `▶` for in-progress, `○` for pending. Re-display the full tracker after every step transition. + +--- + +## Step 1: Choose warehouse (already done) + +The user has already chosen Snowflake. Mark step 1 as done. + +--- + +## Step 2: Account & user + +Ask the user for these fields (explain each briefly): + +- **Account** -- Snowflake account identifier (e.g., `zlvpqre-wr49874`). This is the part before `.snowflakecomputing.com` in the Snowflake URL. +- **User** -- Snowflake user for Confidence to connect as. + +--- + +## Step 3: Role & warehouse + +- **Role** -- Snowflake role (default: `ACCOUNTADMIN`). +- **Warehouse** -- SQL warehouse for query execution (default: `COMPUTE_WH`). + +--- + +## Step 4: Database & schema + +- **Exposure database** -- database for exposure tables (default: `CONFIDENCE`). +- **Exposure schema** -- schema for exposure tables (default: `EXPOSURE`). + +Also generate SQL for creating the database/schema if the user says they don't exist yet: +```sql +CREATE DATABASE IF NOT EXISTS ; +CREATE SCHEMA IF NOT EXISTS .; +GRANT USAGE ON DATABASE TO ROLE ; +GRANT ALL ON SCHEMA . TO ROLE ; +``` + +--- + +## Step 5: Create crypto key + +The user does NOT provide this. The skill creates it automatically via the IAM API: + +```bash +curl -s -w "\n%{http_code}" -X POST "https://iam.${REGION}.confidence.dev/v1/cryptoKeys?crypto_key_id=snowflake-key" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"kind": "SNOWFLAKE"}' +``` + +If the key already exists (HTTP 409), fetch it instead: +```bash +curl -s "https://iam.${REGION}.confidence.dev/v1/cryptoKeys/snowflake-key" \ + -H "Authorization: Bearer $TOKEN" +``` + +Extract the `publicKey` from the response, strip PEM headers and newlines to get raw base64. + +Save the crypto key name (e.g., `cryptoKeys/snowflake-key`) for use in the warehouse config. + +--- + +## Step 6: Register key in Snowflake + +Generate the Snowflake SQL to register the key, **copy it to clipboard**, and tell the user: + +> I've created an authentication key for Snowflake. You need to register it with your Snowflake user. +> The SQL has been copied to your clipboard -- paste it in the Snowflake worksheet and run it. + +The SQL should be: +```sql +ALTER USER SET RSA_PUBLIC_KEY=''; +``` + +**IMPORTANT:** Always ask the user if other Confidence accounts share this Snowflake user. If yes, use `RSA_PUBLIC_KEY_2` instead of `RSA_PUBLIC_KEY` to avoid breaking existing connections. Snowflake accepts auth from either key. + +```bash +echo "ALTER USER SET RSA_PUBLIC_KEY='';" | pbcopy +``` + +--- + +## Step 7: Validate + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/dataWarehouseConfig:validate" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "snowflakeConfig": { + "account": "", + "user": "", + "role": "", + "warehouse": "", + "database": "", + "schema": "", + "cryptoKey": "" + } + }' +``` + +**Response:** +```json +{ + "validation": [{ "key": "...", "description": "...", "success": true/false, "error": "..." }], + "successful": true/false, + "configurationResponse": { /* available schemas, databases, roles */ } +} +``` + +If `successful` is true, move to Step 8. + +**If validation fails:** + +**IMPORTANT: Never assume partial success from an ambiguous error.** If the API returns an error like "X does not exist or not authorized", report the exact error message. Do NOT split it into "connection works but X is missing". Show the user the exact error and let them determine the cause. + +For each validation failure, show: +> Validation failed: `` + +### Snowflake remediation + +Generate the full remediation SQL, **copy it to clipboard via `pbcopy`**, and tell the user to paste it in the Snowflake worksheet (https://app.snowflake.com): + +1. **Fetch the crypto key's public key** from the IAM API: + ```bash + curl -s "https://iam.${REGION}.confidence.dev/v1/cryptoKeys/" -H "Authorization: Bearer $TOKEN" + ``` + Strip the PEM headers (`-----BEGIN/END PUBLIC KEY-----`) and newlines to get the raw base64 string for Snowflake. + +2. **Generate SQL based on the error:** + + Auth failures -> register the public key: + ```sql + -- If this is the only Confidence account using this Snowflake user: + ALTER USER SET RSA_PUBLIC_KEY=''; + -- If another Confidence account already uses RSA_PUBLIC_KEY, use key 2: + ALTER USER SET RSA_PUBLIC_KEY_2=''; + ``` + **IMPORTANT:** Always ask the user if other Confidence accounts share this Snowflake user. If yes, use `RSA_PUBLIC_KEY_2` to avoid breaking existing connections. Snowflake accepts auth from either key. + + Database/schema missing: + ```sql + CREATE DATABASE IF NOT EXISTS ; + CREATE SCHEMA IF NOT EXISTS .; + GRANT USAGE ON DATABASE TO ROLE ; + GRANT USAGE ON SCHEMA . TO ROLE ; + GRANT ALL ON SCHEMA . TO ROLE ; + ``` + +3. **Copy to clipboard and tell the user:** + ```bash + echo "" | pbcopy + ``` + > The SQL commands have been copied to your clipboard. Paste them in the Snowflake worksheet at https://app.snowflake.com and run them. Let me know when done and I'll retry validation. + +If `configurationResponse` contains available options (schemas, databases, roles), present these as choices to help the user. + +--- + +## Step 8: Create warehouse + +**IMPORTANT:** The body is the data warehouse object directly (gRPC transcoding `body: "data_warehouse"`), NOT wrapped in a `dataWarehouse` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/dataWarehouses" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "config": { + "snowflakeConfig": { + "account": "", + "user": "", + "role": "", + "warehouse": "", + "database": "", + "schema": "", + "cryptoKey": "" + } + } + }' +``` + +Save the returned `name` (e.g., `dataWarehouses/...`) for reference. + +--- + +## Step 9: Create connectors + +Create both connectors. **Snowflake connectors require `database` and `schema` fields in snowflakeConfig.** + +### Flag Applied Connection (assignment data -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "flag_applied_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/flagAppliedConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "snowflake": { + "snowflakeConfig": { + "account": "", + "user": "", + "role": "", + "warehouse": "", + "database": "", + "schema": "", + "cryptoKey": "" + }, + "table": "ASSIGNMENTS" + } + }' +``` + +### Event Connection (events -> warehouse) + +**IMPORTANT:** The body is the connection object directly (gRPC transcoding `body: "event_connection"`), NOT wrapped. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://connectors.${REGION}.confidence.dev/v1/eventConnections" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "snowflake": { + "snowflakeConfig": { + "account": "", + "user": "", + "role": "", + "warehouse": "", + "database": "", + "schema": "", + "cryptoKey": "" + }, + "tablePrefix": "EVENTS_" + } + }' +``` + +--- + +## Step 10: Assignment table + +Create an assignment table so Confidence can analyze experiment assignments. + +**IMPORTANT:** The body is the assignment table object directly (gRPC transcoding `body: "assignment_table"`), NOT wrapped in an `assignmentTable` key. + +```bash +curl -s -w "\n%{http_code}" -X POST "https://metrics.${REGION}.confidence.dev/v1/assignmentTables" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "displayName": "Assignments", + "sql": "SELECT targeting_key, rule, assignment_id, assignment_time FROM ..ASSIGNMENTS", + "entityColumn": { "name": "targeting_key" }, + "timestampColumn": { "name": "assignment_time" }, + "exposureKeyColumn": { "name": "rule" }, + "variantKeyColumn": { "name": "assignment_id" }, + "dataDeliveredUntilUpdateStrategyConfig": { + "strategy": "AUTOMATIC", + "automaticUpdateConfig": { + "commitDelay": "300s" + } + } + }' +``` + +--- + +## Step 11: Verify data pipeline + +Verify both connectors by generating test data and checking it lands in the warehouse. + +### 11a. Get a client secret for testing + +The resolver and events APIs require a **client secret** (not a Bearer token). + +1. **List the user's clients** and show them: + ```bash + curl -s "https://iam.${REGION}.confidence.dev/v1/clients" -H "Authorization: Bearer $TOKEN" + ``` + Display each client with its name and last-seen time. If only one client exists, confirm it with the user. If multiple, let them pick. + +2. **Ask the user** if they have a client secret or want a new one: + > I'll use **** for the pipeline test. Do you have the client secret, or should I create a new credential? + +3. If the user wants a new credential, create one on the chosen client: + ```bash + curl -s -X POST "https://iam.${REGION}.confidence.dev/v1//credentials" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"display_name": "Pipeline Test"}' + ``` + Save the secret to a temp file for pipeline use. **Never print the secret to the user's terminal.** + +### 11b. Verify flag assignments + +Resolve a flag to generate assignment data (use an existing flag + client secret): +```bash +curl -s -X POST "https://resolver.${REGION}.confidence.dev/v1/flags:resolve" \ + -H "Content-Type: application/json" \ + -d '{ + "flags": ["flags/"], + "evaluation_context": {"targeting_key": "warehouse-verify-user"}, + "client_secret": "", + "apply": true + }' +``` + +If no flags exist yet, tell the user: +> No flags to test with. Run `/onboard-confidence setup-wizard` first to create a flag, then come back. + +### 11c. Verify events + +First check for an event definition to use: +```bash +curl -s "https://events.${REGION}.confidence.dev/v1/eventDefinitions" \ + -H "Authorization: Bearer $TOKEN" +``` + +If no event definitions exist, create one with a schema: +```bash +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/eventDefinitions?event_definition_id=test-event" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +If an event definition exists but has an empty schema, update it so payload data flows through: +```bash +curl -s -X PATCH "https://events.${REGION}.confidence.dev/v1/eventDefinitions/" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}' +``` + +Then publish test events (uses client secret, NOT Bearer token): +```bash +NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ") +curl -s -X POST "https://events.${REGION}.confidence.dev/v1/events:publish" \ + -H "Content-Type: application/json" \ + -d '{ + "client_secret": "", + "events": [ + { + "event_definition": "eventDefinitions/", + "payload": {"action": "clicked_button", "page": "homepage"}, + "event_time": "'$NOW'" + } + ], + "send_time": "'$NOW'" + }' +``` + +Check response: `{"errors": []}` means success. If `EVENT_DEFINITION_NOT_FOUND`, the definition doesn't exist. If `EVENT_SCHEMA_VALIDATION_FAILED`, the payload doesn't match the schema. + +### 11d. Check data in Snowflake + +Ask the user: "Want me to check the data, or show you the queries?" + +If user has `snowsql` CLI: +```bash +snowsql -a ${SNOWFLAKE_ACCOUNT} -u ${SNOWFLAKE_USER} -r ${SNOWFLAKE_ROLE} -w ${SNOWFLAKE_WAREHOUSE} -d ${SNOWFLAKE_DATABASE} -s ${SNOWFLAKE_SCHEMA} -q " +SELECT targeting_key, rule, assignment_id, assignment_time +FROM ${SNOWFLAKE_DATABASE}.${SNOWFLAKE_SCHEMA}.ASSIGNMENTS +ORDER BY assignment_time DESC LIMIT 5; +" +``` + +If no `snowsql`, use the Snowflake SQL REST API: +```bash +# Get a JWT token for Snowflake (using keypair auth) or prompt user for password +# Then query via the SQL API: +curl -s -X POST "https://${SNOWFLAKE_ACCOUNT}.snowflakecomputing.com/api/v2/statements" \ + -H "Authorization: Bearer ${SNOWFLAKE_JWT}" \ + -H "Content-Type: application/json" \ + -H "X-Snowflake-Authorization-Token-Type: KEYPAIR_JWT" \ + -d '{ + "statement": "SELECT targeting_key, rule, assignment_id, assignment_time FROM '${SNOWFLAKE_DATABASE}'.'${SNOWFLAKE_SCHEMA}'.ASSIGNMENTS ORDER BY assignment_time DESC LIMIT 5", + "warehouse": "'${SNOWFLAKE_WAREHOUSE}'", + "database": "'${SNOWFLAKE_DATABASE}'", + "schema": "'${SNOWFLAKE_SCHEMA}'", + "role": "'${SNOWFLAKE_ROLE}'" + }' +``` + +If neither available, show the queries for the Snowflake worksheet (https://app.snowflake.com): +> ```sql +> -- Assignments +> SELECT targeting_key, rule, assignment_id, assignment_time +> FROM ..ASSIGNMENTS +> ORDER BY assignment_time DESC LIMIT 5; +> +> -- Events (list event tables first, then query) +> SHOW TABLES LIKE 'EVENTS_%' IN .; +> SELECT * FROM .. +> ORDER BY _event_time DESC LIMIT 5; +> ``` + +**Show results:** +``` + ● Assignments: rows -- data flowing + -> () + ● Events: rows -- data flowing + on () +``` + +**If no rows after a few seconds**, tell the user: +> Data delivery can take up to a few minutes depending on your warehouse. Check again shortly, or verify in your Snowflake worksheet. + +--- + +## Step 12: Done + +``` +═══════════════════════════════════════════════════════════════ + Data Warehouse Connected & Verified +═══════════════════════════════════════════════════════════════ + + Warehouse: Snowflake () + Database: + Schema: + Connectors: + ● Flag assignments -> ASSIGNMENTS table (verified) + ● Events -> EVENTS_* tables (running) + Assignment: + ● Assignment table configured (auto-updating) + + Flag assignment and event data is flowing to your + warehouse. Experiment analysis is ready. + +═══════════════════════════════════════════════════════════════ +``` + +--- + +## API Reference (agent-internal -- do NOT show to user) + +### Base URLs + +All APIs require **region-specific URLs**. Extract region from the JWT token claim `https://confidence.dev/region` (value: `EU` or `US`), lowercase it, and use as prefix. + +``` +IAM_API: https://iam.${region}.confidence.dev/v1 +RESOLVER_API: https://resolver.${region}.confidence.dev/v1 +EVENTS_API: https://events.${region}.confidence.dev/v1 +CONNECTORS_API: https://connectors.${region}.confidence.dev/v1 +METRICS_API: https://metrics.${region}.confidence.dev/v1 +``` + +### Endpoints + +**Create crypto key (Bearer token):** +``` +POST ${IAM_API}/cryptoKeys?crypto_key_id= +Body: { "kind": "SNOWFLAKE" } +-> CryptoKey object with publicKey field +``` + +**Get crypto key (Bearer token):** +``` +GET ${IAM_API}/cryptoKeys/ +-> CryptoKey object with publicKey field +``` + +**Validate warehouse config (Bearer token):** +``` +POST ${METRICS_API}/dataWarehouseConfig:validate +Body: { "snowflakeConfig": {...} } +-> { "validation": [...], "successful": bool, "configurationResponse": {...} } +``` + +**Check warehouse exists (Bearer token):** +``` +GET ${METRICS_API}/dataWarehouses:exists +-> { "exists": bool } +``` + +**Create data warehouse (Bearer token, body: "data_warehouse"):** +``` +POST ${METRICS_API}/dataWarehouses +Body (direct object): { "config": { "snowflakeConfig": {...} } } +-> DataWarehouse object +``` + +**Create flag applied connection (Bearer token, body: "flag_applied_connection"):** +``` +POST ${CONNECTORS_API}/flagAppliedConnections +Body (direct object): { "snowflake": { "snowflakeConfig": {..., "database": "...", "schema": "..."}, "table": "ASSIGNMENTS" } } +-> FlagAppliedConnection object +NOTE: Snowflake connectors require database and schema fields in snowflakeConfig +``` + +**Create event connection (Bearer token, body: "event_connection"):** +``` +POST ${CONNECTORS_API}/eventConnections +Body (direct object): { "snowflake": { "snowflakeConfig": {..., "database": "...", "schema": "..."}, "tablePrefix": "EVENTS_" } } +-> EventConnection object +``` + +**Create assignment table (Bearer token, body: "assignment_table"):** +``` +POST ${METRICS_API}/assignmentTables +Body (direct object): { "displayName": str, "sql": str, "entityColumn": {...}, "timestampColumn": {...}, "exposureKeyColumn": {...}, "variantKeyColumn": {...}, "dataDeliveredUntilUpdateStrategyConfig": {...} } +-> AssignmentTable object +``` + +**List clients (Bearer token):** +``` +GET ${IAM_API}/clients +-> { "clients": [...], "nextPageToken": string } +``` + +**Create client credential (Bearer token, body: "client_credential"):** +``` +POST ${IAM_API}/${clientName}/credentials +Body (direct object): { "display_name": string } +-> { "name": "...", "clientSecret": { "secret": string }, ... } + NOTE: secret only returned once on creation +``` + +**Resolve flags (client secret -- NOT Bearer token):** +``` +POST ${RESOLVER_API}/flags:resolve +Body: { "flags": ["flags/"], "evaluationContext": {...}, "clientSecret": string, "apply": bool } +-> { "resolvedFlags": [...] } +``` + +**Publish events (client secret -- NOT Bearer token):** +``` +POST ${EVENTS_API}/events:publish +Body: { "client_secret": string, "events": [...], "send_time": "ISO8601" } +-> { "errors": [...] } +``` + +--- + +## Error Handling Reference (agent-internal) + +### Common HTTP errors + +| Status | Meaning | Recovery | +|--------|---------|----------| +| 400 | Validation error | Parse `.message`, show plain English, re-collect invalid field | +| 401 | Invalid/expired token | Re-trigger Auth0 login | +| 403 | Insufficient permissions | Explain needed role/permission | +| 404 | Resource not found | Check account/resource exists | +| 409 | Conflict (already exists) | Resource already created (e.g., crypto key) | +| 429 | Rate limited | Wait briefly and retry | +| 500+ | Server error | Inform user, suggest retry | + +### Sandbox note + +All `curl`, `open`, and `python3` commands that access external hosts (`auth.confidence.dev`, `iam.confidence.dev`, `metrics.confidence.dev`, `connectors.confidence.dev`, etc.) require `dangerouslyDisableSandbox: true`. On first occurrence, briefly explain to the user that network access outside the sandbox is needed for API calls. diff --git a/skills/setup-warehouse/SKILL.md b/skills/setup-warehouse/SKILL.md new file mode 100644 index 0000000..a9360e6 --- /dev/null +++ b/skills/setup-warehouse/SKILL.md @@ -0,0 +1,187 @@ +--- +description: Set up a data warehouse for Confidence experimentation analytics. Use when the user asks to connect a warehouse, set up BigQuery/Snowflake/Databricks/Redshift, or configure data connectors. +--- + +# Setup Warehouse + +Configure a data warehouse so Confidence can store and analyze your experiment data — flag assignments, events, and metrics. + +A data warehouse is where Confidence writes your experimentation data. It connects to your existing cloud data infrastructure so you can query experiment results, build dashboards, and run statistical analysis. Without a warehouse, Confidence can resolve flags but cannot analyze experiment outcomes. + +## Supported Warehouse Types + +| # | Warehouse | Best for | +|---|-----------|----------| +| 1 | **BigQuery** | Google Cloud users, fastest setup | +| 2 | **Snowflake** | Snowflake users, key-pair authentication | +| 3 | **Databricks** | Databricks users, requires AWS S3 staging bucket | +| 4 | **Redshift** | AWS users, requires S3 staging bucket | + +## Flow + +Present the user with the four options: + +> Which data warehouse do you use? +> 1. BigQuery +> 2. Snowflake +> 3. Databricks +> 4. Redshift + +After the user picks, hand off to the specific warehouse skill: + +- **BigQuery** -> Tell the user: "Starting BigQuery setup..." and invoke `/onboard-confidence:setup-warehouse-bigquery` +- **Snowflake** -> Tell the user: "Starting Snowflake setup..." and invoke `/onboard-confidence:setup-warehouse-snowflake` +- **Databricks** -> Tell the user: "Starting Databricks setup..." and invoke `/onboard-confidence:setup-warehouse-databricks` +- **Redshift** -> Tell the user: "Starting Redshift setup..." and invoke `/onboard-confidence:setup-warehouse-redshift` + +--- + +## Authentication + +**Browser-based Auth0 login.** The skill opens a browser for Auth0 login (Google, email/password, SSO) and captures the token automatically. The user never touches a token. + +### Session-only token management + +The token is kept in the current session only and is never saved to disk. If the session ends or the token expires, the skill will open your browser to log in again. + +On every sub-command start, check if the `TOKEN` variable is set and not expired: + +```bash +if [ -n "$TOKEN" ]; then + PAYLOAD=$(echo "$TOKEN" | cut -d. -f2) + EXP=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('exp', 0)) +") + NOW=$(date +%s) + if [ "$EXP" -gt "$NOW" ]; then + echo "VALID" + else + echo "EXPIRED" + unset TOKEN + fi +fi +``` + +If `TOKEN` is unset or expired, run the Auth0 login flow with the **regular client ID** (`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w`) and the user's `organization` parameter. Store the result in the `TOKEN` shell variable only. **NEVER write the token to disk. NEVER reference `~/.confidence/`.** + +### Extract region from token + +```bash +PAYLOAD=$(echo "$TOKEN" | cut -d. -f2) +REGION=$(echo "$PAYLOAD" | python3 -c " +import sys, json, base64 +p = sys.stdin.read().strip() +p += '=' * (4 - len(p) % 4) if len(p) % 4 else '' +d = json.loads(base64.b64decode(p)) +print(d.get('https://confidence.dev/region', 'EU')) +") +``` + +Then use `${REGION,,}` (lowercase) for URL prefix: `iam.eu.confidence.dev`, `metrics.eu.confidence.dev`, etc. + +### Auth script + +Write the following to `$TMPDIR/confidence_auth.py` with CLIENT_ID=`2fG3H4RhlAbIZm9Rfn32zTaILH7w1X4w` and ORGANIZATION from the token. Run with `python3 $TMPDIR/confidence_auth.py`. Outputs `TOKEN:` on success. + +```python +import http.server, urllib.parse, json, sys, subprocess, hashlib, base64, secrets, string + +code_verifier = ''.join(secrets.choice(string.ascii_letters + string.digits + '-._~') for _ in range(43)) +code_challenge = base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode()).digest()).rstrip(b'=').decode() + +port = 8084 +CLIENT_ID = '' +ORGANIZATION = '' +REDIRECT_URI = f'http://localhost:{port}/callback' +auth_code = None +error = None + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + global auth_code, error + q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query) + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.end_headers() + if 'code' in q: + auth_code = q['code'][0] + self.wfile.write(b'

Login successful!

You can close this tab.

') + else: + error = q.get('error', ['unknown'])[0] + self.wfile.write(b'

Login failed

Please try again.

') + def log_message(self, format, *args): + pass + +params = { + 'client_id': CLIENT_ID, + 'redirect_uri': REDIRECT_URI, + 'response_type': 'code', + 'scope': 'openid profile email offline_access', + 'audience': 'https://confidence.dev/', + 'code_challenge': code_challenge, + 'code_challenge_method': 'S256', +} +if ORGANIZATION: + params['organization'] = ORGANIZATION + +authorize_url = 'https://auth.confidence.dev/authorize?' + urllib.parse.urlencode(params) +subprocess.Popen(['open', authorize_url]) +print('WAITING_FOR_LOGIN', flush=True) + +server = http.server.HTTPServer(('127.0.0.1', port), Handler) +server.timeout = 120 +while auth_code is None and error is None: + server.handle_request() +server.server_close() + +if error: + print(f'AUTH_ERROR:{error}', flush=True) + sys.exit(1) + +import urllib.request +token_data = json.dumps({ + 'grant_type': 'authorization_code', + 'client_id': CLIENT_ID, + 'code': auth_code, + 'redirect_uri': REDIRECT_URI, + 'code_verifier': code_verifier +}).encode() +req = urllib.request.Request( + 'https://auth.confidence.dev/oauth/token', + data=token_data, + headers={'Content-Type': 'application/json'} +) +try: + with urllib.request.urlopen(req) as resp: + token_response = json.loads(resp.read()) + print(f'TOKEN:{token_response["access_token"]}', flush=True) +except Exception as e: + print(f'TOKEN_ERROR:{e}', flush=True) + sys.exit(1) +``` + +### Common notes + +- Port is fixed at **8084** (must match Auth0 Allowed Callback URLs) +- If port 8084 is busy: `lsof -ti:8084 | xargs kill -9 2>/dev/null` +- All network commands require `dangerouslyDisableSandbox: true` +- Never show the token value to the user +- Always use region-specific URLs (e.g., `iam.eu.confidence.dev` not `iam.confidence.dev`) + +--- + +## User-Facing Communication Rules + +**NEVER expose internal technical details to the user.** + +- Do NOT show raw JSON request/response bodies in conversation +- Do NOT show Auth0 configuration details, token values, or OAuth internals +- DO show human-readable status updates: "Opening browser for login...", "Creating your warehouse...", "Connectors configured!" +- DO describe results in plain English +- The agent handles all auth/API complexity silently + +**Step Tracker:** Display a visual step tracker at every phase transition. Update and re-display it each time you move to a new step. Use `●` for completed, `▶` for in-progress, `○` for pending.