Merge origin/staging into improvement/v2-endpoints

Author: TheodoreSpeaks

.agents/skills/add-block/SKILL.md

@@ -163,6 +163,32 @@ export const {ServiceName}Block: BlockConfig = {
 
 Optional companions: `credentialLabels` (override the picker's section/connect-row copy) and `allowServiceAccounts: true` (trigger-mode only — list service accounts, which triggers otherwise exclude; set only when the trigger's polling path can resolve a service-account token). The connect modal, provider families (Google JSON key, Atlassian token, token-paste, client-credential, Slack bot), and the preview gate are all resolved from `serviceAccountProviderId` — you don't wire them per block.
 
+### OAuth deployment availability (required for integration blocks)
+
+A visible tools-category block with OAuth is deployment-gated. Its `oauth-input.serviceId` is
+projected into `apps/sim/lib/integrations/integrations.json`, then resolved through
+`resolveOAuthClientCapabilityId()` in `apps/sim/lib/core/config/env-capabilities.ts`.
+
+When adding or changing an OAuth integration block:
+
+1. Keep exactly one distinct OAuth `serviceId` across the block's `oauth-input` subBlocks.
+2. Confirm that service ID resolves to an entry in `OAUTH_CLIENT_CAPABILITIES`. Google and
+   Microsoft service IDs intentionally share their provider-level capability; do not add duplicate
+   entries for those aliases.
+3. For a new capability, add its required client fields to `OAUTH_CLIENT_CAPABILITIES` and ensure
+   every referenced field exists in the env schema in `apps/sim/lib/core/config/env.ts`. Then add
+   the matching `text` or `secret` input modes to `OAUTH_CLIENT_SETUP_FIELDS` in
+   `scripts/setup/capability-config.ts`. The CLI catalog is exhaustively typed and checked against
+   the runtime field list; do not infer secrecy from the field name.
+4. If the canonical OAuth service declares `serviceAccountProviderId`, keep
+   `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` in
+   `apps/sim/lib/integrations/service-account-metadata.ts` aligned. Set
+   `deploymentRequirement` only when the service-account path is preview-gated or depends on the
+   OAuth client fields; otherwise omit it.
+
+Missing capability metadata is a runtime configuration error, not a reason to make the integration
+silently available.
+
 ### Selectors (with dynamic options)
 ```typescript
 // Channel selector (Slack, Discord, etc.)
@@ -919,12 +945,25 @@ Derive templates from the service's real use cases. Each prompt should name a co
 - **Ground every skill in operations the block actually exposes** — cross-check each skill's steps against `tools.access`. Never describe an action the integration cannot perform.
 - **Derive skills from real, popular use cases found online — never invent them.** Web-search the service's documented use cases (vendor use-case/solutions pages, official docs describing the workflow, reputable "top automations for X" articles) and only add a skill you can source as something people genuinely do with the service. Do not hallucinate skills.
 
-## Generated tool metadata
+## Generated artifacts
 
-Adding a block on its own needs **no** regeneration — a block references existing tool IDs through `tools.access` and does not change any tool's shape.
+Adding a block on its own needs no **tool metadata** regeneration — a block references existing
+tool IDs through `tools.access` and does not change any tool's shape.
 
 But if the same change also adds, edits **or removes** a tool, run `bun run tool-metadata:generate` and commit the result, or CI fails on stale artifacts. That matters here because a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata rather than the executable registry — an unregenerated tool change makes the block's outputs disagree with what the panel renders. See `.agents/skills/tool-registry-boundary/SKILL.md`.
 
+A visible integration block does require the generated integration catalog and docs to be refreshed.
+After adding or changing one, run:
+
+```bash
+bun run scripts/generate-docs.ts
+bun run integration-catalog:check
+```
+
+The catalog check independently derives deployment metadata from the executable block registry and
+compares it with the committed `apps/sim/lib/integrations/integrations.json`. Review the generated
+diff and keep only intentional changes.
+
 ## Checklist Before Finishing
 
 - [ ] `integrationType` is set to the correct `IntegrationType` enum value
@@ -934,12 +973,17 @@ But if the same change also adds, edits **or removes** a tool, run `bun run tool
 - [ ] DependsOn set for fields that need other values
 - [ ] Required fields marked correctly (boolean or condition)
 - [ ] OAuth inputs have correct `serviceId` and `requiredScopes: getScopesForService(serviceId)`
+- [ ] Every OAuth `serviceId` resolves through `resolveOAuthClientCapabilityId()` to the correct `OAUTH_CLIENT_CAPABILITIES` entry
+- [ ] Any new OAuth capability fields exist in `apps/sim/lib/core/config/env.ts`
+- [ ] If the OAuth service supports service accounts, `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` matches its canonical `serviceAccountProviderId` and deployment requirement
 - [ ] Scope descriptions added to `SCOPE_DESCRIPTIONS` in `lib/oauth/utils.ts` for any new scopes
 - [ ] Tools.access lists all tool IDs (snake_case)
 - [ ] Tools.config.tool returns correct tool ID (snake_case)
 - [ ] Outputs match tool outputs
 - [ ] Block + meta registered in registry-maps.ts (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
 - [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts
+- [ ] Ran `bun run scripts/generate-docs.ts`, reviewed the generated diff, and committed the integration catalog changes
+- [ ] `bun run integration-catalog:check` passes
 - [ ] If icon missing: asked user to provide SVG
 - [ ] If triggers exist: `triggers` config set, trigger subBlocks spread
 - [ ] Optional/rarely-used fields set to `mode: 'advanced'`

.agents/skills/add-enrichment/SKILL.md

@@ -63,7 +63,7 @@ Why it matters: the cascade runner only bills (and only reads `output.cost.total
 Create `apps/sim/enrichments/{name}/{name}.ts` and a barrel `index.ts`. Mirror the existing entries (`work-email`, `phone-number`, `company-domain`, `company-info`).
 
 ```typescript
-import { SomeIcon } from 'lucide-react'
+import { SomeIcon } from '@sim/emcn/icons'
 import { filterUndefined } from '@sim/utils/object'
 import { normalizeDomain, splitName, str, toolProvider } from '@/enrichments/providers'
 import type { EnrichmentConfig } from '@/enrichments/types'
@@ -109,7 +109,7 @@ export { myEnrichment } from './my-enrichment'
 ```
 
 Rules:
-- Keep the file **client-safe**: import only `lucide-react`, `@sim/utils/*`, `@/enrichments/providers`, and the types. **Never import `@/tools`** here — the runner does the tool call.
+- Keep the file **client-safe**: import only `@sim/emcn/icons`, `@sim/utils/*`, `@/enrichments/providers`, and the types. **Never import `@/tools`** here — the runner does the tool call.
 - `buildParams` returns `null` when inputs are insufficient (provider skipped). `mapOutput` returns `null`/empty for a miss (falls through). Use `filterUndefined` when assembling optional tool params; coerce numbers explicitly (don't pass `''` to number outputs).
 - Output `id`s are the keys `mapOutput` returns; output `name`s are the default column names (the user can rename them in the config).
 

.agents/skills/add-feature-flag/SKILL.md

@@ -1,38 +1,46 @@
 ---
 name: add-feature-flag
-description: Add a runtime gated feature flag (AppConfig-backed on prod, secret fallback off-prod), gated by org id, user id, or admin
+description: Add a runtime feature flag (AppConfig-backed on prod, secret fallback off-prod), global by default or optionally gated by org id, user id, or platform admin
 argument-hint: <flag-name>
 ---
 
 # Add Feature Flag Skill
 
-You add a **runtime, gated feature flag** to Sim — one that can be turned on for specific orgs, users, or admins and changed on prod with no redeploy (AWS AppConfig). When AppConfig isn't the source of truth, the flag falls back to a single **secret** (on/off only).
+You add a **runtime feature flag** to Sim that can change on prod with no redeploy (AWS AppConfig). Prefer a global on/off flag unless the rollout actually needs per-organization, per-user, or platform-admin targeting. When AppConfig isn't the source of truth, the flag falls back to a single **secret** (on/off only).
 
 ## When to use this vs `env-flags.ts`
 
-- **Feature flag** (`@/lib/core/config/feature-flags.ts`): per-request, gated by `userId`/`orgId`/admin, changeable at runtime. This skill.
+- **Feature flag** (`@/lib/core/config/feature-flags.ts`): runtime global on/off by default, optionally scoped by `userId`/`orgId`/admin. This skill.
 - **Env flag** (`@/lib/core/config/env-flags.ts`): deploy-time capability/environment detection (`isProd`, `isHosted`, `isBillingEnabled`). A module-load boolean. **Do not add gated flags here.**
 
 If the user wants a fixed per-deployment toggle, send them to `env-flags.ts` instead.
 
 ## The flag model
 
-A flag's **gating rule lives only in the hosted AppConfig document**. It is ON for a context when any clause matches:
+A flag's **gating rule lives only in the hosted AppConfig document**. It is ON for a context when any configured clause matches:
 
 ```ts
 interface FeatureFlagRule {
   enabled?: boolean   // global default for everyone
   orgIds?: string[]   // allowlisted organization ids
   userIds?: string[]  // allowlisted user ids
-  admins?: boolean    // platform admins (user.role === 'admin')
+  adminEnabled?: boolean // platform admins (user.role === 'admin')
 }
 ```
 
-Critically, **none of this is expressible in code** — gating (especially `admins`) can only be set through AppConfig, so no environment can grant access from a code literal. Off-AppConfig (self-hosted/OSS/local), a flag is simply on or off, derived from its fallback secret.
+Critically, **none of this is expressible in code** — gating (especially `adminEnabled`) can only be set through AppConfig, so no environment can grant access from a code literal. Off-AppConfig (self-hosted/OSS/local), a flag is simply on or off, derived from its fallback secret.
 
 ## Steps
 
-1. **Define the flag.** Add one entry to the `FEATURE_FLAGS` registry in `apps/sim/lib/core/config/feature-flags.ts`. Each entry is the flag's whole definition — name (kebab-case key), `description`, and the `fallback` secret consulted when AppConfig isn't the source of truth (truthy ⇒ on globally):
+1. **Confirm the granularity before editing code.** If the user has not already specified it, stop and ask:
+
+   > Should `<flag-name>` be a global on/off flag (recommended), or does it need rollout targeting by organization, user, and/or platform admin?
+
+   - Recommend **global**. Do not infer scoped gating merely because the call site already has a user or organization id.
+   - If the user chooses scoped gating but does not name the dimensions, ask which of organization, user, and platform admin it needs. Wire only the selected dimensions.
+   - If the user wants a fixed per-deployment toggle rather than a runtime AppConfig flag, use `env-flags.ts` instead.
+
+2. **Define the flag.** Add one entry to the `FEATURE_FLAGS` registry in `apps/sim/lib/core/config/feature-flags.ts`. Each entry is the flag's whole definition — name (kebab-case key), `description`, and the `fallback` secret consulted when AppConfig isn't the source of truth (truthy ⇒ on globally):
 
    ```ts
    const FEATURE_FLAGS = {
@@ -45,7 +53,19 @@ Critically, **none of this is expressible in code** — gating (especially `admi
 
    `fallback` is the env/secret key (typed as `keyof typeof env`), so add `<FLAG_SECRET>` to `apps/sim/lib/core/config/env.ts` first (and the deployment's secret store) — it won't typecheck otherwise. Do **not** add org/user/admin defaults here — that gating exists only in AppConfig. Adding the entry makes `<flag-name>` a valid `FeatureFlagName`.
 
-2. **Gate the call site.** Call `isFeatureEnabled` with whatever ids you have — admin status is resolved internally, so callers never pass it:
+3. **Gate the call site at the chosen granularity.** For the recommended global mode, pass no context:
+
+   ```ts
+   import { isFeatureEnabled } from '@/lib/core/config/feature-flags'
+
+   if (await isFeatureEnabled('<flag-name>')) {
+     // gated behavior
+   }
+   ```
+
+   Do not fetch, resolve, or thread through user or organization context solely for a global flag.
+
+   For scoped rollout, pass only the dimensions the user selected. Admin status is resolved internally, so ordinary callers pass `userId`, not a role:
 
    ```ts
    import { isFeatureEnabled } from '@/lib/core/config/feature-flags'
@@ -55,19 +75,21 @@ Critically, **none of this is expressible in code** — gating (especially `admi
    }
    ```
 
+   - Organization targeting uses `orgId`; user and platform-admin targeting require `userId`.
    - Missing ids are fine — a clause with no matching id is skipped; with no `userId`, the admin clause resolves to `false` without a DB read.
    - Admin routes that already know the caller is an admin may pass `{ userId, isAdmin: true }` to skip the role lookup.
    - **Client/UI flags:** resolve server-side (in a server component, route, or loader) and pass the boolean down as a prop. There is no client AppConfig.
 
-3. **(Prod) configure in AppConfig.** The infra `feature-flags` profile schema is permissive, so a new flag needs **no infra change**. Operators add the flag under `flags` in the hosted `feature-flags` document — including any `orgIds`/`userIds`/`admins` gating — and start a `sim-<env>-fast` deployment (see the AppConfig runbook in the infra README — same flow as `access-control`). The fallback secret only applies when AppConfig is disabled.
+4. **(Prod) configure in AppConfig.** The infra `feature-flags` profile schema is permissive, so a new flag needs **no infra change**. Operators add the flag to the hosted `feature-flags` document using `enabled` for global rollout or only the selected `orgIds`/`userIds`/`adminEnabled` clauses for scoped rollout, then start a `sim-<env>-fast` deployment (see the AppConfig runbook in the infra README — same flow as `access-control`). The fallback secret only applies when AppConfig is disabled.
 
-4. **Test.** Add a case to `apps/sim/lib/core/config/feature-flags.test.ts`: use `withAppConfig({ flags: { ... } })` to cover the gating rule (mock `isPlatformAdmin` for the `admins` clause), and toggle the fallback secret to cover the off-AppConfig path.
+5. **Test.** Add a case to `apps/sim/lib/core/config/feature-flags.test.ts` that matches the chosen granularity. For a global flag, exercise `isFeatureEnabled('<flag-name>')` with an AppConfig `enabled` rule and toggle the fallback secret for the off-AppConfig path. For scoped rollout, cover only the selected clauses and mock `isPlatformAdmin` when testing `adminEnabled`.
 
-5. **Clean up after rollout.** When the feature ships to everyone, delete the flag's entry from `FEATURE_FLAGS`, the `<FLAG_SECRET>` env entry, the AppConfig document, the call sites, and the test. Leaving dead flags around is the main failure mode of flag systems.
+6. **Clean up after rollout.** When the feature ships to everyone, delete the flag's entry from `FEATURE_FLAGS`, the `<FLAG_SECRET>` env entry, the AppConfig document, the call sites, and the test. Leaving dead flags around is the main failure mode of flag systems.
 
 ## Notes
 
 - Flag keys are `kebab-case`.
 - Never read flags via raw `fetch` or a new AppConfig client — always go through `isFeatureEnabled` / `getFeatureFlags`.
 - Never bake gating into code. The fallback is a single boolean secret; org/user/admin scoping is AppConfig-only.
-- The admin check reads the DB **replica** (`dbReplica`) and is resolved lazily, so an admin-gated flag adds at most one cheap replica read, and only when `admins` is the deciding clause.
+- Never add or propagate request context unless the user chose scoped rollout.
+- The admin check reads the DB **replica** (`dbReplica`) and is resolved lazily, so an admin-gated flag adds at most one cheap replica read, and only when `adminEnabled` is the deciding clause.