integration projections

Author: icecrasher321

.agents/skills/add-integration/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: add-integration
-description: Add a complete Sim integration from API docs, covering tools, block, icon, optional triggers, registrations, and integration conventions. Use when introducing a new service under `apps/sim/tools`, `apps/sim/blocks`, and `apps/sim/triggers`.
+description: Add a complete Sim integration from API docs, covering tools, block, icon, optional triggers, registrations, resolved-secret/model-input safety, and integration conventions. Use when introducing a new service under `apps/sim/tools`, `apps/sim/blocks`, and `apps/sim/triggers`.
 argument-hint: <service-name> [api-docs-url]
 ---
 
@@ -122,6 +122,63 @@ export const {service}{Action}Tool: ToolConfig<Params, Response> = {
 - When using `type: 'json'` and you know the object shape, define `properties` with the inner fields so downstream consumers know the structure. Only use bare `type: 'json'` when the shape is truly dynamic
 - If you do not know the response JSON shape from docs or verified examples, you MUST tell the user and stop. Never guess outputs or response mappings.
 
+### Resolved Secrets at Model and Persistence Boundaries
+
+Classify every request field before implementing the tool:
+
+This is opt-in, not a blanket integration migration. Add a model-input declaration only when the
+service's official documentation or an unambiguous local execution path proves that the exact
+field is consumed by an AI model. If that cannot be established, preserve existing tool behavior
+and leave the field unannotated.
+
+- **Ordinary provider/API input:** leave it unchanged. Do not add blanket result sanitization.
+- **Text or structured content consumed by an AI model:** declare `request.modelInput` with
+  `mode: 'project'` and select only the exact model-visible fields. The shared executor replaces
+  activated Sim secrets with canonical `{{NAME}}` labels before request formatting. For nested or
+  JSON-string fields, use a small shared selector plus `applyProjected`; verify that selecting the
+  rebuilt params reproduces the projected selection.
+- **Opaque model input sent directly to an external provider** such as a model-read URL or image
+  payload: declare `request.opaqueModelInput` with `mode: 'reject-resolved-secrets'` and select only
+  the exact effective value. The shared `executeTool` preflight rejects incomplete or secret-bearing
+  committed provenance before URL/body formatting or network I/O, preserves safe request bytes,
+  and sends no provenance metadata to the provider.
+- **Opaque model input owned by an authenticated internal route** such as uploaded audio, image,
+  video, file bytes, or signed URLs: add `privateProvenance` to a projected request, or use
+  `mode: 'private-provenance'` when there is no textual projection. The route must call
+  `validateOpaqueModelInputProvenance` before downloading or sending content to the model and must
+  apply the workspace-file provenance guard before reading a persisted workspace file.
+- **Sim-owned durable storage** that can later re-enter a workflow/model (table cells, Agent
+  memory, knowledge documents/chunks, and workspace-file contents): transport encrypted
+  field-scoped provenance with
+  `request.secretProvenance`, persist it through the shared sidecar service, and load/import it at
+  every re-entry boundary. Preserve shared legacy behavior for rows/files whose provenance marker
+  is `NULL`; never invent a tool-local migration rule.
+
+Hard rules:
+
+- Never substitute secret plaintext into source or serialize plaintext provenance.
+- Never hand-roll private provenance headers/envelopes; the shared `executeTool` boundary owns
+  transport and strips private metadata from functional results.
+- Never attach private provenance to an external URL or to `directExecution`. Use the centralized
+  `opaqueModelInput` rejection mode for external/direct opaque model inputs, or an authenticated
+  internal route when encrypted provenance must cross the boundary.
+- Never sanitize arbitrary third-party tool results. Projection applies only to secrets activated
+  by Sim's resolved-secret provenance for that execution/tool call.
+- Do not add provenance merely because a value is persisted, returned by a tool, or appears in a
+  filename. Require a concrete Sim `{{...}}` resolution path and a later model/log boundary. If an
+  unsupported field can resolve a secret but does not justify durable tracking (for example a
+  `file_write` path), reject it at that exact ingress.
+- At diagnostic boundaries, project only values carrying execution-scoped provenance. Ordinary
+  provider responses, filenames, URLs, and errors remain unchanged when Sim did not resolve a
+  secret into them.
+
+Add focused tests covering named projection, ordinary identical text without provenance, nested
+shape preservation, malformed/incomplete private metadata failing closed, centralized external
+opaque rejection before formatting/I/O without byte changes or metadata transport, headerless
+legacy requests, and absence of private metadata in the public tool result. For durable sinks, also
+cover legacy `NULL` markers, exact-empty new writes, tracked secret writes, stale/missing sidecars,
+and scope isolation.
+
 ## Step 3: Create Block
 
 ### File Location
@@ -535,6 +592,11 @@ If creating V2 versions (API-aligned outputs):
 - [ ] Created `index.ts` barrel export
 - [ ] Registered all tools in `tools/registry.ts`
 - [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
+- [ ] Classified every model-visible, opaque, and Sim-durable request field
+- [ ] Added shared model-input projection, centralized opaque rejection, or private provenance only
+      where required
+- [ ] Confirmed ordinary third-party tool results are not generically sanitized
+- [ ] Added provenance compatibility and fail-closed boundary tests where applicable
 
 ### Block
 - [ ] Created `blocks/blocks/{service}.ts`

.agents/skills/validate-integration/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: validate-integration
-description: Validate an existing Sim integration (tools, block, registry) against the service's API docs
+description: Validate an existing Sim integration (tools, block, registry, and resolved-secret/model-input boundaries) against the service's API docs and Sim execution conventions
 argument-hint: <service-name> [api-docs-url]
 ---
 
@@ -129,6 +129,51 @@ For **every** tool file, check:
 - [ ] Registry keys use snake_case and match tool IDs exactly
 - [ ] Entries are in alphabetical order within the file
 
+### Resolved-Secret Provenance and Model Input
+
+For every request field, determine whether it is ordinary API input, model-visible text/structured
+content, opaque model input, or a value persisted into Sim-owned durable storage.
+
+Treat model-input provenance as opt-in. Require official documentation or an unambiguous local
+execution path proving that the exact field reaches an AI model. If the evidence is ambiguous,
+leave the integration unchanged; do not infer a model boundary merely from natural-language,
+search, extraction, or "AI-powered" marketing terminology.
+
+- [ ] AI-consumed text/structured fields use `request.modelInput` with `mode: 'project'` and a
+      minimal exact selector; nested/JSON-string adapters preserve shape through `applyProjected`
+- [ ] Opaque AI-consumed values sent directly to an external provider or `directExecution` use
+      `request.opaqueModelInput` with `mode: 'reject-resolved-secrets'` and an exact effective-value
+      selector; the central executor rejects incomplete/secret-bearing committed provenance before
+      formatting or I/O, leaves safe bytes unchanged, and sends no provenance metadata externally
+- [ ] Opaque AI-consumed files/bytes/URLs owned by an authenticated internal route use
+      `privateProvenance` (or `mode: 'private-provenance'`), and the route validates
+      `validateOpaqueModelInputProvenance` before any download or model call
+- [ ] Persisted workspace-file contents are checked with the shared provenance guard only when
+      their bytes or decoded content cross into a model/tool-result boundary; ordinary file APIs
+      remain unchanged. Unsupported secret-bearing file paths are rejected at `file_write`
+- [ ] Sim-owned durable writes that can re-enter workflows/models use field-scoped
+      `request.secretProvenance` and the shared sidecar/load/import services
+- [ ] Private provenance is never attached to external URLs or `directExecution`; those paths use
+      centralized `opaqueModelInput` rejection when their opaque values are model-bound
+- [ ] No tool performs raw secret plaintext/source substitution or serializes plaintext provenance
+- [ ] No `transformResponse` or tool-local helper blanket-sanitizes ordinary third-party results;
+      only execution-scoped, activated Sim provenance is projected at shared model/log boundaries
+- [ ] Private headers/envelopes are produced and stripped by the shared tool executor, never
+      hand-rolled or returned as functional output
+- [ ] Every added provenance hook has a concrete Sim `{{...}}` resolution path and a later
+      persistence/model/log crossing; there is no generic handling for arbitrary filenames,
+      metadata, provider results, or API payloads
+- [ ] Diagnostic projection is applied only to values carrying execution-scoped provenance;
+      ordinary provider responses, filenames, URLs, and errors are unchanged
+- [ ] Tests cover named `{{NAME}}` projection, unproven identical public text, nested shape
+      preservation, malformed/incomplete metadata, centralized opaque rejection before formatting
+      or I/O with safe-byte preservation, headerless legacy requests, metadata stripping, and
+      durable legacy/stale/scope cases when applicable
+
+Treat a missing or bypassed model/durable provenance boundary as **critical**. Do not fix it with a
+tool-specific string replacer or by sanitizing every provider result; repair the shared request,
+authenticated internal-route, persistence, or re-entry boundary that owns the data.
+
 ## Step 4: Validate Block
 
 ### Block ↔ Tool Alignment (CRITICAL)
@@ -301,6 +346,12 @@ Group findings by severity:
 - Service-account metadata disagrees with the canonical OAuth service configuration
 - `tools.config.tool` returning wrong tool ID for an operation
 - Type coercions in `tools.config.tool` instead of `tools.config.params`
+- AI-consumed request fields bypass the shared projection, centralized opaque rejection, or
+  private-provenance boundary
+- Opaque model input is downloaded or sent before provenance and workspace-file checks
+- A Sim-owned durable sink/re-entry path drops encrypted provenance or breaks legacy `NULL` data
+- A tool substitutes secret plaintext into source, leaks private metadata, or generically sanitizes
+  unrelated third-party results
 
 **Warning** (follows conventions incorrectly or has usability issues):
 - Optional field not set to `mode: 'advanced'`
@@ -375,6 +426,9 @@ After fixing, confirm:
 - [ ] Validated memory load safety using `.agents/skills/memory-load-check/SKILL.md` when tools list/search/download/import/export/batch data
 - [ ] Validated error handling (error checks, meaningful messages)
 - [ ] Validated registry entries (tools and block, alphabetical, correct imports)
+- [ ] Validated model-visible/opaque inputs and Sim-durable provenance at their owning boundaries
+- [ ] Confirmed legacy persisted data keeps working and tracked invalid provenance fails closed
+- [ ] Confirmed ordinary third-party results remain unchanged absent activated Sim provenance
 - [ ] Validated `{Service}BlockMeta` exported with at least 7 templates
 - [ ] Reported all issues grouped by severity
 - [ ] Fixed all critical and warning issues

.claude/commands/add-integration.md

@@ -1,5 +1,5 @@
 ---
-description: Add a complete Sim integration from API docs, covering tools, block, icon, optional triggers, registrations, and integration conventions. Use when introducing a new service under `apps/sim/tools`, `apps/sim/blocks`, and `apps/sim/triggers`.
+description: Add a complete Sim integration from API docs, covering tools, block, icon, optional triggers, registrations, resolved-secret/model-input safety, and integration conventions. Use when introducing a new service under `apps/sim/tools`, `apps/sim/blocks`, and `apps/sim/triggers`.
 argument-hint: <service-name> [api-docs-url]
 ---
 
@@ -121,6 +121,63 @@ export const {service}{Action}Tool: ToolConfig<Params, Response> = {
 - When using `type: 'json'` and you know the object shape, define `properties` with the inner fields so downstream consumers know the structure. Only use bare `type: 'json'` when the shape is truly dynamic
 - If you do not know the response JSON shape from docs or verified examples, you MUST tell the user and stop. Never guess outputs or response mappings.
 
+### Resolved Secrets at Model and Persistence Boundaries
+
+Classify every request field before implementing the tool:
+
+This is opt-in, not a blanket integration migration. Add a model-input declaration only when the
+service's official documentation or an unambiguous local execution path proves that the exact
+field is consumed by an AI model. If that cannot be established, preserve existing tool behavior
+and leave the field unannotated.
+
+- **Ordinary provider/API input:** leave it unchanged. Do not add blanket result sanitization.
+- **Text or structured content consumed by an AI model:** declare `request.modelInput` with
+  `mode: 'project'` and select only the exact model-visible fields. The shared executor replaces
+  activated Sim secrets with canonical `{{NAME}}` labels before request formatting. For nested or
+  JSON-string fields, use a small shared selector plus `applyProjected`; verify that selecting the
+  rebuilt params reproduces the projected selection.
+- **Opaque model input sent directly to an external provider** such as a model-read URL or image
+  payload: declare `request.opaqueModelInput` with `mode: 'reject-resolved-secrets'` and select only
+  the exact effective value. The shared `executeTool` preflight rejects incomplete or secret-bearing
+  committed provenance before URL/body formatting or network I/O, preserves safe request bytes,
+  and sends no provenance metadata to the provider.
+- **Opaque model input owned by an authenticated internal route** such as uploaded audio, image,
+  video, file bytes, or signed URLs: add `privateProvenance` to a projected request, or use
+  `mode: 'private-provenance'` when there is no textual projection. The route must call
+  `validateOpaqueModelInputProvenance` before downloading or sending content to the model and must
+  apply the workspace-file provenance guard before reading a persisted workspace file.
+- **Sim-owned durable storage** that can later re-enter a workflow/model (table cells, Agent
+  memory, knowledge documents/chunks, and workspace-file contents): transport encrypted
+  field-scoped provenance with
+  `request.secretProvenance`, persist it through the shared sidecar service, and load/import it at
+  every re-entry boundary. Preserve shared legacy behavior for rows/files whose provenance marker
+  is `NULL`; never invent a tool-local migration rule.
+
+Hard rules:
+
+- Never substitute secret plaintext into source or serialize plaintext provenance.
+- Never hand-roll private provenance headers/envelopes; the shared `executeTool` boundary owns
+  transport and strips private metadata from functional results.
+- Never attach private provenance to an external URL or to `directExecution`. Use the centralized
+  `opaqueModelInput` rejection mode for external/direct opaque model inputs, or an authenticated
+  internal route when encrypted provenance must cross the boundary.
+- Never sanitize arbitrary third-party tool results. Projection applies only to secrets activated
+  by Sim's resolved-secret provenance for that execution/tool call.
+- Do not add provenance merely because a value is persisted, returned by a tool, or appears in a
+  filename. Require a concrete Sim `{{...}}` resolution path and a later model/log boundary. If an
+  unsupported field can resolve a secret but does not justify durable tracking (for example a
+  `file_write` path), reject it at that exact ingress.
+- At diagnostic boundaries, project only values carrying execution-scoped provenance. Ordinary
+  provider responses, filenames, URLs, and errors remain unchanged when Sim did not resolve a
+  secret into them.
+
+Add focused tests covering named projection, ordinary identical text without provenance, nested
+shape preservation, malformed/incomplete private metadata failing closed, centralized external
+opaque rejection before formatting/I/O without byte changes or metadata transport, headerless
+legacy requests, and absence of private metadata in the public tool result. For durable sinks, also
+cover legacy `NULL` markers, exact-empty new writes, tracked secret writes, stale/missing sidecars,
+and scope isolation.
+
 ## Step 3: Create Block
 
 ### File Location
@@ -534,6 +591,11 @@ If creating V2 versions (API-aligned outputs):
 - [ ] Created `index.ts` barrel export
 - [ ] Registered all tools in `tools/registry.ts`
 - [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
+- [ ] Classified every model-visible, opaque, and Sim-durable request field
+- [ ] Added shared model-input projection, centralized opaque rejection, or private provenance only
+      where required
+- [ ] Confirmed ordinary third-party tool results are not generically sanitized
+- [ ] Added provenance compatibility and fail-closed boundary tests where applicable
 
 ### Block
 - [ ] Created `blocks/blocks/{service}.ts`