perf improvements

Author: icecrasher321

apps/docs/content/docs/en/platform/enterprise/self-hosted.mdx

@@ -23,7 +23,7 @@ ENTERPRISE_ENABLED=true
 NEXT_PUBLIC_ENTERPRISE_ENABLED=true
 ```
 
-That turns on organizations, permission groups, SSO, whitelabeling, audit logs, session policies, data retention, data drains, workspace forks, and the inbox.
+That turns on organizations, permission groups, SSO, whitelabeling, audit logs, session policies, data retention, data drains, workspace forks, sandboxes, and the inbox.
 
 ### Turning one feature off
 
@@ -51,6 +51,9 @@ The individual flags also work on their own if you would rather opt in one at a
 | Data drains | `DATA_DRAINS_ENABLED` | `NEXT_PUBLIC_DATA_DRAINS_ENABLED` |
 | Workspace forks | `FORKING_ENABLED` | — |
 | Sim Mailer inbox | `INBOX_ENABLED` | `NEXT_PUBLIC_INBOX_ENABLED` |
+| Sandboxes | `SANDBOXES_ENABLED` | `NEXT_PUBLIC_SANDBOXES_ENABLED` |
+
+Sandboxes also need a remote execution provider, since the deployment builds the images itself: set `E2B_API_KEY`, or `SANDBOX_PROVIDER=daytona` with a `DAYTONA_API_KEY` that has `write:snapshots` and `write:sandboxes`. Without one the settings section appears but builds fail.
 
 <Callout type="warning">
   Data retention is the one feature that deletes data. Its flag controls the cleanup pass, not the settings screen — retention windows are always configurable. Nothing is ever deleted until you enable it, and even then only against windows you configured explicitly. Sim never applies the hosted plan defaults to a self-hosted deployment.

apps/docs/content/docs/en/workflows/blocks/function.mdx

@@ -74,6 +74,82 @@ Beyond the Python standard library, the sandbox ships E2B's data-science stack p
 - **Math and testing:** `sympy`, `pytest`
 - **Sim additions:** `awscli`, `yq`, `csvkit`
 
+## Sandboxes
+
+A sandbox is a named dependency set your workspace maintains — a language plus a
+list of pip or npm packages. Select one on a Function block and its code can
+import everything on that list. Leave it empty and the block runs on the default
+image, exactly as before.
+
+Create and edit sandboxes in **Settings → Sandboxes**. Only workspace admins can
+create or edit them. On sim.ai they need an active Max or Enterprise plan;
+self-hosted deployments turn them on with `SANDBOXES_ENABLED` (see
+[self-hosted enterprise](/platform/enterprise/self-hosted)). The section is
+hidden when a deployment has no sandbox provider configured.
+
+1. **Name** the sandbox — `bigquery-etl`, `scraping`, whatever the job is.
+2. Pick the **language**. A sandbox is language-scoped, so a Python block only
+   ever lists Python sandboxes.
+3. Paste your **dependencies**, one per line. Version pins are optional.
+
+```
+google-cloud-bigquery==3.25.0
+pyairtable>=3.0
+pandas
+```
+
+Then open the block's advanced options and choose the sandbox under **Sandbox**.
+
+In JavaScript the sandbox applies to code that uses `import` or `require` — that is
+what sends the block to a remote sandbox in the first place, so a block without them
+keeps running locally and ignores the selection. Python always runs remotely, so a
+selected sandbox always applies.
+
+<Callout type="info">
+Two sandboxes with the same language and the same package list share one build,
+so duplicating a set costs nothing. Editing a package list starts a new build;
+runs already in flight keep using the old one. Deleting a sandbox frees its build
+once nothing else uses it.
+</Callout>
+
+### Build status
+
+On sim.ai, each dependency set is prebuilt into a reusable image, so runs pay no
+install cost. The status row in Settings shows **Queued**, **Building**,
+**Ready**, or **Failed**. A failed build reports what went wrong — a package that
+does not exist, a version that has no match, a resolver conflict — with the
+installer log behind a disclosure.
+
+Running a block before its sandbox is **Ready** stops the run and shows you the
+status. A failed build is retried periodically on its own; to retry immediately,
+save the sandbox again in Settings.
+
+On a self-hosted deployment using Daytona, dependencies install inside the
+sandbox at the start of every run instead, adding roughly 10–30 seconds per
+execution. Prebuilt images require E2B.
+
+### What is allowed
+
+Package names and version specifiers only. URLs, `git+` references, `-e`, local
+paths, `--index-url`, and npm aliases are rejected, with the offending line
+number reported. A sandbox may declare up to 50 packages.
+
+## Scoping secrets for agent tools
+
+When a Function block is used as an Agent tool, its code can read every workspace
+secret by default — both `{{MY_SECRET}}` and `environmentVariables['MY_SECRET']`.
+
+To narrow that, set **Secret access** to *Selected secrets* in the block's
+tool configuration and pick the names the code may read. Two things change:
+
+- Only those secrets are injected. `{{OTHER_SECRET}}` no longer resolves either.
+- The selected **names** are added to the tool's description, so the model knows
+  what it can reference. Values are never sent to the model — they are injected
+  server-side at execution.
+
+Leaving the default (*All secrets*) resolves the list at run time, so a secret
+added next month is included automatically.
+
 ## Examples
 
 ### Reshape an API response
@@ -170,6 +246,6 @@ The lazy `sim.files` and `sim.values` helpers are available only in JavaScript f
   { question: "When does code run locally vs. in a sandbox?", answer: "JavaScript without external imports runs in a local isolated sandbox for speed. JavaScript that uses import or require runs in E2B. Python always runs in the E2B sandbox, with or without imports." },
   { question: "How do I reference outputs from other blocks inside my code?", answer: "Use angle-bracket syntax directly, like <agent.content> or <api.data>, with no quotes around the tag — Sim replaces it with the real value before execution. For environment variables, use double curly braces: {{API_KEY}}." },
   { question: "What does the Function block return?", answer: "Two outputs: result (the return value of your code, read as <function.result>) and stdout (anything logged with console.log or print, read as <function.stdout>). Include a return statement in JavaScript, or print JSON in Python, to pass data downstream." },
-  { question: "Can I make HTTP requests from a Function block?", answer: "Yes. fetch() is available in JavaScript with async/await; libraries like axios are not, only the built-in fetch. In Python, use requests or httpx in the E2B sandbox." },
+  { question: "Can I make HTTP requests from a Function block?", answer: "Yes. fetch() is available in JavaScript with async/await; libraries like axios are only available when the block has a sandbox selected. In Python, use requests or httpx in the E2B sandbox." },
   { question: "Is there a timeout for Function block execution?", answer: "Yes, a configurable execution timeout. If your code exceeds it, the run is terminated and the block reports an error. Keep this in mind for external calls or heavy processing." },
 ]} />

apps/sim/app/api/credentials/route.test.ts

@@ -3,17 +3,24 @@
  *
  * @vitest-environment node
  */
-import { auditMock, authMockFns, createMockRequest, posthogServerMock } from '@sim/testing'
+import {
+  auditMock,
+  authMockFns,
+  createMockRequest,
+  dbChainMockFns,
+  posthogServerMock,
+  resetDbChainMock,
+} from '@sim/testing'
 import { beforeEach, describe, expect, it, vi } from 'vitest'
 import { TokenServiceAccountValidationError } from '@/lib/credentials/token-service-accounts/errors'
 
 const {
   mockCheckWorkspaceAccess,
-  mockGetWorkspaceMembership,
+  mockGetCredentialCreationWorkspaceContext,
   mockVerifyAndBuildServiceAccountSecret,
 } = vi.hoisted(() => ({
   mockCheckWorkspaceAccess: vi.fn(),
-  mockGetWorkspaceMembership: vi.fn(),
+  mockGetCredentialCreationWorkspaceContext: vi.fn(),
   mockVerifyAndBuildServiceAccountSecret: vi.fn(),
 }))
 
@@ -25,7 +32,7 @@ vi.mock('@/lib/workspaces/permissions/utils', () => ({
 }))
 
 vi.mock('@/lib/credentials/environment', () => ({
-  getWorkspaceMembership: mockGetWorkspaceMembership,
+  getCredentialCreationWorkspaceContext: mockGetCredentialCreationWorkspaceContext,
 }))
 
 vi.mock('@/lib/credentials/oauth', () => ({
@@ -52,6 +59,7 @@ const WORKSPACE_ID = '11111111-2222-4333-8444-555555555555'
 describe('POST /api/credentials', () => {
   beforeEach(() => {
     vi.clearAllMocks()
+    resetDbChainMock()
     authMockFns.mockGetSession.mockResolvedValue({
       user: { id: 'user-1', name: 'Test User', email: 'test@example.com' },
     })
@@ -60,7 +68,12 @@ describe('POST /api/credentials', () => {
       canWrite: true,
       canAdmin: true,
     })
-    mockGetWorkspaceMembership.mockResolvedValue({ ownerId: 'user-1', memberUserIds: ['user-1'] })
+    mockGetCredentialCreationWorkspaceContext.mockResolvedValue({
+      ownerId: 'user-1',
+      organizationId: 'org-1',
+      memberUserIds: ['user-1'],
+      canWrite: true,
+    })
   })
 
   describe('client-credential service accounts', () => {
@@ -156,5 +169,42 @@ describe('POST /api/credentials', () => {
       expect(data.error).toContain('clientSecret is required')
       expect(mockVerifyAndBuildServiceAccountSecret).not.toHaveBeenCalled()
     })
+
+    it('re-authorizes a personal credential after the shared org/user locks', async () => {
+      mockGetCredentialCreationWorkspaceContext
+        .mockResolvedValueOnce({
+          ownerId: 'user-1',
+          organizationId: 'org-1',
+          memberUserIds: ['user-1'],
+          canWrite: true,
+        })
+        .mockResolvedValueOnce({
+          ownerId: 'org-owner',
+          organizationId: 'org-1',
+          memberUserIds: ['org-owner'],
+          canWrite: false,
+        })
+
+      const req = createMockRequest('POST', {
+        workspaceId: WORKSPACE_ID,
+        type: 'env_personal',
+        envKey: 'MY_API_KEY',
+      })
+
+      const response = await POST(req)
+      const data = await response.json()
+
+      expect(response.status).toBe(403)
+      expect(data).toEqual({ error: 'Write permission required' })
+      expect(mockGetCredentialCreationWorkspaceContext).toHaveBeenCalledTimes(2)
+      expect(dbChainMockFns.execute).toHaveBeenCalled()
+      expect(mockGetCredentialCreationWorkspaceContext.mock.invocationCallOrder[0]).toBeLessThan(
+        dbChainMockFns.execute.mock.invocationCallOrder[0]
+      )
+      expect(dbChainMockFns.execute.mock.invocationCallOrder.at(-1)).toBeLessThan(
+        mockGetCredentialCreationWorkspaceContext.mock.invocationCallOrder[1]
+      )
+      expect(dbChainMockFns.insert).not.toHaveBeenCalled()
+    })
   })
 })

apps/sim/app/api/credentials/route.ts

@@ -13,6 +13,7 @@ import {
 } from '@/lib/api/contracts/credentials'
 import { getValidationErrorMessage, parseRequest } from '@/lib/api/server'
 import { getSession } from '@/lib/auth'
+import { acquireOrganizationUserMutationLocks } from '@/lib/billing/organizations/membership'
 import { generateRequestId } from '@/lib/core/utils/request'
 import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
 import {
@@ -21,7 +22,7 @@ import {
   SHARED_CREDENTIAL_TYPES,
 } from '@/lib/credentials/access'
 import { AtlassianValidationError } from '@/lib/credentials/atlassian-service-account'
-import { getWorkspaceMembership } from '@/lib/credentials/environment'
+import { getCredentialCreationWorkspaceContext } from '@/lib/credentials/environment'
 import { syncWorkspaceOAuthCredentialsForUser } from '@/lib/credentials/oauth'
 import {
   ServiceAccountSecretError,
@@ -509,10 +510,52 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
       resolvedProviderId === SLACK_CUSTOM_BOT_PROVIDER_ID && clientCredentialId
         ? clientCredentialId
         : generateId()
-    const { ownerId: workspaceOwnerId, memberUserIds: workspaceMemberUserIds } =
-      await getWorkspaceMembership(workspaceId)
 
-    await db.transaction(async (tx) => {
+    const creationResult = await db.transaction(async (tx) => {
+      /**
+       * Discover the organization lock scope inside this transaction, then
+       * acquire the same organization → user → membership locks as org
+       * removal/transfer and re-authorize from the transaction before writing.
+       *
+       * If this insert wins, transfer sees the new source-owned personal
+       * credential and blocks. If transfer wins, its permission/member cleanup
+       * is visible to the authoritative re-read below and the insert is
+       * refused.
+       */
+      const plannedContext = await getCredentialCreationWorkspaceContext({
+        executor: tx,
+        workspaceId,
+        userId: session.user.id,
+      })
+      if (!plannedContext) {
+        return { success: false as const, status: 403 as const, error: 'Write permission required' }
+      }
+
+      await acquireOrganizationUserMutationLocks(tx, {
+        userId: session.user.id,
+        organizationIds: plannedContext.organizationId ? [plannedContext.organizationId] : [],
+      })
+
+      const currentContext = await getCredentialCreationWorkspaceContext({
+        executor: tx,
+        workspaceId,
+        userId: session.user.id,
+        forUpdate: true,
+      })
+      if (!currentContext) {
+        return { success: false as const, status: 403 as const, error: 'Write permission required' }
+      }
+      if (currentContext.organizationId !== plannedContext.organizationId) {
+        return {
+          success: false as const,
+          status: 409 as const,
+          error: 'Workspace organization changed while creating the credential. Please retry.',
+        }
+      }
+      if (!currentContext.canWrite) {
+        return { success: false as const, status: 403 as const, error: 'Write permission required' }
+      }
+
       // service_account has no DB-level unique index on (workspaceId, providerId,
       // displayName), so we re-check inside the tx. OAuth/env_* are guarded by
       // partial unique indexes and fall through to the 23505 handler below.
@@ -542,9 +585,9 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
         updatedAt: now,
       })
 
-      if ((type === 'env_workspace' || type === 'service_account') && workspaceOwnerId) {
-        if (workspaceMemberUserIds.length > 0) {
-          for (const memberUserId of workspaceMemberUserIds) {
+      if ((type === 'env_workspace' || type === 'service_account') && currentContext.ownerId) {
+        if (currentContext.memberUserIds.length > 0) {
+          for (const memberUserId of currentContext.memberUserIds) {
             const isAdmin = memberUserId === session.user.id
             await tx.insert(credentialMember).values({
               id: generateId(),
@@ -572,7 +615,12 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
           updatedAt: now,
         })
       }
+
+      return { success: true as const }
     })
+    if (!creationResult.success) {
+      return NextResponse.json({ error: creationResult.error }, { status: creationResult.status })
+    }
 
     const [created] = await db
       .select()

apps/sim/app/api/cron/cleanup-sandbox-images/route.ts

@@ -0,0 +1,31 @@
+import { createLogger } from '@sim/logger'
+import { getErrorMessage } from '@sim/utils/errors'
+import { type NextRequest, NextResponse } from 'next/server'
+import { verifyCronAuth } from '@/lib/auth/internal'
+import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
+import { runCleanupSandboxImages } from '@/background/cleanup-sandbox-images'
+
+export const dynamic = 'force-dynamic'
+
+const logger = createLogger('CleanupSandboxImagesAPI')
+
+/**
+ * Retention sweep for prebuilt sandbox images. A runtime-strategy deployment has
+ * nothing to collect, so the sweep is a no-op there rather than a special case
+ * here.
+ */
+export const GET = withRouteHandler(async (request: NextRequest) => {
+  const authError = verifyCronAuth(request, 'sandbox image cleanup')
+  if (authError) return authError
+
+  try {
+    const result = await runCleanupSandboxImages()
+    return NextResponse.json({ success: true, ...result })
+  } catch (error) {
+    logger.error('Failed to sweep sandbox images', { error })
+    return NextResponse.json(
+      { error: getErrorMessage(error, 'Failed to sweep sandbox images') },
+      { status: 500 }
+    )
+  }
+})