Recover stalled integration tasks durably

[3mdistal]

Problem

Slack integration work is first persisted in SQL, then handed to a fresh serverless function for processing. On Netlify, the webhook function can finish and freeze before that self-dispatch leaves the process. The task row survives, but its only recovery loop was process-local, so the request could remain stranded and the user would never receive a Slack reply.

Approach

Keep SQL as the source of truth and add a durable way to wake persisted work on Netlify. When explicitly enabled for a selected integration scope, the initial handoff uses an acknowledged background function and a separate one-minute scheduler recovers eligible stale tasks. The existing portable self-dispatch remains the default and fallback, so other hosts do not depend on Netlify-specific behavior.

This changes task dispatch and recovery only. It preserves same-thread serialization and the existing delivery-only retry boundary: once an agent or Content mutation succeeds, retrying a failed Slack response must not repeat that mutation.

What changed

• Added an opt-in background handoff that waits for Netlify to acknowledge the queued execution, then falls back to the portable processor if acknowledgement fails.
• Added a signed, bounded recovery sweep with atomic claims and lease-aware stale cutoffs, preventing concurrent webhook, worker, and scheduler paths from processing the same task.
• Persisted the channel scope used for rollout decisions so later recovery applies the same allowlist as the initial handoff, including non-Slack integrations.
• Added scoped queue diagnostics for recent tasks and dispatch outcomes without exposing cross-tenant queue data.
• Generate the background worker and scheduler only for Dispatch in workspace deployments; standalone apps emit their own pair only when opted in.
• Updated integration guidance, generated skill copies, the Core changeset, and the Dispatch changelog.
• Stabilized an upstream A2A fake-timer test inherited from main by giving it the same loaded-CI wall-clock allowance as its sibling; runtime behavior is unchanged.

Safety and operations

• Rollout: AGENT_INTEGRATION_DURABLE_DISPATCH is default-off at build and runtime. AGENT_INTEGRATION_DURABLE_DISPATCH_SCOPES can limit enablement by platform, thread, or channel. With the feature disabled, the existing portable path remains unchanged.
• Rollback: Unset the rollout flag to return to the portable processor. No data rollback is required.
• Data: The queue schema changes are additive. They record dispatch attempts, outcomes, timestamps, and rollout scope; there is no backfill, deletion, or destructive migration.
• Authentication: The scheduler uses a short-lived HMAC token bound to the recovery route. If A2A_SECRET is missing, generated recovery functions log the configuration problem and return 204 instead of failing every minute.
• Cost: Enabling the feature adds one scheduled invocation per minute for each emitted recovery function, plus background executions for queued integration work. Merging with the flag disabled adds no scheduled recovery traffic.

Verification

• After integrating current main, 328 focused tests across nine dispatch, recovery, task-store, plugin, deployment-generation, and authentication files passed.
• Core typecheck, package build, and distribution import checks passed on the synchronized head.
• The isolated A2A client suite passed all 24 tests locally, including 10 consecutive runs after the test-only CI stabilization.
• Formatting and diff checks passed.
• All required GitHub checks pass on the final head, including Fast Tests, the standalone Chat smoke test, build, typecheck, integration suites, scaffold generation, security checks, and deploy previews.
• Builder's independent final-head review found no remaining production, security, concurrency, retry/replay, or compatibility issues; all review threads are resolved.

Browser QA was skipped because this PR does not change browser UI behavior. The production Slack acceptance story has not yet been run because the feature remains disabled.

Review focus

• Can any overlap between the webhook, background worker, and scheduled sweep bypass the atomic claim or choose the wrong stale cutoff?
• Does scoped enablement remain identical from initial dispatch through recovery and immediate successor handoff?
• Can recovery ever repeat a successful agent or Content mutation when only Slack delivery failed?
• Do both standalone and workspace-generated functions preserve HMAC authentication, safe missing-secret behavior, and Dispatch-only scheduler ownership?

Follow-ups

• Required before enablement: run the scoped production Slack create → manual Content rename → thread correction story and confirm that the same Content ID is updated, no duplicate record is created, and the confirmation is delivered exactly once.
• Operational monitoring: observe queue age, dispatch outcomes, recovery claims, and Netlify invocation/runtime usage during the scoped run.
• Later product decision: expand the scope allowlist only after the acceptance story and queue-health observation pass.

[github-actions]

Here's a visual recap of what changed:

Open the full interactive recap

[builder-io-integration]

Builder reviewed your changes — looks good ✅

Review Details

Incremental Code Review Summary

Reviewed the latest PR state with four parallel agents, including the new upstream A2A fake-timer test stabilization and the durable dispatch/recovery implementation. The recovery path continues to use atomic compare-and-set claims, subject-bound HMAC authentication, persisted dispatch scopes, tenant-aware task processing, and explicit deployment/runtime gating. No new confirmed production, security, concurrency, retry/replay, or backward-compatibility issues were identified.

One review pass questioned the absence of per-organization predicates in the workspace-wide recovery sweep. This sweep is intentionally a workspace-level internal operation over all eligible persisted tasks; each task remains processed with its own owner/org context, while the global rollout scope is configuration rather than an end-user authorization boundary. No cross-tenant data exposure or unauthorized user-controlled access path was demonstrated.

The additional test-only change is limited to relaxing a fake-timer deadline for loaded CI and does not alter runtime behavior. Focused reviewers reported 169–328 affected tests passing; unrelated auth log-count flakes were not attributable to this PR.

Risk classification remains High due to durable processing, recovery, authentication, and deployment behavior.

🧪 Browser testing: Skipped — PR only modifies core/server/deployment behavior and tests; no user-facing browser UI impact.