Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions packages/workshop-backend/__integration__/open-gadget-rpc.test.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { abortAllDurableObjects } from "cloudflare:test";
import { exports } from "cloudflare:workers";
import { newWebSocketRpcSession, type RpcStub } from "capnweb";
import {
Expand Down Expand Up @@ -133,3 +134,26 @@ describe.skip("openGadget errors across native RPC and Cap'n Web", () => {
expectRpcCode(browserError, OPEN_GADGET_ERROR_CODES.workspaceAccessDenied);
});
});

// The DO-reset recovery contract: a user-DO reset must not poison the API session. A stub is
// bound to one incarnation of the object and is permanently broken by a reset, so the
// authenticated API re-resolves its user-DO stub per call — the same session recovers on the
// next request with no reconnect. abortAllDurableObjects() is the non-graceful teardown, the
// local stand-in for the storage-timeout/overload resets observed in production. (Deliberately
// not evictDurableObject(): eviction is graceful — it drains in-flight work and never breaks a
// stub — so it cannot reproduce this failure.)
describe("user-DO reset recovery", () => {
it("recovers on the same session after the user DO is reset", async () => {
using publicApi = await connect();
const account = await createAccount(publicApi, "reset");
using authenticated = await publicApi.authenticate(account.token);

expect(await authenticated.listModels()).toBeInstanceOf(Array);

await abortAllDurableObjects();

// Same socket, same AuthenticatedApiImpl. With the per-call stub getter this reaches the
// restarted object; with a session-cached stub it would reject with the abort error forever.
expect(await authenticated.listModels()).toBeInstanceOf(Array);
});
});
19 changes: 13 additions & 6 deletions packages/workshop-backend/src/server.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { RpcStub, RpcTarget, newWorkersRpcResponse } from "capnweb";
import { validateRpc } from "capnweb-validate";
import type { JWTPayload } from "jose";
import { PublicApi, AuthenticatedApi, Overseer, GadgetMetadataWithTimestamps, AiChatAuthorInfo, AiModelConfig, AiGatewayInfo, AiModelProvider, ConnectedAccountsSubscriber, ConnectedAccountsFilter, GatekeeperVendorFilter, ObserverConfigCallback, BlueprintLibrarySummary, BlueprintPublicInfo, BlueprintUserSummary, BlueprintBindingAssignment, AgentSpawnerConfig, WorkpieceId, BLUEPRINT_SCREENSHOT_PATH_PREFIX, BLUEPRINT_SCREENSHOT_R2_PREFIX, blueprintScreenshotUrl, ServerConfig, CloudflareUsageInfo, CloudflareAccountOption, LoginAttempt, GatekeeperAppInfo, AdminApi, GatekeeperVendorInfo, OutputFormatOffer, ListOutputsResult, createOpenGadgetError, getOpenGadgetErrorCode, OPEN_GADGET_ERROR_CODES } from '@gadgets/workshop-shared/api';
import { PublicApi, AuthenticatedApi, Overseer, GadgetMetadataWithTimestamps, AiChatAuthorInfo, AiModelConfig, AiGatewayInfo, AiModelProvider, ConnectedAccountsSubscriber, ConnectedAccountsFilter, GatekeeperVendorFilter, ObserverConfigCallback, BlueprintLibrarySummary, BlueprintPublicInfo, BlueprintUserSummary, BlueprintBindingAssignment, AgentSpawnerConfig, WorkpieceId, BLUEPRINT_SCREENSHOT_PATH_PREFIX, BLUEPRINT_SCREENSHOT_R2_PREFIX, blueprintScreenshotUrl, ServerConfig, CloudflareUsageInfo, CloudflareAccountOption, LoginAttempt, GatekeeperAppInfo, AdminApi, GatekeeperVendorInfo, OutputFormatOffer, ListOutputsResult, createOpenGadgetError, getOpenGadgetErrorCode, OPEN_GADGET_ERROR_CODES, AUTH_ERROR_CODES, createAuthError } from '@gadgets/workshop-shared/api';
import type { UiFeatureFlags } from "@gadgets/workshop-shared/feature-flags";
import { getServerConfig } from "./deployment-config.js";
import { isPasswordAuthEnabled, getAuthGatekeeperAllowlist } from "./auth/config.js";
Expand Down Expand Up @@ -74,7 +74,7 @@ type Env = Cloudflare.Env & {
@validateRpc()
class AuthenticatedApiImpl extends RpcTarget implements AuthenticatedApi {
constructor(private ctx: ExecutionContext, private env: Env,
private user: DurableObjectStub<UserDurableObject>,

@ndisidore ndisidore Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is likely a controversial decision.
Its driven by https://developers.cloudflare.com/durable-objects/best-practices/error-handling/ specifically the block

Many exceptions leave the DurableObjectStub in a "broken" state, such that all attempts to send additional requests will just fail immediately with the original exception. To avoid this, you should avoid reusing a DurableObjectStub after it throws an exception. You should instead create a new one for any subsequent requests.

When the the user DO resets e.g. storage timeout, overloaded abort (which is exactly what we saw in the logs) that stub is permanently poisoned. Even if we retry it will fail.

This is not super obvious because the premise does hold for the workspace path: Overseer stubs get re-resolved through the namespace on each open, so a retried openGadget genuinely reaches the restarted object. The UserDO path is the only one where a stub is cached across calls.

This should be cheap and safe: namespace.get(id) is not a network call. Stub creation is local and lazy.

Why not just force a re-load? a reload doesn't avoid the retry; it is the retry, multiplied by everything else and makes blast radius wildly disproportionate. It give a worse UX as well as the entire page resets (as opposed to trying to recover silently where possible)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think it should be a problem, but do different stubs mean we lose request ordering to the userDO? i couldn't find any instances were that would be a big problem though, so it seems like a worthwhile tradeoff.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haha when writing this I told claude "have some pre-canned responses ready for the inevitable push back"
and it had one for e-order! the concern is valid. but practically nothing in our code relies on cross-call ordering through the user stub

preserving ordering while recovering poisoned stubs would require caching and centrally invalidating the stub after native RPC failures across every UserDO operation, adding substantial kernel complexity

private userId: DurableObjectId,
private abortSession: (reason: Error) => void) {
super();

Expand All @@ -87,6 +87,13 @@ class AuthenticatedApiImpl extends RpcTarget implements AuthenticatedApi {
private adminSettings: DurableObjectNamespace<AdminSettings>;
private users: DurableObjectNamespace<UserDurableObject>;

// A stub is bound to one incarnation of the DO and is poisoned once that incarnation resets,
// so re-resolve per call instead of caching one for the session (per the DO error-handling
// docs: create a fresh stub per attempt). Stub creation is local and lazy — not a network call.
private get user(): DurableObjectStub<UserDurableObject> {
return this.users.get(this.userId);
}

#isAdmin(): boolean {
let name = this.user.id.name;
let admins = this.env.ADMINS;
Expand Down Expand Up @@ -664,7 +671,7 @@ class PublicApiImpl extends RpcTarget implements PublicApi {
async authenticate(token: string): Promise<AuthenticatedApi> {
let split = token.split(':');
if (split.length !== 2) {
throw new Error("Invalid session token.");
throw createAuthError(AUTH_ERROR_CODES.invalidSessionToken);
}

let userId = this.users.idFromName(split[0]);
Expand All @@ -675,12 +682,12 @@ class PublicApiImpl extends RpcTarget implements PublicApi {
user_id: userId.toString(),
source: "session_token",
});
return new AuthenticatedApiImpl(this.ctx, this.env, stub, this.abortSession);
return new AuthenticatedApiImpl(this.ctx, this.env, userId, this.abortSession);
}

async authenticateFromCfAccess(): Promise<AuthenticatedApi> {
if (!this.accessPayload) {
throw new Error("Not authenticated with Access.");
throw createAuthError(AUTH_ERROR_CODES.notAuthenticatedWithAccess);
}

let email = this.accessPayload.email as string;
Expand All @@ -700,7 +707,7 @@ class PublicApiImpl extends RpcTarget implements PublicApi {
user_id: userId.toString(),
source: "cf_access",
});
return new AuthenticatedApiImpl(this.ctx, this.env, stub, this.abortSession);
return new AuthenticatedApiImpl(this.ctx, this.env, userId, this.abortSession);
}

async login(username: string, passwordHash: Uint8Array): Promise<string | null> {
Expand Down
13 changes: 10 additions & 3 deletions packages/workshop-backend/src/user.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { RpcStub } from "capnweb";
import { GadgetMetadataWithTimestamps, AiChatAuthorInfo, AiModelConfig, SUGGESTED_MODELS, CollaboratorRole, ConnectedAccountsSubscriber, ConnectedAccountsFilter, GatekeeperVendorFilter, GadgetMetadata, BlueprintMetadata, BlueprintLibrarySummary, BlueprintSource, BlueprintUserSummary, BLUEPRINT_SCREENSHOT_R2_PREFIX, GatekeeperVendorInfo, BlueprintOutput, OutputSummary, WorkpieceId, ListOutputsResult } from '@gadgets/workshop-shared/api';
import { GadgetMetadataWithTimestamps, AiChatAuthorInfo, AiModelConfig, SUGGESTED_MODELS, CollaboratorRole, ConnectedAccountsSubscriber, ConnectedAccountsFilter, GatekeeperVendorFilter, GadgetMetadata, BlueprintMetadata, BlueprintLibrarySummary, BlueprintSource, BlueprintUserSummary, BLUEPRINT_SCREENSHOT_R2_PREFIX, GatekeeperVendorInfo, BlueprintOutput, OutputSummary, WorkpieceId, ListOutputsResult, AUTH_ERROR_CODES, createAuthError } from '@gadgets/workshop-shared/api';
import { Gatekeeper, GatekeeperUser, GatekeeperUserVerifier, GatekeeperVendor, AccountDescription, VendorDescription, GatekeeperConnectCallback, SupportedResource, ResourceConfiguratorFrame, AppUiContext, GatekeeperUiFrame } from "@gadgets/workshop-shared/gatekeeper";
import { shouldAutoProvisionAccount, ambientGatekeeperMode } from "./provisioning-policy.js";
import { CloudflareGatekeeperUser } from "@gadgets/workshop-shared/cloudflare-gatekeeper";
Expand Down Expand Up @@ -296,12 +296,19 @@ export class UserDurableObject extends DurableObject<Cloudflare.Env> {
}

async authenticate(token: string): Promise<void> {
let tokenBytes = Uint8Array.fromBase64(token);
let tokenBytes: Uint8Array;
try {
tokenBytes = Uint8Array.fromBase64(token);
} catch {
// A corrupt (non-Base64) token must classify as an auth failure like any other bad token,
// not surface as the decoder's SyntaxError.
throw createAuthError(AUTH_ERROR_CODES.invalidSessionToken);
}
let hash = await crypto.subtle.digest('SHA-256', tokenBytes);
let tokenId = new Uint8Array(hash).toHex();
let session = this.storage.sessions.get(tokenId);
if (!session) {
throw new Error("invalid session token");
throw createAuthError(AUTH_ERROR_CODES.invalidSessionToken);
}
}

Expand Down
9 changes: 6 additions & 3 deletions packages/workshop-frontend/src/BlueprintLandingPage.tsx
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { logRpcFailure, withDoResetRetry } from './rpcErrors'
import { useState, useEffect, useCallback, useMemo, useRef, type ReactNode } from 'react'
import { useNavigate, useParams, useRouter } from '@tanstack/react-router'
import { RpcStub, RpcTarget } from 'capnweb'
Expand Down Expand Up @@ -120,7 +121,9 @@ export default function BlueprintLandingPage({ rpcStub }: Props) {
// When authenticated, fetch models for binding assignment.
useEffect(() => {
if (isAuthenticated && authenticatedApi) {
authenticatedApi.listModels().then(setModels).catch(console.error)
withDoResetRetry(() => authenticatedApi.listModels())
.then(setModels)
.catch(err => logRpcFailure('Failed to load models:', err))
} else {
setModels([])
}
Expand Down Expand Up @@ -181,7 +184,7 @@ export default function BlueprintLandingPage({ rpcStub }: Props) {
ready() {}
}

authenticatedApi.subscribeConnectedAccounts(new AccountsSubscriber())
withDoResetRetry(() => authenticatedApi.subscribeConnectedAccounts(new AccountsSubscriber()))
.then(stub => {
if (cancelled) {
stub[Symbol.dispose]()
Expand All @@ -190,7 +193,7 @@ export default function BlueprintLandingPage({ rpcStub }: Props) {
}
})
.catch(err => {
console.error('Failed to subscribe to connected accounts:', err)
logRpcFailure('Failed to subscribe to connected accounts:', err)
})

return () => {
Expand Down
41 changes: 33 additions & 8 deletions packages/workshop-frontend/src/ChatInterface.tsx
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { isDurableObjectResetError, isTransientRpcError, logRpcFailure, reportDoResetError } from "./rpcErrors";
import {
Fragment,
memo,
Expand Down Expand Up @@ -1780,6 +1781,7 @@ export const ChatInput = ({
attachLabel,
draftUpdateBanner,
blockedReason,
chatKey,
onStop,
showThinkingTraces = true,
onToggleThinkingTraces,
Expand Down Expand Up @@ -1825,6 +1827,8 @@ export const ChatInput = ({
/** When set, the composer is disabled and shows this message — the user must resolve something
* (e.g. accept/deny a pending connection request) before they can type or send. */
blockedReason?: string;
/** Identity of the chat the composer is bound to; a change clears chat-scoped hints. */
chatKey?: number | null;
onStop?: () => void;
showThinkingTraces?: boolean;
onToggleThinkingTraces?: () => void;
Expand All @@ -1838,6 +1842,11 @@ export const ChatInput = ({
const [capsules, setCapsules] = useState<InputCapsule[]>([]);
const [pendingAttachments, setPendingAttachments] = useState<PendingAttachment[]>([]);
const [isSending, setIsSending] = useState(false);
const [sendHiccup, setSendHiccup] = useState(false);
const chatKeyRef = useRef(chatKey);
chatKeyRef.current = chatKey;

useEffect(() => setSendHiccup(false), [chatKey]);
const [isAttachmentDragActive, setIsAttachmentDragActive] = useState(false);
const [selectedSlashCommand, setSelectedSlashCommand] = useState<SelectedSlashCommand | null>(null);
// The caret the slash command picker parses at. Deliberately updated only when it moves to a
Expand Down Expand Up @@ -2276,6 +2285,7 @@ export const ChatInput = ({

const handleSend = async () => {
if (sendInFlightRef.current || isSending || isBlocked) return;
setSendHiccup(false);
const attachmentsSnapshot = pendingAttachments;
const readyAttachments = attachmentsSnapshot
.filter((attachment) => attachment.uploadState === "ready" && attachment.ref)
Expand Down Expand Up @@ -2454,8 +2464,12 @@ export const ChatInput = ({
};

const submitMessage = () => {
const submittedChatKey = chatKey;
void handleSend().catch((err) => {
console.error("Failed to send chat message:", err);
// The onSend handlers already log; the composer only needs the hint state.
if (isTransientRpcError(err) && chatKeyRef.current === submittedChatKey) {
setSendHiccup(true);
}
});
};

Expand Down Expand Up @@ -3050,6 +3064,11 @@ export const ChatInput = ({
</div>
)}
{draftUpdateBanner}
{sendHiccup && (
<div className="px-4 pt-2 text-xs text-kumo-warning">
Connection hiccup — your message may not have been sent. Check the thread, then try again.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ndisidore what happens in this case?

  1. The user tries to send a message to chat
  2. The workspace reads the user’s profile and model configuration through its cached User DO connection.
  3. The User DO has reset, so that connection fails.
  4. The message is not written to the Overseer DO.

it looks like the message's body & attachments aren't thrown away, but retrying won't get a fresh User DO stub yet. so when the user tries to resend, will it work?

</div>
)}
{/* Textarea */}
<div className="relative px-4 pb-1 pt-3">
{slashCommandPicker.popup}
Expand Down Expand Up @@ -5213,9 +5232,10 @@ function ChatInterface({
forceUpdate();
}
} catch (err) {
console.error("Failed to subscribe to chats:", err);
reportIssue('chat.subscription-load', err)
toasts.add({ title: "Unable to load conversations", variant: "error" });
if (!logRpcFailure("Failed to subscribe to chats:", err)) {
reportIssue('chat.subscription-load', err)
toasts.add({ title: "Unable to load conversations", variant: "error" });
}
}
};

Expand Down Expand Up @@ -5366,8 +5386,10 @@ function ChatInterface({
);
}
} catch (err) {
console.error("Failed to send message:", err);
toasts.add({ title: "Failed to send message", variant: "error" });
if (isDurableObjectResetError(err)) reportDoResetError("chat.send", err);
if (!logRpcFailure("Failed to send message:", err)) {
toasts.add({ title: "Failed to send message", variant: "error" });
}
throw err;
}
};
Expand All @@ -5388,8 +5410,10 @@ function ChatInterface({
message, model, capsules, attachments, formats);
onNavigateToChatRef.current(newChatId);
} catch (err) {
console.error("Failed to create new chat:", err);
toasts.add({ title: "Failed to start conversation", variant: "error" });
if (isDurableObjectResetError(err)) reportDoResetError("chat.new", err);
if (!logRpcFailure("Failed to create new chat:", err)) {
toasts.add({ title: "Failed to start conversation", variant: "error" });
}
throw err;
}
};
Expand Down Expand Up @@ -7576,6 +7600,7 @@ function ChatInterface({
<div className={`flex-shrink-0 bg-kumo-base ${sidebarMode ? "" : "border-t border-kumo-line"}`}>
<div className={useConstrainedChatWidth ? "mx-auto w-full max-w-[920px]" : ""}>
<ChatInput
chatKey={selectedChatId}
createCapsuleGatekeeper={(accountId, url) =>
overseer.newGatekeeper(accountId, url)
}
Expand Down
3 changes: 2 additions & 1 deletion packages/workshop-frontend/src/ConnectAccountModal.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { RpcStub } from 'capnweb'
import { AuthenticatedApi, GatekeeperVendorFilter } from '@gadgets/workshop-shared/api'
import { VendorDescription } from '@gadgets/workshop-shared/gatekeeper'
import VendorCard from './VendorCard'
import { withDoResetRetry } from './rpcErrors'

interface ConnectAccountModalProps {
visible: boolean
Expand Down Expand Up @@ -41,7 +42,7 @@ export default function ConnectAccountModal({
const fetchVendors = async () => {
setVendorsLoading(true)
try {
const vendorList = await authenticatedApi.listGatekeeperVendors(filter)
const vendorList = await withDoResetRetry(() => authenticatedApi.listGatekeeperVendors(filter))
const unavailable = vendorList.filter(v => v.unavailable)
if (unavailable.length > 0) {
toasts.add({
Expand Down
2 changes: 2 additions & 0 deletions packages/workshop-frontend/src/Connections.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ export default function Connections({ overseer, gadget, chatId, authenticatedApi
setHooks(hookList.filter((hook) => hook.gadgetId === id))
onHasGatekeepersChange?.(bindingList.length > 0)
} catch (err) {
// Loud on purpose: this panel has no retry path, so a quieted transient failure would
// silently render "no connected resources".
console.error('Failed to load gatekeepers:', err)
reportIssue('connections.load', err)
toasts.add({ title: 'Failed to load connections', variant: 'error' })
Expand Down
9 changes: 5 additions & 4 deletions packages/workshop-frontend/src/GatekeeperModal.tsx
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { logRpcFailure, withDoResetRetry } from './rpcErrors'
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { Dialog, useKumoToastManager, type PortalContainer } from '@cloudflare/kumo'
import {
Expand Down Expand Up @@ -368,7 +369,7 @@ export default function GatekeeperModal({
setSpawnerEnv(
(spawnerEnvCandidatesRef.current ?? []).map(entry => ({ ...entry, enabled: true })))

authenticatedApi.listModels().then(models => {
withDoResetRetry(() => authenticatedApi.listModels()).then(models => {
if (cancelled) return
setAvailableModels(models)
if (models.length > 0) {
Expand All @@ -387,7 +388,7 @@ export default function GatekeeperModal({
toasts.add({ title: "Couldn't load AI models", variant: 'error' })
})

authenticatedApi.listGatekeeperVendors().then(vendors => {
withDoResetRetry(() => authenticatedApi.listGatekeeperVendors()).then(vendors => {
if (cancelled) return
setVendors(vendors)
}).catch(err => {
Expand Down Expand Up @@ -431,7 +432,7 @@ export default function GatekeeperModal({
}

const subscriber = new AccountsSubscriber()
authenticatedApi.subscribeConnectedAccounts(subscriber)
withDoResetRetry(() => authenticatedApi.subscribeConnectedAccounts(subscriber))
.then(stub => {
if (cancelled) {
stub[Symbol.dispose]()
Expand All @@ -440,7 +441,7 @@ export default function GatekeeperModal({
}
})
.catch(error => {
console.error('Failed to subscribe to connected accounts:', error)
logRpcFailure('Failed to subscribe to connected accounts:', error)
})

return () => {
Expand Down
11 changes: 7 additions & 4 deletions packages/workshop-frontend/src/ObserverConfigModal.tsx
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { withDoResetRetry } from './rpcErrors'
import { useState, useEffect, useRef } from 'react'
import { Dialog, Select, Loader, Text, useKumoToastManager } from '@cloudflare/kumo'
import { Warning, Plus, ArrowClockwise, CheckCircle } from '@phosphor-icons/react'
Expand Down Expand Up @@ -143,13 +144,15 @@ export default function ObserverConfigModal({
}
}

authenticatedApi
.subscribeConnectedAccounts(new Subscriber(), { includeForcedAutoProvisionedAccounts: true })
withDoResetRetry(() => authenticatedApi
.subscribeConnectedAccounts(new Subscriber(), { includeForcedAutoProvisionedAccounts: true }))
.then(stub => {
if (cancelled) { stub[Symbol.dispose](); return }
subStub = stub
})
.catch(err => {
// Loud on purpose: the modal has no retry path, so a quieted transient failure would
// strand the user on a permanent loader.
console.error('Failed to subscribe to connected accounts:', err)
toasts.add({ title: 'Failed to load your connected accounts', variant: 'error' })
})
Expand All @@ -163,10 +166,10 @@ export default function ObserverConfigModal({
// ── load vendor metadata for display and resource-scope resolution ─────────────
useEffect(() => {
let cancelled = false
Promise.all([
withDoResetRetry(() => Promise.all([
authenticatedApi.listGatekeeperVendors(),
authenticatedApi.listAddableGatekeepers(),
])
]))
.then(([vendors, addable]) => {
if (cancelled) return
const map = new Map<string, GatekeeperVendorInfo>()
Expand Down
Loading
Loading