Add Functions gRPC core helpers#282
Conversation
Expose low-level protobuf codecs and single work-item execution helpers for Azure Functions Durable JS gRPC consolidation without adding Functions metadata support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Expose stable byte-oriented worker processing methods and endpoint/taskHub client options for Azure Functions Durable JS gRPC consolidation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the work-item-executor extraction and base64 protobuf helpers with a minimal, Python-aligned implementation. processOrchestratorRequest and processEntityBatchRequest now reuse the existing worker execution path via an in-process capturing stub (mirroring durabletask-python PR #155's null-stub pattern) instead of refactoring the worker. Removed the base64 helper module, the object-level execute* methods, the V2 EntityRequest host path, and the related exports and tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a minimal “single work-item” execution surface to the Durable Task JS worker so host integrations (e.g., Azure Functions) can execute one orchestrator/entity batch request per invocation without running the long-lived gRPC streaming loop. This aligns with the SDK’s role as a low-level Durable Task Scheduler (sidecar) client/worker implementation.
Changes:
- Added byte-level helper methods on
TaskHubGrpcWorkerto process serializedOrchestratorRequest/EntityBatchRequestpayloads and return serialized responses. - Introduced an in-process
CapturingSidecarStubthat captures completion payloads instead of sending them over gRPC. - Added end-to-end tests and README documentation for the new host-integration surface.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| README.md | Documents the new low-level host integration APIs and clarifies the non–Durable Functions programming model scope. |
| packages/durabletask-js/test/functions-grpc-support.spec.ts | Adds e2e-style tests covering the new byte processors for orchestrations and entities. |
| packages/durabletask-js/src/worker/task-hub-grpc-worker.ts | Implements the new public byte-processing methods and the capturing sidecar stub. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t glue Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ease pipeline - Add DurableOrchestrationContext/DurableEntityContext (context.df.* adapters) + wrapOrchestrator/wrapEntity - Add RetryOptions, callHttp (throws), parentInstanceId; align newGuid/callSubOrchestrator with v3 signatures - Add v3 client query-return types (DurableOrchestrationStatus, OrchestrationRuntimeStatus, EntityStateResponse, PurgeHistoryResult) - Add client getStatus/getStatusAll/getStatusBy/readEntityState/purgeInstanceHistory/startNew/waitForCompletionOrCreateCheckStatusResponse - Add deprecated DurableOrchestrationClient alias - Wire host-provided maxGrpcMessageSizeInBytes into gRPC channel options (Python parity) - Remove dead CapturingSidecarStub.abandonRequest field (keep required no-op method) - Add durable-functions package to build/release pipelines
| "dependencies": { | ||
| "@azure/functions": "^4.16.1", | ||
| "@grpc/grpc-js": "^1.14.4", | ||
| "@microsoft/durabletask-js": "0.3.0" |
There was a problem hiding this comment.
TODO: remember to bump this to 0.4.0, and we need to release @microsoft/durabletask-js first
also confirm whether @azure/functions 4.16.1 has the extension changes required,
There was a problem hiding this comment.
Release-readiness findings (version bump to 0.4.0 is done in code; the two open items below are release-time actions):
1) @azure/functions 4.16.1 floor — confirmed adequate.
- v4.16.0 carried the substantive change: bumped
@azure/function-extensions-base0.2.0 → 0.3.0 and dropped Node 18 from CI (which lines up with this package's Node >= 22 floor). - v4.16.1 is a security-servicing release over 4.16.0 (MSRC updates); v4.16.2 is functionally identical to 4.16.1 (only reclaimed the npm
latesttag from an erroneously published 3.6.0). - This package uses only stable generic-binding APIs (
trigger.generic/input.genericcarryingdurableRequiresGrpc), available since v4.0, and does not depend on@azure/function-extensions-basedirectly. So^4.16.1is a sound "recent + security-patched" floor (it resolves to 4.16.2 today). The actual durable-gRPC enablement (durableRequiresGrpc→ gRPC channel) is host-extension-side, not gated by the npm library version.
2) Core release ordering — a real blocker to verify before publish.
- The core package (
@microsoft/durabletask-js) on this branch still declaresversion: 0.3.0, but it adds new public APIs this package relies on (processOrchestratorRequest,addNamedOrchestrator,CapturingSidecarStub, …). The published0.3.0on npm does NOT contain those. - This package's dependency is pinned to
"@microsoft/durabletask-js": "0.3.0". Locally the monorepo workspace links the in-tree core (which has the new APIs), so E2E passes; but once published to npm,0.3.0would resolve to the old published core and fail at runtime (processOrchestratorRequest is not a function). - Release checklist: (a) bump the core version (e.g. 0.4.0) and publish
@microsoft/durabletask-jsfirst; (b) update this package's dep from0.3.0to the new core version (0.4.0/^0.4.0); (c) then publishdurable-functions@0.4.0.
Leaving this thread open until the core is released and the dependency is bumped.
Drop rootDir, outDir, include, and exclude since they are inherited from ./tsconfig.json. Keep only the baseUrl + paths override that points @microsoft/durabletask-js imports at the built dist output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ment call style and update tests
…to DurableFunctionsClient
…m/microsoft/durabletask-js into pr/282/yunchuwang/functions-grpc-support
… Functions - Introduced `EntityId` class for classic v3 entity identifiers. - Enhanced `DurableEntityContext` to include `isNewlyConstructed` and `entityId` properties. - Implemented `signalEntity` method in `DurableEntityContext` for signaling other entities. - Updated `DurableOrchestrationContext` to track and set custom status. - Added unit tests for new features and behaviors.
…ayload in DurableFunctionsClient
Collapse the 4.0.0-alpha.0 changelog to a single bullet and rewrite the README to focus on what the package supports and why it is needed, dropping the implementation-plan / phase-status / open-questions sections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…onContext type aliases Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rator Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… context.log to classic orchestration context Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…3-compatible) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…InstanceHistoryBy (v3-compat) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
andystaples
left a comment
There was a problem hiding this comment.
Review summary
Two things ship here: (1) the advertised core gRPC byte-processors (processOrchestratorRequest / processEntityBatchRequest + CapturingSidecarStub) in task-hub-grpc-worker.ts, and (2) an entire new durable-functions@4.0.0-alpha.0 package (packages/azure-functions-durable) — the intended next-major replacement for Azure/azure-functions-durable-js v3.
The core helpers are clean, and reusing the existing execution path with an in-process capturing stub is a nice low-risk approach. The v3 compat adapters (context.df.* for orchestrations/entities and the client method surface) are genuinely broad and well documented.
My main concern is the "drop-in / run unchanged" framing in the README: this is a strong compatibility shim for the common method surface, but several heavily-used v3 patterns and the entire durable-client authoring model are not covered — and there's one silent-failure bug in the orchestrator adapter. Details are inline; grouped here:
Bug (please fix)
wrapOrchestratorpicks classic-vs-native by parameter arity, which silently mis-runs a native single-arg orchestrator (async function*(ctx)) — the body never executes and no error is thrown.
Behavioral breaks vs v3 (worth closing or documenting)
createTimerreturns a non-cancelableTask; theTask.any+timeoutTask.cancel()timeout pattern breaks, and theTaskresult shape changed (isCompleted/isFaulted/result→isComplete/isFailed/getResult()).- Orchestration/entity contexts no longer extend
InvocationContext. - Missing top-level exports:
DummyOrchestrationContext/DummyEntityContext(breaks users' tests), error types (DurableError/AggregatedError/TaskFailedError) forinstanceof,ManagedIdentityTokenSource/TokenSource; theapp.client.*durable-client trigger helpers andsetExceptionPropertiesProviderare also gone. - Client deltas:
getStatusreturns| undefinedand ignoresshowHistory;createCheckStatusResponsenow requires a definedrequest;startNewdrops theversionoption.
Nits
parseJsondoesn't guard""(JSON.parse("")throws).- Core: a version-mismatch abandon makes
processOrchestratorRequestthrow a generic error. - Spurious
deserializeBinaryreorder in the generated proto.d.ts. engines.node >= 22vs@types/node@^18; the Node-22 floor is a notable consumer bump.
Overall a promising foundation — I'd fix the arity bug and soften/annotate the compatibility claims before this reads as a transparent v3 replacement. Leaving this as comments (non-blocking).
… empty-string parseJson Two self-contained fixes in the durable-functions v4 compat package: - rewind(): delegate to core TaskHubGrpcClient.rewindInstance() instead of throwing 'rewind is not yet supported'; the optional reason defaults to an empty string. - parseJson(): treat empty-string serialized input/output/customStatus as undefined so getStatusAll/getStatusBy no longer throw 'Unexpected end of JSON input'. Adds one regression test per fix (durable-functions unit suite: 67 tests, all green). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brings the PR branch up to date with main so the compat layer can build against the current core public API: cancellable TimerTask (#293), native AggregateError WhenAll semantics (#302), rewind support (#296), and related fixes. Clean auto-merge (proto + worker). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolves andystaples' review comments plus a ListAll E2E bug on packages/azure-functions-durable: - #1 orchestrator arity: detect generator kind (sync=classic v3, async=core native) instead of an arity heuristic; driver now handles sync+async generators so single-param native orchestrators actually execute. - #2 parseJson tolerates non-JSON serializedOutput (fixes getStatusAll 400). - #3 createCheckStatusResponse accepts undefined request (baseUrl fallback). - #4 createTimer returns TimerTask (cancelable) in its type signature. - #5 re-export TaskFailedError from core + compat; document removed v3-only exports in CHANGELOG/README. - #6/#7 document getStatus + classic-context breaking changes. - #8 bump @types/node to ^22 to match engines.node>=22. - #9 set version 0.4.0 + document release order. - #10 distinct abandon error for the single work-item execution path. - #11 revert proto codegen churn to origin/main. Adds regression tests for items #1/#2/#3/#5/#10. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…, startNew version
- (1) orchestration-context: detect classic vs core-native orchestrators by generator/async kind (async* and async => native; sync generator => classic; plain sync => arity fallback) and drive both sync/async generators; add end-to-end regression tests through the real core executor.
- (2) createTimer already returns cancelable TimerTask (type now visible); add type/Task.any cancel test.
- (3) doc-only: classic contexts no longer extend InvocationContext (entity ctx is {df}); add replay-nondeterminism rationale to CHANGELOG/README.
- (4) createCheckStatusResponse already accepts undefined request with baseUrl fallback (v3-faithful).
- (5) getStatus returns non-optional DurableOrchestrationStatus and throws on not-found (v3 parity); showHistory populates core history; showInput maps to fetchPayloads; startNew forwards v3 version option. Docs note gRPC-path boundaries.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- getStatus throws on not-found with a v3-style 'DurableClient error:' message (verified v3 DurableClient.getStatus throws on the extension's HTTP 404; returns non-optional DurableOrchestrationStatus). - showInput now gates only the top-level input (payloads always fetched); output/customStatus are always returned, matching v3. - showHistoryOutput now strips input/result payloads from history entries when falsy and keeps them when true. history stays core HistoryEvent[] (v3 types history as Array<unknown>). - startNew forwards the v3 version option (already wired to core scheduleNewOrchestration). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
v3 DurableClient.getStatus throws on the extension's HTTP 404 with 'This usually means we could not find any data associated with the instanceId provided: <id>.'. Mirror that sentence verbatim (only the HTTP-404-specific first sentence is replaced, since the gRPC path has no HTTP 404). Update the not-found tests to assert the v3 message. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The single-work-item abandon path is JS-specific for the Azure Functions host integration. durabletask-python has no equivalent single-work-item helper, and its worker-loop abandon hands the item back over a real sidecar stub rather than no-opping, so the prior 'matches the Python provider, whose null stub no-ops' note was inaccurate. Comment-only change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The prior note referenced durabletask-python behavior, which is fragile: the Python single-work-item support lives on an unmerged fork PR and its details can drift. Trim the comment to the JS-only rationale for the no-op abandon on the single-work-item host path. Comment-only change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…pers for v3 parity Restores the 10 client-starter registration helpers (http, timer, storageBlob, storageQueue, serviceBusQueue, serviceBusTopic, eventHub, eventGrid, cosmosDB, generic). Each adds the durableClient input binding and injects a DurableFunctionsClient as the handler's 2nd arg, so (trigger, client, context) handlers work without manual df.input.durableClient() + df.getClient(context). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Surface the classic v3 client.startNew alias alongside the canonical scheduleNewOrchestration in the README examples, and add a verified alias -> core method table. Docs-only; these aliases already ship in client.ts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Functions host E2E workflow (#303) was workflow_dispatch-only. Enable it on pull_request scoped to the Functions surface (test/e2e-functions/**, packages/azure-functions-durable/**, and this workflow file) so changes to the consolidated gRPC path are exercised; core-only PRs keep their own suites. Also build the in-repo durable-functions (+ core) packages after the root npm ci and before the test-app install, since the test-app now consumes them via file: links (see the test-app repoint) rather than published packages. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
…t<TState> The v3-compat EntityContext<TState>/EntityHandler<TState> aliases accepted a state type parameter but ignored it, so a bare context.df.getState() returned unknown. Genuine v3 entity code (e.g. the ported BasicNode ClassBasedEntities sample) relies on getState() returning the context's TState, so it failed to compile against the compat package. Make DurableEntityContext generic over TState and default the getState<T = TState> method generic to it, so a bare getState() returns TState | undefined (matching v3) while an explicit getState<X>() still overrides. Because TState only appears as an erased method-generic default, DurableEntityContext<A> and <B> stay structurally identical, so app.entity() registration still accepts specifically typed handlers (verified: full compat unit suite green). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
The ported BasicNode test-app depended on published durable-functions@^3 and @azure/functions@^4.11.2. Repoint it at the in-repo v4 compat build so the E2E suite exercises this branch: - durable-functions -> file:../../../packages/azure-functions-durable - @azure/functions -> ^4.16.1 (matches the compat's target extension build) - overrides += @microsoft/durabletask-js -> file:../../../packages/durabletask-js The core override is required: the compat pins core to a published version that predates the Functions worker helpers (processOrchestratorRequest/ processEntityBatchRequest), so without it npm resolves stale published core and the worker path fails at runtime. Regenerated the test-app lock against the file: refs (npm ls confirms both resolve to the local packages). The v3-authored samples build unchanged against the v4 SDK (tsc green). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
…ipApiVersionCheck The Functions host E2E suite exercises the consolidated gRPC path, which only exists in the extension's preview bundle. Point test-app/host.json at Microsoft.Azure.Functions.ExtensionBundle.Preview [4.29.0, 5.0.0) (which carries the DurableRequiresGrpc / ConfigureForGrpcProtocol path from extension #3260) and lift the client gRPC receive cap via maxGrpcMessageSizeInBytes = int.MaxValue so >4 MB status/output reads no longer trip the 4 MiB default. The preview bundle's Azure Storage SDK targets a newer REST API version than current Azurite accepts, so start Azurite with --skipApiVersionCheck everywhere it is launched (CI workflow, the local wrapper script, and the README) — without it the storage listeners fail and every orchestration times out. Also correct the now-stale "published durable-functions" docs to reflect the file:-linked in-repo wiring the test-app actually uses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
…ty gaps remain Revert the pull_request trigger added in 2982f34 (keeping that commit's build-order fix and file:-linking comment updates). The consolidated gRPC path still has 11 known error-text/status fidelity gaps in the ported specs -- not-found message wording, activity-failure prefixes, and terminal-op 400-vs-200 semantics -- which would make an automatic pull_request run red. Those gaps are tracked separately; until they are addressed the suite runs on manual dispatch only and does not gate PRs. Also quote the "Build ... for file: linking" step name so the workflow is valid YAML: an unquoted plain scalar containing ": " is rejected by spec-compliant parsers, which would otherwise block workflow_dispatch as well. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
Re-enable the automatic pull_request trigger (scoped to test/e2e-functions/**, packages/azure-functions-durable/**, and this workflow file) alongside workflow_dispatch, now that the error-fidelity gaps are being fixed to green. Reconcile the workflow header + README wording to reflect PR-triggered runs and note that the remaining terminal/invalid-state control-plane specs are skipped pending a server-side gRPC status-detail fix (tracked separately). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
…face terminate/suspend/resume/raiseEvent run over the consolidated gRPC path, which collapses failures into opaque codes. Add a shared _mapControlPlaneError helper: - NOT_FOUND (5) -> rethrow `No instance with ID '<id>' found.` (v3 parity). - UNKNOWN (2) on a NON-terminal instance -> look up the runtime status and rethrow `Cannot <op> orchestration instance in the <State> state.` (matches host wording). - terminal instance / any other error -> rethrow unchanged (terminal-op status policy deferred to #315). Add 6 focused unit tests: not-found (terminate + raiseEvent), non-terminal state (suspend + resume), terminal rethrow, and non-gRPC passthrough. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
…315 Over the gRPC path, terminate/suspend/resume of a terminal (Completed/Terminated) instance surfaces an opaque `2 UNKNOWN` (HTTP 400) instead of v3's swallow -> 200. Skip the 3 affected specs (terminate-completed, terminate-terminated, and suspend/resume-completed) with references to #315, which tracks the terminal-op status policy / server-side status-detail fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
…idelity The consolidated gRPC path surfaces its own error shapes rather than v3's extension-HTTP wrapping. Update the Node localized expectation strings to match: - Activity failures -> `TaskFailedError: Activity task #N failed:` (core does not carry the activity/orchestrator name, so v3's `... function '<name>' failed:` no longer applies). - Entity failures -> `EntityOperationFailedException: Operation 'get' of entity '@counter@myCounter' failed:` (CaughtEntityException replaces the v3 bug #642 `Error: [object Object]`, intentionally not reproduced). - suspend/resume invalid-state -> the friendly `Cannot <op> orchestration instance in the <State> state.` message the compat now rethrows (v3's HTTP-500 string was a transport artifact, not a real message). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e5be22b-b727-4ccf-8668-60397bb403a0
What
Adds the minimal host-integration surface so Azure Functions can drive a single durable work item per invocation, without running the long-lived gRPC worker loop. This mirrors the Azure Functions consolidation done for Python in durabletask-python PR #155.
API
Two methods on
TaskHubGrpcWorker:processOrchestratorRequest(request: Uint8Array): Promise<Uint8Array>processEntityBatchRequest(request: Uint8Array): Promise<Uint8Array>Each deserializes a TaskHubSidecarService protobuf payload, executes one work item, and returns the serialized response. Host integrations own any transport encoding (for example base64); base64 stays out of the core SDK.
Design (aligned with Python #155)
Rather than refactoring the worker, these methods reuse the worker's existing internal execution path (
_executeOrchestratorInternal/_executeEntityInternal) and pass an in-processCapturingSidecarStubthat records the completion payload instead of sending it over gRPC. This is the same null-stub pattern Python uses (AzureFunctionsNullStub).Deliberately NOT included (kept minimal)
execute*public methods (the byte processors are the single entry point).EntityRequesthost path (not present in Python fix: Reset customStatus on continue-as-new in InMemoryOrchestrationBackend #155; the worker's internal V2 handling for the DTS backend is unchanged).metadataGenerator.Tests
packages/durabletask-js/test/functions-grpc-support.spec.tscovers both byte processors end to end.