feat: pipeline backfill child workflow + never-fail error model#285
Open
feat: pipeline backfill child workflow + never-fail error model#285
Conversation
Adds a plan doc for restructuring the pipeline workflow: - Extract backfill into a child workflow (pipelineBackfillWorkflow) that can fail independently while the parent pipeline workflow never fails - Activity always returns errors instead of throwing - Per-stream error isolation via new `errored` stream status - Reclassify system_error into transient vs permanent - Recovery signals for targeted error resolution Made-with: Cursor Committed-By-Agent: cursor
6922495 to
d8f8980
Compare
Implements the backfill child workflow plan: Phase 1: pipelineSync activity always returns errors instead of throwing ApplicationFailure.retryable - eliminates workflow death path Phase 2: per-stream error isolation - errored streams get status 'errored' in source state so other streams can continue; eof fires when all streams are complete or errored Phase 3: new pipelineBackfillWorkflow child workflow that accumulates errors across chunks and fails with nonRetryable at eof if permanent errors exist Phase 4: pipelineWorkflow spawns pipelineBackfillWorkflow via startChild with race against runInterrupted for clean cancellation; backfill runs concurrently with liveLoop in Promise.all Phase 5: reclassify system_error - network errors and 5xx become transient_error; system_error is now permanent in the classifier Phase 6: TraceErrorException preserves failure_type through collectMessages instead of discarding it with plain Error Phase 7: recovery signals (credentials_updated, config_updated, deployment_updated) trigger error recovery in the pipeline workflow Made-with: Cursor Committed-By-Agent: cursor Made-with: Cursor Committed-By-Agent: cursor Made-with: Cursor Committed-By-Agent: cursor Made-with: Cursor Committed-By-Agent: cursor Made-with: Cursor Committed-By-Agent: cursor
d8f8980 to
81d6296
Compare
tonyxiao
pushed a commit
that referenced
this pull request
Apr 15, 2026
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the pipeline backfill child workflow plan — restructures the pipeline workflow so it never fails, with backfill extracted into a child workflow that can fail independently.
Phase 1: Activity never throws for classified errors
pipelineSyncalways returns{ errors, state, eof }instead of throwingApplicationFailure.retryablePhase 2: Per-stream error isolation
'errored'in source state and are skipped on subsequent chunkscompleteorerroredPhase 3:
pipelineBackfillWorkflowchild workflowpipelineSync, accumulates errors across chunks, evaluates at eofApplicationFailure.nonRetryableif permanent errors exist, otherwise returnsSourceStatePhase 4:
pipelineWorkflowspawns childrenstartChild+Promise.raceagainstrunInterrupted()for clean cancellationbackfillLoopruns concurrently withliveLoopinPromise.allPhase 5: Reclassify
system_errortransient_errorsystem_error, now classified as permanentPhase 6:
TraceErrorExceptioncollectMessagesthrowsTraceErrorExceptionpreservingfailure_typeinstead of plainErrorPhase 7: Recovery signals
credentials_updated,config_updated,deployment_updatedsignals trigger error recoveryTest plan
pnpm build)pnpm lint)pnpm format)Plan doc:
docs/plans/2026-04-14-backfill-child-workflow.md