Skip to content

feat: pipeline backfill child workflow + never-fail error model#285

Open
tonyxiao wants to merge 2 commits intov2from
plan/backfill-child-workflow
Open

feat: pipeline backfill child workflow + never-fail error model#285
tonyxiao wants to merge 2 commits intov2from
plan/backfill-child-workflow

Conversation

@tonyxiao
Copy link
Copy Markdown
Collaborator

@tonyxiao tonyxiao commented Apr 14, 2026

Summary

Implements the pipeline backfill child workflow plan — restructures the pipeline workflow so it never fails, with backfill extracted into a child workflow that can fail independently.

Phase 1: Activity never throws for classified errors

  • pipelineSync always returns { errors, state, eof } instead of throwing ApplicationFailure.retryable
  • Eliminates the workflow-death path where 10 retries exhaust and the workflow dies

Phase 2: Per-stream error isolation

  • Errored streams get status 'errored' in source state and are skipped on subsequent chunks
  • Other streams continue to completion; eof fires when all streams are complete or errored

Phase 3: pipelineBackfillWorkflow child workflow

  • Loops pipelineSync, accumulates errors across chunks, evaluates at eof
  • Throws ApplicationFailure.nonRetryable if permanent errors exist, otherwise returns SourceState

Phase 4: pipelineWorkflow spawns children

  • Uses startChild + Promise.race against runInterrupted() for clean cancellation
  • backfillLoop runs concurrently with liveLoop in Promise.all
  • Initial backfill and periodic reconcile both use the same child workflow

Phase 5: Reclassify system_error

  • Network errors (ECONNRESET, ETIMEDOUT, etc.) and Stripe 5xx → transient_error
  • Everything else stays system_error, now classified as permanent

Phase 6: TraceErrorException

  • collectMessages throws TraceErrorException preserving failure_type instead of plain Error

Phase 7: Recovery signals

  • credentials_updated, config_updated, deployment_updated signals trigger error recovery

Test plan

  • All 20 service workflow tests pass (2 pre-existing skips)
  • All 100 source-stripe tests pass
  • All 46 protocol tests pass
  • New test: "continues past returned transient errors without entering error state"
  • Build passes (pnpm build)
  • Lint passes (pnpm lint)
  • Format passes (pnpm format)

Plan doc: docs/plans/2026-04-14-backfill-child-workflow.md

@tonyxiao tonyxiao changed the title plan: pipeline backfill child workflow + never-fail pipeline feat: pipeline backfill child workflow + never-fail error model Apr 14, 2026
Adds a plan doc for restructuring the pipeline workflow:
- Extract backfill into a child workflow (pipelineBackfillWorkflow) that
  can fail independently while the parent pipeline workflow never fails
- Activity always returns errors instead of throwing
- Per-stream error isolation via new `errored` stream status
- Reclassify system_error into transient vs permanent
- Recovery signals for targeted error resolution

Made-with: Cursor
Committed-By-Agent: cursor
@tonyxiao tonyxiao force-pushed the plan/backfill-child-workflow branch 4 times, most recently from 6922495 to d8f8980 Compare April 14, 2026 07:35
Implements the backfill child workflow plan:

Phase 1: pipelineSync activity always returns errors instead of
throwing ApplicationFailure.retryable - eliminates workflow death path

Phase 2: per-stream error isolation - errored streams get status
'errored' in source state so other streams can continue; eof fires
when all streams are complete or errored

Phase 3: new pipelineBackfillWorkflow child workflow that accumulates
errors across chunks and fails with nonRetryable at eof if permanent
errors exist

Phase 4: pipelineWorkflow spawns pipelineBackfillWorkflow via
startChild with race against runInterrupted for clean cancellation;
backfill runs concurrently with liveLoop in Promise.all

Phase 5: reclassify system_error - network errors and 5xx become
transient_error; system_error is now permanent in the classifier

Phase 6: TraceErrorException preserves failure_type through
collectMessages instead of discarding it with plain Error

Phase 7: recovery signals (credentials_updated, config_updated,
deployment_updated) trigger error recovery in the pipeline workflow

Made-with: Cursor
Committed-By-Agent: cursor
Made-with: Cursor
Committed-By-Agent: cursor
Made-with: Cursor
Committed-By-Agent: cursor
Made-with: Cursor
Committed-By-Agent: cursor
Made-with: Cursor
Committed-By-Agent: cursor
@tonyxiao tonyxiao force-pushed the plan/backfill-child-workflow branch from d8f8980 to 81d6296 Compare April 14, 2026 07:42
tonyxiao pushed a commit that referenced this pull request Apr 15, 2026
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant