Skip to content

fix: retry transient result publication - #81

Merged
hjcud merged 1 commit into
developfrom
feature/80-result-transport-retry
Aug 9, 2026
Merged

fix: retry transient result publication#81
hjcud merged 1 commit into
developfrom
feature/80-result-transport-retry

Conversation

@hjcud

@hjcud hjcud commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

What changed

  • retry only transient URLError / TimeoutError failures inside HttpResultSink
  • reuse the exact same event body, headers, and delivery attempt for at most three transport attempts
  • use bounded exponential backoff (250ms, 500ms by default)
  • never retry HTTPError or a non-200 response
  • preserve the existing A terminal failure must survive its own publish failing, and correlation ID must be a validated UUID #74 behavior after exhaustion: the handler keeps its original terminal reason and does not turn the job into HANDLER_ERROR retry

Incident evidence

INT03 run 9d4a31d5-d432-356a-9570-11dc9026cf77 correctly became terminal INSTRUMENT_CATALOG_VERSION_UNSUPPORTED, but its FAILED event hit one HTTPS read timeout. The RUNNING event had succeeded 10 seconds earlier, API health remained 200, and the worker remained healthy. A one-shot result transport therefore left the API run RUNNING even though the job decision was permanent.

TDD and verification

The three new tests first failed because HttpResultSink had no bounded retry controls. After implementation:

  • transient timeout then success: same request, one logical event — passed
  • three transient failures: exactly three attempts and bounded backoff — passed
  • HTTP 503: exactly one attempt, no retry — passed
  • existing terminal reason preservation tests — passed
  • Ruff — passed
  • mypy — passed, 74 source files
  • all non-Docker tests — passed, 3 skipped
  • git diff --check — passed

No release/run, worker/ASG/DLQ/DB operation, or existing-run mutation was performed by this change.

Closes #80

@hjcud
hjcud merged commit e58b92d into develop Aug 9, 2026
9 checks passed
@hjcud
hjcud deleted the feature/80-result-transport-retry branch August 9, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: retry transient terminal result transport without retrying the job

1 participant