Skip to content

fix: preserve terminal failures when result publication fails - #75

Merged
kcrmin merged 1 commit into
developfrom
fix/74-terminal-failure-survives-publish
Aug 9, 2026
Merged

fix: preserve terminal failures when result publication fails#75
kcrmin merged 1 commit into
developfrom
fix/74-terminal-failure-survives-publish

Conversation

@hjcud

@hjcud hjcud commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

What changed

  • preserve a terminal JobNotSatisfiable decision and its original reason code when publishing the failure event itself raises
  • log the publication exception with the run, attempt, and original reason instead of letting the outer worker convert it to HANDLER_ERROR:* and retry
  • validate and normalize BACKTEST_WORKER_CORRELATION_ID as a UUID before building any other production worker dependency
  • add focused coverage for successful terminal failure publication, publication failure fallback, fail-fast invalid configuration, and canonical UUID output

Root cause

A malformed worker correlation ID caused terminal failure-event construction to raise. That secondary exception escaped OrchestratorJobHandler, so the outer worker classified it as a retryable handler error and hid the original REQUIRED_INPUT_UNAVAILABLE decision.

Impact

Terminal binding failures now remain permanent even when their result event cannot be published, and their original reason survives in worker attempt/DLQ handling. Invalid correlation IDs fail at process startup instead of remaining latent until the first result event.

This PR changes only backtest-engine. The backend compiled-plan handoff and root user-data UUID generation remain separately owned and are not modified here.

TDD evidence

Before implementation, the three new regression tests failed with:

  • OSError: result sink unavailable escaping the handler
  • the unrelated missing runtime-policy setting appearing before correlation validation
  • no _required_uuid configuration validator

After implementation:

  • python -m ruff check src tests — passed
  • python -m mypy — passed, 74 source files
  • python -m pytest tests/test_wiring.py tests/test_production.py tests/test_worker.py -q — passed, 61 tests
  • python -m pytest -p no:cacheprovider -m "not docker" — 1240 passed, 3 skipped, 168 deselected
  • git diff --check — passed

No AWS run, release, DLQ operation, worker restart, ASG operation, or database mutation was performed.

Closes #74

@kcrmin

kcrmin commented Aug 9, 2026

Copy link
Copy Markdown
Member

검토하고 병합합니다. 요청한 두 가지가 정확히 들어갔습니다 — 발행 실패를 삼키고 원래 reason_codePERMANENT_FAILURE 를 보존하며, _required_uuid 가 다른 의존성 구성보다 먼저 실행되어 잘못된 값이 기동 시점에 걸립니다.

루트 쪽과 맞물리는 것도 확인했습니다. user-data 가 공급하는 파생 UUID 가 str(uuid.UUID(value)) 정규화를 통과하고 값이 바뀌지 않습니다.

i-07a6870a8c4c199dc -> b915ae58-9bfd-5c36-8a36-f28a590403d2   정규화 결과 동일
i-000276dbff2ac0e58 -> 2fec26d4-6d05-51b8-8b58-98938a242337
i-0417d4380dfe35adf -> 33bebc4d-d3af-5755-8cff-dade185b4266

이 저장소는 원래 hjcud 소유이고 제가 병합·후속 작업을 하는 것은 제품 권한자(kcrmin)의 지시에 따른 것입니다 — "hjcud한테 왔다갔다 하는것 없이 그냥 이거 관련된 작업들은 여기서 작업해줘". 왕복을 줄여 INT03 을 오늘 끝내기 위한 조치이고, 이후 이 저장소의 일반 소유는 그대로입니다.

같은 릴리스에 backend #250(체크섬 handoff)과 루트 user-data UUID 수정을 함께 올립니다.

@kcrmin
kcrmin merged commit a6b094d into develop Aug 9, 2026
9 checks passed
@kcrmin
kcrmin deleted the fix/74-terminal-failure-survives-publish branch August 9, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A terminal failure must survive its own publish failing, and correlation ID must be a validated UUID

2 participants