ci: continuously roll over long-running OTel tests

[zhongkechen]

Summary

• poll after completing the delayed callback and validate terminal executions in the same check job
• report known premature failures immediately while retaining active state and blocking rollover
• migrate incomplete legacy check results through the current callback-aware checker so due callbacks can advance
• launch the next invocation and execution runs only after every execution reaches a terminal state
• preserve callback progress across pending checks and transient history-read failures
• refresh the telemetry validation window after callback completion polling
• support in-flight runs whose legacy check results omit rollover_ready by deriving readiness from all saved execution histories
• propagate resolved SDK and conformance-test commits into replacement runs
• retain normalized span dumps for successful and failed long-running validations

Testing

• hatch run test:all (325 passed)
• hatch run types:check
• hatch run yaml:lint .github/workflows
• hatch check code
• hatch check fmt
• git diff --check

[zhongkechen]

Addressed all three findings in 044b10c:

• Added an explicit rollover_ready result/output. It is true only after every execution is terminal; partial premature failures remain pending, preserve state, and cannot launch replacements.
• Refresh the completion timestamp after callback polling before constructing telemetry queries and reports.
• Treat transient history misses during callback polling as retryable. If polling times out, the check returns pending with state_changed=true, and the workflows upload the callback-sent state before retiring the old artifact.

Added focused regressions for partial/terminal premature failures, transient polling reads, refreshed validation time, and all three workflow contracts. Full validation: 320 tests passed, plus typing, YAML lint, Ruff, formatting, and diff checks.

[zhongkechen]

Addressed the in-flight compatibility issue in cd69b79.

When a checked-out legacy source writes a result without rollover_ready, the reusable workflow now:

• limits fallback handling to legacy passed/failed results;
• reads the exact execution-history filenames listed in the persisted state;
• accepts both top-level terminal statuses and terminal execution events;
• sets rollover ready only when every expected history is present and terminal.

This lets legacy terminal successes and failures roll over while keeping partial-premature failures blocked. Added an executable missing-field regression covering legacy success, all-terminal failure, and partial failure. Validation on the rebased PR head: 325 tests passed, plus typing, YAML lint, Ruff, formatting, and diff checks.

[zhongkechen]

Addressed both findings in e2a1299.

• Known premature executions are now reported as failed with rollover_ready: false after due callbacks are sent and their state is persisted. The active state/stack remains available while other executions finish.
• When an old checker returns a result without rollover_ready and saved histories are incomplete, the workflow now reruns the check with the current callback-aware implementation. This advances pending callbacks instead of retaining the same frozen legacy state.

Added regressions for premature failure plus a pending callback and for legacy failed-result migration. Full local validation: 325 tests passed, plus types, YAML lint, Ruff, formatting, and git diff --check.

[github-actions]

Codex AI review

No actionable findings. Residual risk remains around live AWS callback polling and GitHub Actions artifact/output handoff, which unit tests cannot fully exercise.

Reviewed commit e2a1299816cba5fb3e9d33691e5aedd1f1e045bf. Workflow run

[github-actions]

Claude AI review

Reviewed the full diff against the base revision — the long_running.py check() refactor (callback-progress tracking, _poll_pending_executions, premature/terminal/pending branch matrix, rollover_ready), the otel_write_trace_artifact change in extension.py, and the rollover_ready/check_status output plumbing plus legacy-migration shell logic across all six workflow files and their tests.

No actionable findings. I traced each branch and could not confirm any correctness, typing, error-handling, or schema-compatibility defect:

• callback_progressed captures the pre-mutation callback_sent_at_ms, polls only when a callback was freshly sent to a still-running execution, and skips polling on the recovered-callback and terminal paths.
• _poll_pending_executions is deadline-bounded, tolerates transient get_execution_history failures without losing persisted callback state, and cannot loop indefinitely.
• The pending / pending+premature / terminal / terminal+premature branches set state_changed/rollover_ready consistently; the new Persist updated callback state and Retire previous state artifact conditions rotate the same-named artifact correctly (upload-new-then-delete-old, delete-only on rollover, retain on no-change/error).
• main()'s reordering validates check_timeout/check_interval before the check branch (which now uses them) while launch still returns before touching those non-existent attributes.
• bool(options.get("otel_write_trace_artifact")) preserves prior behavior in the general validation flow and always dumps spans only for long-running.
• rollover_ready is empty on short/launch phases, so next-long-running-* never fires on PR/push/fork runs; test-count and legacy bash-path assertions are covered.

Residual test risk (not defects):

• Rollover relaunch (.github/workflows/python-opentelemetry.yml:344, and the java/typescript equivalents) deletes the state artifact and immediately launches the next run in the same workflow run; the launch guard exit 1s if the just-deleted artifact still lists. This tightens a former ~24h window to seconds and depends on GitHub's artifact-list API reflecting the DELETE promptly — plausible flakiness source, untested, but not confirmable as a defect.
• check_status output (e.g. python-opentelemetry-long-running.yml:11) is declared but consumed by no caller — harmless.
• Callback-send, X-Ray validation, and artifact upload/delete paths remain mocked in unit tests and are only exercised end-to-end by scheduled live runs.

Reviewed commit e2a1299816cba5fb3e9d33691e5aedd1f1e045bf. Workflow run