chore: Harden integration-test crash uploads in CI by tustanivsky · Pull Request #1496 · getsentry/sentry-unreal

tustanivsky · 2026-07-24T08:10:56Z

The integration tests were flaky in CI for several distinct reasons - each confirmed from the failure artifacts this PR started capturing:

Shutdown upload races - non-crash tests (message / log / metric / tracing) upload through the SDK transport on sentry_close, bounded by ShutdownTimeout (default 5000 ms). Large envelopes or a slow link can exceed that window and the in-flight upload gets cancelled. The same drain-on-shutdown bound matters for the sentry-xbox extension, so we want generous headroom here regardless of the desktop legs. On the crashpad legs, the crashing process can also exit before the out-of-process handler finishes its upload unless it is configured to wait.
OutOfMemory, Windows native backend - the OOM test exhausted the whole runner's memory, so the sentry-crash daemon captured the crash but sometimes failed to upload the envelope (WinHttpSendRequest error 8, ERROR_NOT_ENOUGH_MEMORY); with no retry or durable fallback the event was lost.
StackOverflow, Windows, both backends - no thread stack guarantee was in effect due to sentry-native bugs, so the in-process crash handler had no stack to run on and no minidump was created.
Android crash test (SauceLabs) - real devices emit thousands of unrelated logcat lines per second (in one failing capture, 7.8k of the 10k captured lines were AuthPII spam spanning 1.2 s), evicting UE's EVENT_CAPTURED marker before the harness reads the log. The crash itself succeeded.
Replay duration assertion - the server derives replay duration from segment timestamps, which shift with crash-handling timing, so an exact-match assertion is inherently flaky.

Key Changes

sample/Config/DefaultEngine.ini

ShutdownTimeout=30000 - gives the native daemon's crash upload and all async (non-crash) event uploads time to finish on shutdown before the transport is torn down. Drains early when the queue empties, so there is no slowdown when uploads are fast.
CrashpadWaitForUpload=True - on the crashpad legs, the process waits for the crashpad upload to complete before exiting, so the crash is reliably delivered during the crash run.

integration-test-windows.yml / integration-test-linux.yml

Include logs from sample-build/SentryPlayground/.sentry-native/ in the on-failure artifacts so the crash DB (minidumps, pending/completed uploads, daemon logs) is available when a test fails.

integration-test-android.yml

SAUCE_LOGCAT_FILTER (via feat(sauce): Filter Android logcat by given tags app-runner#53, already in the pinned revision) captures only UE/Sentry/tombstone tags so app markers can't be evicted by device spam.

SentryPlaygroundTestUtils.{h,cpp}, SentryCrashTest.cpp

New SetMemoryLimit(HeadroomMB) util: the process self-assigns a Job Object with JOB_OBJECT_LIMIT_PROCESS_MEMORY = current commit + headroom (Windows-only, no-op elsewhere). The OOM test calls it with 4 GB, so allocation fails in-process while the runner (and the crash daemon) keep full memory headroom. Also drops the OOM test from ~6.5 min of pagefile thrashing to ~7 s.

Integration.Desktop.Tests.ps1

Dropped the exact replay-duration assertion; the server derives duration from segment timestamps, which shift with crash-handling timing.

Notes

ShutdownTimeout and the .sentry-native artifact help both the crashpad and native desktop legs; CrashpadWaitForUpload is a no-op on the native legs, so it is safe there.
The .sentry-native artifact path assumes DatabaseLocation=ProjectUserDirectory resolves to the packaged project dir (sibling of Saved/Logs/).

Related Items

#skip-changelog

The desktop crash test can be flaky because both desktop backends can race the upload on shutdown: crashpad uploads out-of-process while the crashing process is already exiting, and the native out-of-process daemon drains its transport bounded by ShutdownTimeout (default 5000ms), which can be too short for a large envelope or slow link (the same root cause seen dropping crashes on the Xbox native backend). Non-crash tests upload via the SDK transport on sentry_close and hit the same limit. Raise ShutdownTimeout so the native daemon's crash upload and async event uploads have time to drain, and enable CrashpadWaitForUpload so the crashpad legs wait for the crash upload before exiting. Also include the sentry-native database (.sentry-native) in the on-failure artifacts for the Windows and Linux integration tests to simplify diagnosing missing/failed uploads.

actions/upload-artifact skips hidden files by default, which dropped the dotted .sentry-native crash DB (daemon log, minidumps) from the failure artifacts.

Stack-overflow crash capture intermittently dies in-process (secondary 0xC0000005, no minidump produced) because the default 64 KiB guarantee is not always enough for UE's handler chain. SENTRY_HANDLER_STACK_SIZE is read by sentry-native at init, so no rebuild is needed.

Points the submodule at fix/thread-stack-guarantee-static-init (master + fix): kernel32 caching now precedes backend startup in static builds so the default thread stack guarantee is actually applied, and the native backend now sets it like the other backends do. This should let the in-process handlers survive stack-overflow crashes in the Windows integration tests.

Dispatched runs get their own concurrency group (run_id fallback), so multiple validation runs can execute in parallel without canceling the PR run.

The crash-capture test intermittently fails with "Expected 1 EVENT_CAPTURED line(s) but found 0": real SauceLabs devices emit thousands of unrelated log lines per second (AuthPII alone was ~8k of the ~10k captured lines), evicting UE's output from the captured log window before the harness reads it. Use app-runner's SAUCE_LOGCAT_FILTER (getsentry/app-runner#53, already in the pinned revision) to filter at capture time.

The server computes replay duration from segment timestamps, which shift with crash handling timing (observed 6 vs the sidecar's nominal 5 seconds), so an exact match is inherently flaky.

The stack guarantee fix branch was rebased for the upstream PR (getsentry/sentry-native#1918); the previously pinned commit no longer exists on any branch.

The thread stack guarantee fix (getsentry/sentry-native#1918) will be picked up via a separate dependency update PR once it merges upstream.

Minidumps and envelopes contain process memory of the crashed test app (on Linux even the full environment block), which may hold CI secrets. Restrict the artifact to the daemon logs, which carry the diagnostic value.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1db445a. Configure here.}

JoshuaMoelans

LGTM, just small q on the Replay Duration

JoshuaMoelans · 2026-07-28T11:57:57Z

-        It "Should have expected replay duration" {
-            # The staged clip's sidecar claims a 5000 ms duration
-            $script:ReplayEvent.duration | Should -Be 5
-        }
-


q: does it make sense to fully remove the assertion, or could we simply weaken it (e.g., > 3)? maybe not necessary since we already check for the segment count above though

Given that we can't reliably determine replay duration here I'd say we drop this assertion entirely. Probably segment count check is sufficient for now.

tustanivsky changed the title ~~Harden integration-test crash uploads in CI~~ chore: Harden integration-test crash uploads in CI Jul 24, 2026

tustanivsky force-pushed the chore/harden-integration-test-uploads branch from 8efa3b0 to d6b9465 Compare July 24, 2026 08:17

tustanivsky added 8 commits July 24, 2026 16:52

Include hidden files so .sentry-native is captured in the artifact

541676f

actions/upload-artifact skips hidden files by default, which dropped the dotted .sentry-native crash DB (daemon log, minidumps) from the failure artifacts.

Add memory limit util

eb6b6e1

Allow manual dispatch of package-plugin-workflow

b5e1bec

Dispatched runs get their own concurrency group (run_id fallback), so multiple validation runs can execute in parallel without canceling the PR run.

Drop exact replay duration assertion in integration tests

bafedea

The server computes replay duration from segment timestamps, which shift with crash handling timing (observed 6 vs the sidecar's nominal 5 seconds), so an exact match is inherently flaky.

Update sentry-native pin to reworded fix branch

278fc97

The stack guarantee fix branch was rebased for the upstream PR (getsentry/sentry-native#1918); the previously pinned commit no longer exists on any branch.

tustanivsky force-pushed the chore/harden-integration-test-uploads branch 2 times, most recently from 49baacf to 278fc97 Compare July 27, 2026 06:22

tustanivsky mentioned this pull request Jul 27, 2026

fix: apply default thread stack guarantee in static builds and the native backend getsentry/sentry-native#1918

Merged

tustanivsky added 2 commits July 27, 2026 09:55

Remove workflow_dispatch trigger used for CI validation

0419125

Revert sentry-native pin to 0.15.4

670682a

The thread stack guarantee fix (getsentry/sentry-native#1918) will be picked up via a separate dependency update PR once it merges upstream.

tustanivsky mentioned this pull request Jul 28, 2026

Increase crash handler thread stack guarantee to 128 KiB on Windows #1500

Merged

tustanivsky added 2 commits July 28, 2026 11:02

Merge branch 'main' into chore/harden-integration-test-uploads

3590381

Remove SENTRY_HANDLER_STACK_SIZE env var

4e80b05

tustanivsky marked this pull request as ready for review July 28, 2026 08:04

sentry-warden Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread .github/workflows/integration-test-linux.yml Outdated

Capture only logs from .sentry-native in failure artifacts

1db445a

Minidumps and envelopes contain process memory of the crashed test app (on Linux even the full environment block), which may hold CI secrets. Restrict the artifact to the daemon logs, which carry the diagnostic value.

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread .github/workflows/integration-test-windows.yml

tustanivsky requested review from JoshuaMoelans, jpnurmi and mujacica July 28, 2026 09:44

JoshuaMoelans approved these changes Jul 28, 2026

View reviewed changes

tustanivsky merged commit 93efaf5 into main Jul 28, 2026
124 of 128 checks passed

tustanivsky deleted the chore/harden-integration-test-uploads branch July 28, 2026 12:30

Navigation Menu

Uh oh!

Uh oh!

chore: Harden integration-test crash uploads in CI - #1496

chore: Harden integration-test crash uploads in CI#1496
tustanivsky merged 14 commits into
mainfrom
chore/harden-integration-test-uploads

tustanivsky commented Jul 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

JoshuaMoelans left a comment

Uh oh!

JoshuaMoelans Jul 28, 2026

Uh oh!

tustanivsky Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

tustanivsky commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Changes

Notes

Related Items

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JoshuaMoelans left a comment

Choose a reason for hiding this comment

Uh oh!

JoshuaMoelans Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

tustanivsky Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tustanivsky commented Jul 24, 2026 •

edited

Loading