Skip to content

fix: upgrade @sentry/* to ^10.70.0 to fix span retention memory leaks - #1572

Merged
brendan-kellam merged 2 commits into
mainfrom
brendan/bump-sentry
Aug 12, 2026
Merged

fix: upgrade @sentry/* to ^10.70.0 to fix span retention memory leaks#1572
brendan-kellam merged 2 commits into
mainfrom
brendan/bump-sentry

Conversation

@brendan-kellam

@brendan-kellam brendan-kellam commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Problem

The web process leaks ~570 MiB/h of old-space JS heap (728 → 3589 MiB over 5h yesterday), until GC stalls trip the 1s liveness probe and the kubelet kills the container — the ~5–9h restart cadence on app.sourcebot.dev. The heap-limit raise in #1569 removed the old pinned-at-ceiling thrash but, as flagged there, did not stop the growth.

Root-causing this with in-prod V8 sampling heap profiles (inspector on the live next-server) showed the retained memory is request-scoped tracing machinery — Sentry/OTel startSpan and context clone allocations, Prisma client span wrappers, Next req/res objects — with no app code among the top allocators and every guessable buffer (exporter span buckets, incremental cache, breadcrumbs, listeners) measured and bounded. Entire request graphs survive response completion.

Fix

Bump @sentry/nextjs, @sentry/node, and @sentry/profiling-node from ^10.40.0 to ^10.70.0. Upstream shipped fixes matching the measured signature exactly, all after 10.40.0:

Release Fix
10.56.0 #21242WeakRef for Span↔Scope circular references + sent-span cache converted to an LRUMap capped at 10k (closes #18339, "memory leak with high tracesSampleRate")
10.49.0 #20328WeakRef for OTel context stored on scope (contexts were our second-biggest retained cohort)
10.43.0 #19328vercel-ai tool-call span map leak; the VercelAI integration is active in this process

Direct evidence the running build predates the fix: probing the live exporter found _sentSpans at 29,333 entries — three times the 10k cap #21242 introduced.

tracesSampleRate is 0.1, but homepage traces run ~900 spans, so absolute span volume reaches the same failure mode as the "high tracesSampleRate" issue.

Verification

  • All @sentry/* packages resolve to 10.70.0; zero residual 10.40.x in yarn.lock
  • yarn workspace @sourcebot/backend build — exit 0
  • yarn workspace @sourcebot/web test run — 1239/1239 pass (106 files)
  • yarn workspace @sourcebot/web build — compiles clean
  • Post-deploy: heap-used slope on the "Sourcebot app latency & stalls" dashboard should flatten (each series sawtooths with GC instead of climbing monotonically), and container uptime should stop cliffing every ~5–9h

If the slope doesn't flatten

The fallback is the feature-gate bisect from the investigation: env-gate profileSessionSampleRate (currently hardcoded 1.0 with profileLifecycle: 'trace'), then LocalVariablesAsync — both have their own upstream leak history and both are fed by the still-unfixed Sentry egress ETIMEDOUTs (84–134/h), which remain worth fixing independently.

🤖 Generated with Claude Code


Note

Cursor Bugbot is generating a summary for commit a38db5e. Configure here.

Summary by CodeRabbit

  • Bug Fixes
    • Resolved memory leaks that could occur when request data was retained indefinitely by monitoring spans.
    • Updated error-monitoring components to improve memory management and stability.

The web process leaks ~570 MiB/h of old-space heap. In-prod sampling heap
profiles show the retained memory is request-scoped tracing machinery — spans,
OTel contexts, and the request/response objects they reference — with no app
code among the top allocators.

Upstream shipped fixes matching that signature exactly, all after our pinned
10.40.0:

- 10.56.0: WeakRef for Span<->Scope circular references, and the sent-span
  cache moved to an LRUMap capped at 10k entries (#21242, closes #18339
  "memory leak with high tracesSampleRate")
- 10.49.0: WeakRef for OTel context stored on scope (#20328)
- 10.43.0: vercel-ai tool-call span map leak (#19328) — the VercelAI
  integration is active in this process

The running build demonstrably predates the fix: the live exporter's
_sentSpans map measured 29,333 entries, three times the post-fix cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1016463f-0b6c-4844-aa7c-313c1fefbd54

📥 Commits

Reviewing files that changed from the base of the PR and between 84a75e4 and 89bd559.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (3)
  • CHANGELOG.md
  • packages/backend/package.json
  • packages/web/package.json

Walkthrough

The backend and web packages upgrade Sentry dependencies to ^10.70.0. The changelog records the upgrade and its memory-leak correction.

Changes

Sentry dependency updates

Layer / File(s) Summary
Update Sentry dependency ranges
packages/backend/package.json, packages/web/package.json, CHANGELOG.md
The backend and web packages update Sentry dependencies from ^10.40.0 to ^10.70.0. The changelog documents the memory-leak correction.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

Suggested reviewers: msukkari

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the Sentry package upgrade and its purpose of fixing span retention memory leaks.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch brendan/bump-sentry

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@brendan-kellam
brendan-kellam merged commit a4a0f0e into main Aug 12, 2026
12 of 14 checks passed
@brendan-kellam
brendan-kellam deleted the brendan/bump-sentry branch August 12, 2026 19:10
@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

License Audit

⚠️ Status: PASS

Metric Count
Total packages 2166
Resolved (non-standard) 26
Unresolved 0
Strong copyleft 0
Weak copyleft 27

Weak Copyleft Packages (informational)

Package Version License
@img/sharp-libvips-darwin-arm64 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-darwin-x64 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linux-arm 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linux-arm64 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linux-ppc64 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linux-riscv64 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linux-s390x 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linux-x64 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linuxmusl-arm64 1.3.2 LGPL-3.0-or-later
@img/sharp-libvips-linuxmusl-x64 1.3.2 LGPL-3.0-or-later
@img/sharp-wasm32 0.35.3 Apache-2.0 AND LGPL-3.0-or-later AND MIT
@img/sharp-win32-arm64 0.35.3 Apache-2.0 AND LGPL-3.0-or-later
@img/sharp-win32-ia32 0.35.3 Apache-2.0 AND LGPL-3.0-or-later
@img/sharp-win32-x64 0.35.3 Apache-2.0 AND LGPL-3.0-or-later
axe-core 4.10.3 MPL-2.0
lightningcss 1.32.0 MPL-2.0
lightningcss-android-arm64 1.32.0 MPL-2.0
lightningcss-darwin-arm64 1.32.0 MPL-2.0
lightningcss-darwin-x64 1.32.0 MPL-2.0
lightningcss-freebsd-x64 1.32.0 MPL-2.0
lightningcss-linux-arm-gnueabihf 1.32.0 MPL-2.0
lightningcss-linux-arm64-gnu 1.32.0 MPL-2.0
lightningcss-linux-arm64-musl 1.32.0 MPL-2.0
lightningcss-linux-x64-gnu 1.32.0 MPL-2.0
lightningcss-linux-x64-musl 1.32.0 MPL-2.0
lightningcss-win32-arm64-msvc 1.32.0 MPL-2.0
lightningcss-win32-x64-msvc 1.32.0 MPL-2.0
Resolved Packages (26)
Package Version Original Resolved Source
@sentry/cli 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-darwin 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-darwin 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-arm 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-arm 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-arm64 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-arm64 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-i686 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-i686 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-x64 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-linux-x64 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-win32-arm64 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-win32-arm64 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-win32-i686 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-win32-i686 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-win32-x64 2.58.5 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
@sentry/cli-win32-x64 2.58.6 FSL-1.1-MIT FSL-1.1-MIT npm registry metadata + published tarball LICENSE (Functional Source License 1.1, MIT Future License)
codemirror-lang-elixir 4.0.0 UNKNOWN Apache-2.0 GitHub repo (livebook-dev/codemirror-lang-elixir LICENSE) + published tarball LICENSE
khroma 2.1.0 UNKNOWN MIT GitHub repo (fabiospampinato/khroma license) + published tarball LICENSE
lezer-elixir 1.1.2 UNKNOWN Apache-2.0 GitHub repo (livebook-dev/lezer-elixir LICENSE) + published tarball LICENSE
map-stream 0.1.0 UNKNOWN MIT GitHub repo (dominictarr/map-stream LICENCE) + published tarball LICENSE
memorystream 0.3.1 UNKNOWN MIT extracted from object (package.json licenses[0].type) + GitHub repo LICENSE
pause-stream 0.0.11 ["MIT","Apache2"] (MIT OR Apache-2.0) extracted from array ["MIT","Apache2"] + GitHub repo LICENSE ("Dual Licensed MIT and Apache 2")
posthog-js 1.369.0 SEE LICENSE IN LICENSE Apache-2.0 GitHub repo (PostHog/posthog-js LICENSE, Apache-2.0 text) + published tarball LICENSE
valid-url 1.0.9 UNKNOWN MIT GitHub repo (ogt/valid-url LICENSE) + published tarball LICENSE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory leak with high tracesSampleRate

1 participant