Memory leak with Sentry.Profiling 6.6.0 on Linux: TraceEvent call-stack/method index tables accumulate unboundedly under traffic (~0.6 GB/day to OOM)

[therickfactr]

Package

Sentry.Profiling

.NET Flavor

.NET

.NET Version

8.0

OS

Linux

SDK Version

6.6.0 (Sentry.AspNetCore 6.6.0 + Sentry.Profiling 6.6.0)

Self-Hosted Sentry Version

No response (SaaS)

Steps to Reproduce

1. ASP.NET Core service on Linux x64 (Rocky 9, kernel 5.14), 8 GB VM, systemd-hosted, builder.WebHost.UseSentry(...) with options.AddProfilingIntegration().
2. TracesSampleRate: 0.1, ProfilesSampleRate: 0.1.
3. Run under real production traffic (an idle instance does not reproduce — sampled transactions drive it).
4. Watch process RSS over days.

Expected Result

Profiling overhead reaches a bounded steady state.

Actual Result

RSS grows ~0.6 GB/day under steady traffic until kernel OOM-kill at ~6.1 GB anon RSS after ~8 days of uptime. Growth rate is traffic-proportional (~100 MiB/hr at evening peak, less overnight).

dotnet-gcdump on the live process (t+5.5 h after a restart, working set 1581 MiB and climbing) shows 478 MB live heap / 8.67 M objects, and the top retained large types are all Microsoft.Diagnostics.Tracing (TraceEvent) bookkeeping:

   Object Bytes     Count  Type
     35,087,256         1  CallStackInfo[]  [Microsoft.Diagnostics.Tracing.TraceEvent.dll]
     35,087,256         1  System.Collections.Generic.List<Microsoft.Diagnostics.Tracing.Etlx.CallStackIndex>[]
      2,548,664         1  MethodInfo[]  [Microsoft.Diagnostics.Tracing.TraceEvent.dll]
      2,548,536         1  CodeAddressInfo[]  [Microsoft.Diagnostics.Tracing.TraceEvent.dll]
      2,172,576         1  Entry<System.Int64,TraceCodeAddresses+ILMapIndex>[]
      1,810,368         1  Entry<System.UInt64,CodeAddressIndex>[]
      1,048,600         1  MethodLoadUnloadVerboseTraceData[]  [TraceEvent]
        876,536         1  TraceMethod[]  [TraceEvent]
        849,528         1  ILToNativeMap[]  [TraceEvent]
        424,776         1  CodeAddressIndex[]  [TraceEvent]
        161,984         1  Entry<System.Int32,EventPipe.StackCache+StackMarker>[]

(These are the >100 KB single arrays; the remaining growth is millions of small entries in the same structures.)

A/B on the identical build: set ProfilesSampleRate: 0 and restart → t+10 h working set 477 MiB and falling; gcdump shows 31 MB live / 470 k objects with zero TraceEvent types. Leak fully cured by disabling profiling alone (tracing, logs, error reporting all left enabled).

Analysis

The continuous in-process EventPipe/TraceEvent session backing transaction profiling accumulates call-stack, code-address, and IL-map index tables (CallStackInfo, TraceCodeAddresses maps, ILToNativeMap, StackCache) for the life of the session, proportional to the diversity/volume of sampled stacks. They are never trimmed.

Related history — all distinct mechanisms, and all already present in 6.6.0:

• #3407 was closed by #3491 ("decrease runtime circular buffer for diagnostics client") — native EventPipe buffer sizing, not managed index growth.
• Sentry profiler seems to be leaking memory #5113 was closed by #5133 (shipped 6.4.0) — ConcurrentBag/ConcurrentQueue stranded per-thread nodes, not TraceEvent index tables.
• SamplingTransactionProfilerFactory.Dispose disposes the Task, not the SampleProfilerSession (resource leak) #5418 (open) — SampleProfilerSession not disposed on factory Dispose; a shutdown-path resource leak, distinct from this steady-state runtime growth (possibly compounding).

As far as I can tell this steady-state TraceEvent index accumulation has no existing open or closed issue.

Happy to attach both full gcdump type histograms (with-profiling vs without) if useful.

[linear-code]

NET-783

[jamescrosswell]

Thanks @therickfactr ❤️

Claude reproduced this for me in a Linux container with a minimal harness (ASP.NET-free: a bare EventPipe + TraceLog session plus synthetic stack-shape churn), and confirms both the symptom and your analysis.

Root cause

Stems from TraceCallStacks.InternCallStackIndex in TraceEvent's TraceLog.cs. The call-stack table is an append-only trie: every distinct stack shape interns a new CallStackInfo, and each one also appends to the parallel callees list.

TraceLog's real-time mode does have trimming, but only for three eventsToStacks, eventsToCodeAddresses and cswitchBlockingEventsToStacks — and on the EventPipe path those are already cleared after every event. The interning tables
(callStacks, callees, codeAddresses) have no eviction path at all.

This is tracked upstream:

• [TraceEvent] Microsoft.Diagnostics.Tracing.Etlx.TraceCallStacks+CallStackInfo[] - High memory usage microsoft/perfview#1199

But the upstream was closed as not planned... so we need to either get a PR accepted upstream to fix or fork from upstream (yuck).

Two surprises

Growth tracks distinct stack shapes, not sample volume. With a single fixed stack shape, 11x more samples grew the table only 24%. This is why an idle instance doesn't reproduce and your traffic-proportional rate does — real traffic keeps minting new stack
shapes. It also means the leak rate depends on application shape (async state machines, generics, DI-heavy call paths) more than on request count.

It is independent of ProfilesSampleRate above 0. The accumulation happens inside TraceLog's event processing whether or not a profiler is attached to consume samples — the session is started once by SamplingTransactionProfilerFactory and runs continuously. ProfilesSampleRate: 0 cures it only because no factory (and therefore no session) is constructed at all. That matches your A/B exactly, and it means lowering the sample rate is not a mitigation — only disabling profiling outright is.

Recycling the session does NOT work

The obvious remedy is to tear down and restart the EventPipe session once the call-stack table crosses a budget. It works on the managed side and fails on the whole-process side. Three 600s runs, identical workload:

run	call stacks (end)	managed heap	RSS start→end
baseline	42,321, still climbing	3.8 → 6.6 MiB	70.6 → 79.3 MiB (+8.7)
recycling (8 recycles)	bounded ≤10.8k	~3.6 MiB flat	73.7 → 98.5 MiB (+24.8)
recycling, no session overlap (6)	bounded ≤10.3k	~3.8 MiB flat	71.2 → 90.8 MiB (+19.6)

The interning tables stay bounded and managed heap goes flat — but each recycle costs ~3.1-3.3 MiB of RSS that is never returned so RSS grew ~3x faster than the leak being fixed (the cure is worse than the disease). Disposing the old session before starting the replacement changes nothing (3.3 vs 3.1 MiB/recycle), so it isn't the two 16 MB EventPipe buffers briefly coexisting — it's a per-session cost. Scaled to the reporter's ~222 stacks/sec, even a 1M-stack budget would leave ~120 MiB permanently resident and still leak ~50 MiB/day.

Recommended direction / fix

A new EventPipe session is not needed - only a reset of the managed interning tables. The session is precisely what costs the
3 MiB.

So instead, add a reset to TraceCallStacks in the getsentry/perfview fork (clearing callStacks, callees, threads) and call it from SamplingTransactionProfilerFactory between profiles.

Nothing outside those tables retains a CallStackIndex across events. PastEventInfo is a fixed 2048-entry ring, the
eventsTo*Stacks arrays are cleared per-event on the EventPipe path, and SampleProfileBuilder's index maps are constructed fresh per profile. TraceCallStacks.SetSize is existing precedent for shrinking the array, though it doesn't touch callees/threads,
which a correct reset would need to.

Not yet implemented or validated — this is just what came out of the initial analysis.

Repro environment

Linux container (mcr.microsoft.com/dotnet/runtime:9.0, .NET 9.0.8, kernel 6.10) rather than .NET 8 / Rocky 9, so it's the same mechanism rather than a byte-for-byte match of your environment. RSS measured as VmRSS from /proc/self/status.

[jamescrosswell]

Planned sequencing for the fix

1. Fast-forward the getsentry/perfview mirror - getsentry/perfview main is 144 commits behind microsoft/perfview main
   • @vaind do you have push rights for getsentry/perfview - are you able to do this?
2. Bump the submodule in sentry-dotnet to the new head - check nothing breaks
3. Write a fix
4. PR for the fix upstream (in the microsoft/perfview repo) - If Microsoft are willing to accept the PR, we can avoid having to fork from upstream (which I'd really really like to do).

Notes on the bump (step 1)

• Nothing in the 219 commits touches the call-stack interning tables, so the bump won't fix this on its own — the work in step 3 is still needed.
• Two realtime-path fixes in range may help independently: Fixes that repeatedly adding unhandled event handler to TraceLog to real time session conversion microsoft/perfview#2159 (repeatedly adding an unhandled event handler during realtime session conversion) and Don't do FlushRealTimeEvents if already in FlushRealTimeEvents microsoft/perfview#2166 (reentrant FlushRealTimeEvents). Both are in code the profiler exercises continuously.
• Main compat risk is NetTrace format version handling — Increment Versions and Disable Nettrace V6 Support microsoft/perfview#2178 disables V6 support and Symbolication for AOT-compiled stack traces on iOS #2180 changes version handling generally. Sentry.Profiling consumes the live nettrace stream, whose format version tracks the customer's runtime, so the draft PR needs to show profiling still starts across the .NET versions we support.
• FastSerialization has breaking changes in range (Breaking Changes for FastSerialization microsoft/perfview#2121), but we ship it purely as a profiling implementation detail rather than as public surface, so I think that's acceptable.

[therickfactr]

Following up on the upstream path: I've opened microsoft/perfview#2451 with the production evidence and the mechanism from @jamescrosswell's analysis above, framed as a question about whether an unbounded interning table is considered in-scope for the real-time/EventPipe path.

Rather than reopening microsoft/perfview#1199 — which was closed in 2022 because the original reporter stopped seeing it, not because it was resolved or ruled out — it's a fresh issue linking back to it, so it lands in the triage queue instead of a dormant thread.

It explicitly floats the bounded-reset direction (clearing callStacks/callees/threads between profiles without tearing down the session, with TraceCallStacks.SetSize as precedent), so if Microsoft engage, step 4 of your plan may not need a fork. Happy to supply dumps or a repro harness if that helps.