Logs: optimize performance

[jpnurmi]

In sentry-unreal, UE_LOG can be wired into sentry-native logs. That means the native SDK may receive not only explicit game/application logs, but also high-volume Unreal Engine logs from many engine and game threads.

This makes log ingestion a hot path: in busy Unreal-like workloads, every log reads the global scope to apply attributes and then passes through the native log batcher. Under bursty multi-threaded logging, that exposes two main bottlenecks:

• logger/producer threads: global scope lock for applying attributes (#ff0000)
• batcher/consumer thread: log item serialization (#0000ff) and cleanup (#00ff00)

[linear-code]

NATIVE-200

[sentry-pierre]

Confirmed both bottlenecks in the current implementation:

• construct_log takes the global options lock (SENTRY_WITH_OPTIONS) and the global scope lock (SENTRY_WITH_SCOPE in sentry__apply_attributes) on every producer-thread call.
• sentry__batcher_flush serializes and cleans up log items on the single consumer thread.

The flame graph analysis lines up with the code. A maintainer can take it from here.

[jpnurmi]

Potential solutions

• Batcher: move/post-pone the cost of serialization to the tranport thread. Processing of batches becomes blazing fast and no longer prevents them from being queued. However, it can create a congestion in the transport thread, so it would be good to introduce bgworker task priorities to let higher priority envelopes bypass the queue.

  

• Scope lock: replace the global scope mutex with a read-write lock to allow parallel reads

  

[jpnurmi]

• Batcher: move/post-pone the cost of serialization to the tranport thread. Processing of batches becomes blazing fast and no longer prevents them from being queued. However, it can create a congestion in the transport thread, so it would be good to introduce bgworker task priorities to let higher priority envelopes bypass the queue.

While this would effectively make it nearly impossible to lose logs (as long as an immediate burst doesn't exceed the hard-wired buffers), it would just move the problem elsewhere by piling a queue in the transport thread. Unlike the global scope bottleneck in the logger thread, this work is already done off the user threads and does not directly impact the performance of an app/game.

I think increasing the UE_LOG buffers (#1867) and improving the performance of sentry_log is a good start. Let's see where we get with those two improvements i.e. whether we still see logs being discarded and whether optimizing the batcher side is necessary at all.