feat(llmobs): flush writers on SIGTERM with graceful termination period#17174
feat(llmobs): flush writers on SIGTERM with graceful termination period#17174ZStriker19 wants to merge 4 commits intomainfrom
Conversation
Registers LLMObs.disable with register_on_exit_signal so SIGTERM (e.g. Kubernetes pod termination) triggers a graceful flush of buffered spans and eval metrics before exit. Also joins writer threads with a 5-second default timeout, matching the tracer's SHUTDOWN_TIMEOUT convention. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…IGTERM shutdown Adds _on_exit_signal classmethod that checks config._llmobs_graceful_termination_period (DD_LLMOBS_GRACEFUL_TERMINATION_PERIOD, default 0). If zero, disables immediately on SIGTERM. If non-zero, starts a threading.Timer so the app can continue writing spans during a Kubernetes graceful termination window before flushing and stopping. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…MEOUT hardcoded The join timeout is an implementation detail (thread cleanup after a synchronous flush), not a meaningful user-facing knob. Keep it as a hardcoded 5s constant. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Codeowners resolved as |
|
✨ Fix all issues with BitsAI or with Cursor
|
Performance SLOsComparing candidate zach.groves/llmobs-graceful-sigterm-shutdown-main (0eb5b86) with baseline main (e2ab999) 📈 Performance Regressions (2 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 103.162µs (SLO: <130.000µs 📉 -20.6%) vs baseline: +1.4% Memory: ✅ 43.924MB (SLO: <46.000MB -4.5%) vs baseline: +5.7% ✅ add_inplace_aspectTime: ✅ 100.702µs (SLO: <130.000µs 📉 -22.5%) vs baseline: -2.8% Memory: ✅ 43.880MB (SLO: <46.000MB -4.6%) vs baseline: +5.5% ✅ add_inplace_noaspectTime: ✅ 28.409µs (SLO: <40.000µs 📉 -29.0%) vs baseline: +1.3% Memory: ✅ 43.955MB (SLO: <46.000MB -4.4%) vs baseline: +5.7% ✅ add_noaspectTime: ✅ 49.232µs (SLO: <70.000µs 📉 -29.7%) vs baseline: +0.8% Memory: ✅ 43.919MB (SLO: <46.000MB -4.5%) vs baseline: +5.5% ✅ bytearray_aspectTime: ✅ 248.651µs (SLO: <400.000µs 📉 -37.8%) vs baseline: -4.4% Memory: ✅ 44.000MB (SLO: <46.000MB -4.3%) vs baseline: +5.6% ✅ bytearray_extend_aspectTime: ✅ 650.370µs (SLO: <800.000µs 📉 -18.7%) vs baseline: -3.1% Memory: ✅ 43.935MB (SLO: <46.000MB -4.5%) vs baseline: +5.6% ✅ bytearray_extend_noaspectTime: ✅ 266.312µs (SLO: <400.000µs 📉 -33.4%) vs baseline: -1.0% Memory: ✅ 43.918MB (SLO: <46.000MB -4.5%) vs baseline: +5.7% ✅ bytearray_noaspectTime: ✅ 137.977µs (SLO: <300.000µs 📉 -54.0%) vs baseline: -2.4% Memory: ✅ 43.893MB (SLO: <46.000MB -4.6%) vs baseline: +5.4% ✅ bytes_aspectTime: ✅ 216.688µs (SLO: <300.000µs 📉 -27.8%) vs baseline: -2.6% Memory: ✅ 43.885MB (SLO: <46.000MB -4.6%) vs baseline: +5.4% ✅ bytes_noaspectTime: ✅ 132.783µs (SLO: <200.000µs 📉 -33.6%) vs baseline: -1.4% Memory: ✅ 43.906MB (SLO: <46.000MB -4.6%) vs baseline: +5.6% ✅ bytesio_aspectTime: ✅ 3.758ms (SLO: <5.000ms 📉 -24.8%) vs baseline: -0.3% Memory: ✅ 44.059MB (SLO: <46.000MB -4.2%) vs baseline: +6.2% ✅ bytesio_noaspectTime: ✅ 318.022µs (SLO: <420.000µs 📉 -24.3%) vs baseline: -0.2% Memory: ✅ 43.890MB (SLO: <46.000MB -4.6%) vs baseline: +5.7% ✅ capitalize_aspectTime: ✅ 89.141µs (SLO: <300.000µs 📉 -70.3%) vs baseline: -0.8% Memory: ✅ 43.970MB (SLO: <46.000MB -4.4%) vs baseline: +5.9% ✅ capitalize_noaspectTime: ✅ 246.506µs (SLO: <300.000µs 📉 -17.8%) vs baseline: -2.5% Memory: ✅ 43.817MB (SLO: <46.000MB -4.7%) vs baseline: +5.6% ✅ casefold_aspectTime: ✅ 88.084µs (SLO: <500.000µs 📉 -82.4%) vs baseline: -0.9% Memory: ✅ 43.943MB (SLO: <46.000MB -4.5%) vs baseline: +5.7% ✅ casefold_noaspectTime: ✅ 300.993µs (SLO: <500.000µs 📉 -39.8%) vs baseline: -1.4% Memory: ✅ 43.882MB (SLO: <46.000MB -4.6%) vs baseline: +5.5% ✅ decode_aspectTime: ✅ 87.226µs (SLO: <100.000µs 📉 -12.8%) vs baseline: +1.1% Memory: ✅ 43.900MB (SLO: <46.000MB -4.6%) vs baseline: +5.8% ✅ decode_noaspectTime: ✅ 153.949µs (SLO: <210.000µs 📉 -26.7%) vs baseline: -1.6% Memory: ✅ 43.785MB (SLO: <46.000MB -4.8%) vs baseline: +5.3% ✅ encode_aspectTime: ✅ 84.007µs (SLO: <200.000µs 📉 -58.0%) vs baseline: -0.3% Memory: ✅ 43.896MB (SLO: <46.000MB -4.6%) vs baseline: +5.7% ✅ encode_noaspectTime: ✅ 141.576µs (SLO: <200.000µs 📉 -29.2%) vs baseline: -1.4% Memory: ✅ 44.019MB (SLO: <46.000MB -4.3%) vs baseline: +5.9% ✅ format_aspectTime: ✅ 14.613ms (SLO: <19.200ms 📉 -23.9%) vs baseline: +0.2% Memory: ✅ 44.076MB (SLO: <46.000MB -4.2%) vs baseline: +5.9% ✅ format_map_aspectTime: ✅ 16.384ms (SLO: <21.500ms 📉 -23.8%) vs baseline: ~same Memory: ✅ 44.040MB (SLO: <46.000MB -4.3%) vs baseline: +5.5% ✅ format_map_noaspectTime: ✅ 373.383µs (SLO: <500.000µs 📉 -25.3%) vs baseline: -1.6% Memory: ✅ 43.859MB (SLO: <46.000MB -4.7%) vs baseline: +5.6% ✅ format_noaspectTime: ✅ 305.753µs (SLO: <500.000µs 📉 -38.8%) vs baseline: -2.4% Memory: ✅ 43.975MB (SLO: <46.000MB -4.4%) vs baseline: +5.8% ✅ index_aspectTime: ✅ 136.990µs (SLO: <300.000µs 📉 -54.3%) vs baseline: 📈 +11.2% Memory: ✅ 43.844MB (SLO: <46.000MB -4.7%) vs baseline: +5.6% ✅ index_noaspectTime: ✅ 40.302µs (SLO: <300.000µs 📉 -86.6%) vs baseline: -0.6% Memory: ✅ 43.920MB (SLO: <46.000MB -4.5%) vs baseline: +5.8% ✅ join_aspectTime: ✅ 210.807µs (SLO: <300.000µs 📉 -29.7%) vs baseline: +0.3% Memory: ✅ 43.979MB (SLO: <46.000MB -4.4%) vs baseline: +5.8% ✅ join_noaspectTime: ✅ 143.999µs (SLO: <300.000µs 📉 -52.0%) vs baseline: -1.9% Memory: ✅ 43.959MB (SLO: <46.000MB -4.4%) vs baseline: +5.8% ✅ ljust_aspectTime: ✅ 573.929µs (SLO: <700.000µs 📉 -18.0%) vs baseline: 📈 +14.5% Memory: ✅ 44.000MB (SLO: <46.000MB -4.3%) vs baseline: +6.0% ✅ ljust_noaspectTime: ✅ 254.453µs (SLO: <300.000µs 📉 -15.2%) vs baseline: -0.9% Memory: ✅ 43.885MB (SLO: <46.000MB -4.6%) vs baseline: +5.7% ✅ lower_aspectTime: ✅ 295.960µs (SLO: <500.000µs 📉 -40.8%) vs baseline: -0.8% Memory: ✅ 43.998MB (SLO: <46.000MB -4.4%) vs baseline: +5.7% ✅ lower_noaspectTime: ✅ 233.175µs (SLO: <300.000µs 📉 -22.3%) vs baseline: -0.5% Memory: ✅ 43.873MB (SLO: <46.000MB -4.6%) vs baseline: +5.4% ✅ lstrip_aspectTime: ✅ 0.278ms (SLO: <3.000ms 📉 -90.7%) vs baseline: +1.2% Memory: ✅ 43.742MB (SLO: <46.000MB -4.9%) vs baseline: +5.1% ✅ lstrip_noaspectTime: ✅ 0.178ms (SLO: <3.000ms 📉 -94.1%) vs baseline: +0.4% Memory: ✅ 43.900MB (SLO: <46.000MB -4.6%) vs baseline: +5.6% ✅ modulo_aspectTime: ✅ 14.330ms (SLO: <18.750ms 📉 -23.6%) vs baseline: +0.4% Memory: ✅ 44.116MB (SLO: <46.000MB -4.1%) vs baseline: +5.7% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 14.818ms (SLO: <19.350ms 📉 -23.4%) vs baseline: ~same Memory: ✅ 44.061MB (SLO: <46.000MB -4.2%) vs baseline: +5.8% ✅ modulo_aspect_for_bytesTime: ✅ 14.330ms (SLO: <18.900ms 📉 -24.2%) vs baseline: -1.1% Memory: ✅ 44.019MB (SLO: <46.000MB -4.3%) vs baseline: +5.5% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.619ms (SLO: <19.150ms 📉 -23.7%) vs baseline: +0.3% Memory: ✅ 44.015MB (SLO: <46.000MB -4.3%) vs baseline: +5.5% ✅ modulo_noaspectTime: ✅ 0.354ms (SLO: <3.000ms 📉 -88.2%) vs baseline: -3.9% Memory: ✅ 43.864MB (SLO: <46.000MB -4.6%) vs baseline: +5.7% ✅ replace_aspectTime: ✅ 18.385ms (SLO: <24.000ms 📉 -23.4%) vs baseline: -1.1% Memory: ✅ 44.025MB (SLO: <46.000MB -4.3%) vs baseline: +5.8% ✅ replace_noaspectTime: ✅ 278.207µs (SLO: <400.000µs 📉 -30.4%) vs baseline: -1.7% Memory: ✅ 43.982MB (SLO: <46.000MB -4.4%) vs baseline: +5.8% ✅ repr_aspectTime: ✅ 320.272µs (SLO: <420.000µs 📉 -23.7%) vs baseline: -0.6% Memory: ✅ 43.825MB (SLO: <46.000MB -4.7%) vs baseline: +5.3% ✅ repr_noaspectTime: ✅ 46.697µs (SLO: <90.000µs 📉 -48.1%) vs baseline: -0.2% Memory: ✅ 43.906MB (SLO: <46.000MB -4.6%) vs baseline: +5.5% ✅ rstrip_aspectTime: ✅ 380.661µs (SLO: <500.000µs 📉 -23.9%) vs baseline: -1.9% Memory: ✅ 43.938MB (SLO: <46.000MB -4.5%) vs baseline: +5.7% ✅ rstrip_noaspectTime: ✅ 183.423µs (SLO: <300.000µs 📉 -38.9%) vs baseline: +0.2% Memory: ✅ 44.060MB (SLO: <46.000MB -4.2%) vs baseline: +6.0% ✅ slice_aspectTime: ✅ 183.600µs (SLO: <300.000µs 📉 -38.8%) vs baseline: +0.5% Memory: ✅ 43.842MB (SLO: <46.000MB -4.7%) vs baseline: +5.6% ✅ slice_noaspectTime: ✅ 55.112µs (SLO: <90.000µs 📉 -38.8%) vs baseline: +1.8% Memory: ✅ 43.903MB (SLO: <46.000MB -4.6%) vs baseline: +5.4% ✅ stringio_aspectTime: ✅ 3.796ms (SLO: <5.000ms 📉 -24.1%) vs baseline: -0.9% Memory: ✅ 43.787MB (SLO: <46.000MB -4.8%) vs baseline: +5.1% ✅ stringio_noaspectTime: ✅ 375.479µs (SLO: <500.000µs 📉 -24.9%) vs baseline: +8.2% Memory: ✅ 44.010MB (SLO: <46.000MB -4.3%) vs baseline: +5.7% ✅ strip_aspectTime: ✅ 274.910µs (SLO: <350.000µs 📉 -21.5%) vs baseline: +0.5% Memory: ✅ 43.958MB (SLO: <46.000MB -4.4%) vs baseline: +5.6% ✅ strip_noaspectTime: ✅ 177.971µs (SLO: <240.000µs 📉 -25.8%) vs baseline: +0.9% Memory: ✅ 43.942MB (SLO: <46.000MB -4.5%) vs baseline: +5.9% ✅ swapcase_aspectTime: ✅ 331.495µs (SLO: <500.000µs 📉 -33.7%) vs baseline: -2.4% Memory: ✅ 43.981MB (SLO: <46.000MB -4.4%) vs baseline: +5.9% ✅ swapcase_noaspectTime: ✅ 265.214µs (SLO: <400.000µs 📉 -33.7%) vs baseline: -2.1% Memory: ✅ 43.907MB (SLO: <46.000MB -4.5%) vs baseline: +5.7% ✅ title_aspectTime: ✅ 320.341µs (SLO: <500.000µs 📉 -35.9%) vs baseline: ~same Memory: ✅ 43.840MB (SLO: <46.000MB -4.7%) vs baseline: +5.6% ✅ title_noaspectTime: ✅ 253.621µs (SLO: <400.000µs 📉 -36.6%) vs baseline: -2.5% Memory: ✅ 43.909MB (SLO: <46.000MB -4.5%) vs baseline: +5.7% ✅ translate_aspectTime: ✅ 492.405µs (SLO: <700.000µs 📉 -29.7%) vs baseline: -1.1% Memory: ✅ 43.981MB (SLO: <46.000MB -4.4%) vs baseline: +5.7% ✅ translate_noaspectTime: ✅ 420.755µs (SLO: <500.000µs 📉 -15.8%) vs baseline: -2.6% Memory: ✅ 43.781MB (SLO: <46.000MB -4.8%) vs baseline: +5.2% ✅ upper_aspectTime: ✅ 294.396µs (SLO: <500.000µs 📉 -41.1%) vs baseline: -2.3% Memory: ✅ 43.808MB (SLO: <46.000MB -4.8%) vs baseline: +5.5% ✅ upper_noaspectTime: ✅ 231.406µs (SLO: <400.000µs 📉 -42.1%) vs baseline: -2.5% Memory: ✅ 44.039MB (SLO: <46.000MB -4.3%) vs baseline: +6.0% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 504.327µs (SLO: <700.000µs 📉 -28.0%) vs baseline: 📈 +18.6% Memory: ✅ 43.533MB (SLO: <46.000MB -5.4%) vs baseline: +4.7% ✅ ospathbasename_noaspectTime: ✅ 431.598µs (SLO: <700.000µs 📉 -38.3%) vs baseline: -0.9% Memory: ✅ 44.001MB (SLO: <46.000MB -4.3%) vs baseline: +5.8% ✅ ospathjoin_aspectTime: ✅ 627.973µs (SLO: <700.000µs 📉 -10.3%) vs baseline: -0.8% Memory: ✅ 43.950MB (SLO: <46.000MB -4.5%) vs baseline: +5.5% ✅ ospathjoin_noaspectTime: ✅ 635.850µs (SLO: <700.000µs -9.2%) vs baseline: -0.9% Memory: ✅ 43.995MB (SLO: <46.000MB -4.4%) vs baseline: +5.7% ✅ ospathnormcase_aspectTime: ✅ 347.852µs (SLO: <700.000µs 📉 -50.3%) vs baseline: -2.0% Memory: ✅ 43.612MB (SLO: <46.000MB -5.2%) vs baseline: +4.8% ✅ ospathnormcase_noaspectTime: ✅ 358.806µs (SLO: <700.000µs 📉 -48.7%) vs baseline: -0.8% Memory: ✅ 43.884MB (SLO: <46.000MB -4.6%) vs baseline: +5.5% ✅ ospathsplit_aspectTime: ✅ 482.583µs (SLO: <700.000µs 📉 -31.1%) vs baseline: -1.1% Memory: ✅ 44.034MB (SLO: <46.000MB -4.3%) vs baseline: +5.7% ✅ ospathsplit_noaspectTime: ✅ 493.552µs (SLO: <700.000µs 📉 -29.5%) vs baseline: -1.2% Memory: ✅ 43.891MB (SLO: <46.000MB -4.6%) vs baseline: +5.4% ✅ ospathsplitdrive_aspectTime: ✅ 372.484µs (SLO: <700.000µs 📉 -46.8%) vs baseline: -0.9% Memory: ✅ 43.980MB (SLO: <46.000MB -4.4%) vs baseline: +5.7% ✅ ospathsplitdrive_noaspectTime: ✅ 72.266µs (SLO: <700.000µs 📉 -89.7%) vs baseline: -0.8% Memory: ✅ 43.639MB (SLO: <46.000MB -5.1%) vs baseline: +4.9% ✅ ospathsplitext_aspectTime: ✅ 455.160µs (SLO: <700.000µs 📉 -35.0%) vs baseline: -0.6% Memory: ✅ 43.699MB (SLO: <46.000MB -5.0%) vs baseline: +5.2% ✅ ospathsplitext_noaspectTime: ✅ 466.908µs (SLO: <700.000µs 📉 -33.3%) vs baseline: ~same Memory: ✅ 43.884MB (SLO: <46.000MB -4.6%) vs baseline: +5.3% 🟡 Near SLO Breach (3 suites)🟡 djangosimple - 30/30✅ appsecTime: ✅ 19.693ms (SLO: <22.300ms 📉 -11.7%) vs baseline: -0.3% Memory: ✅ 69.304MB (SLO: <73.500MB -5.7%) vs baseline: +5.6% ✅ exception-replay-enabledTime: ✅ 1.324ms (SLO: <1.450ms -8.7%) vs baseline: -0.1% Memory: ✅ 67.750MB (SLO: <71.500MB -5.2%) vs baseline: +5.4% ✅ iastTime: ✅ 19.800ms (SLO: <22.250ms 📉 -11.0%) vs baseline: +0.5% Memory: ✅ 69.413MB (SLO: <75.000MB -7.4%) vs baseline: +5.8% ✅ profilerTime: ✅ 15.123ms (SLO: <16.550ms -8.6%) vs baseline: -0.5% Memory: ✅ 60.423MB (SLO: <61.000MB 🟡 -0.9%) vs baseline: +5.5% ✅ resource-renamingTime: ✅ 19.645ms (SLO: <21.750ms -9.7%) vs baseline: ~same Memory: ✅ 69.481MB (SLO: <73.500MB -5.5%) vs baseline: +5.9% ✅ span-code-originTime: ✅ 20.167ms (SLO: <28.200ms 📉 -28.5%) vs baseline: +1.0% Memory: ✅ 69.404MB (SLO: <75.000MB -7.5%) vs baseline: +5.3% ✅ tracerTime: ✅ 19.735ms (SLO: <21.750ms -9.3%) vs baseline: ~same Memory: ✅ 69.414MB (SLO: <75.000MB -7.4%) vs baseline: +5.8% ✅ tracer-and-profilerTime: ✅ 21.055ms (SLO: <23.500ms 📉 -10.4%) vs baseline: +0.1% Memory: ✅ 71.388MB (SLO: <75.000MB -4.8%) vs baseline: +5.5% ✅ tracer-dont-create-db-spansTime: ✅ 19.851ms (SLO: <21.500ms -7.7%) vs baseline: +0.4% Memory: ✅ 69.226MB (SLO: <75.000MB -7.7%) vs baseline: +5.5% ✅ tracer-minimalTime: ✅ 16.772ms (SLO: <17.500ms -4.2%) vs baseline: ~same Memory: ✅ 69.029MB (SLO: <75.000MB -8.0%) vs baseline: +5.2% ✅ tracer-nativeTime: ✅ 19.656ms (SLO: <21.750ms -9.6%) vs baseline: +0.3% Memory: ✅ 69.324MB (SLO: <72.500MB -4.4%) vs baseline: +5.7% ✅ tracer-no-cachesTime: ✅ 17.708ms (SLO: <19.650ms -9.9%) vs baseline: +0.5% Memory: ✅ 69.029MB (SLO: <75.000MB -8.0%) vs baseline: +5.2% ✅ tracer-no-databasesTime: ✅ 19.346ms (SLO: <20.100ms -3.8%) vs baseline: -0.3% Memory: ✅ 69.088MB (SLO: <75.000MB -7.9%) vs baseline: +5.3% ✅ tracer-no-middlewareTime: ✅ 19.445ms (SLO: <21.500ms -9.6%) vs baseline: -0.2% Memory: ✅ 69.068MB (SLO: <75.000MB -7.9%) vs baseline: +5.3% ✅ tracer-no-templatesTime: ✅ 19.801ms (SLO: <22.000ms -10.0%) vs baseline: +1.5% Memory: ✅ 69.263MB (SLO: <73.500MB -5.8%) vs baseline: +5.6% 🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 3.376ms (SLO: <4.750ms 📉 -28.9%) vs baseline: +0.4% Memory: ✅ 56.452MB (SLO: <66.500MB 📉 -15.1%) vs baseline: +5.3% ✅ appsec-postTime: ✅ 2.861ms (SLO: <6.750ms 📉 -57.6%) vs baseline: -0.3% Memory: ✅ 56.405MB (SLO: <66.500MB 📉 -15.2%) vs baseline: +5.3% ✅ appsec-telemetryTime: ✅ 3.389ms (SLO: <4.750ms 📉 -28.7%) vs baseline: +1.1% Memory: ✅ 56.574MB (SLO: <66.500MB 📉 -14.9%) vs baseline: +5.6% ✅ debuggerTime: ✅ 1.879ms (SLO: <2.000ms -6.1%) vs baseline: ~same Memory: ✅ 49.274MB (SLO: <51.500MB -4.3%) vs baseline: +5.8% ✅ iast-getTime: ✅ 1.869ms (SLO: <2.000ms -6.5%) vs baseline: -0.4% Memory: ✅ 46.003MB (SLO: <49.000MB -6.1%) vs baseline: +5.6% ✅ profilerTime: ✅ 1.908ms (SLO: <2.100ms -9.1%) vs baseline: -0.5% Memory: ✅ 52.511MB (SLO: <53.500MB 🟡 -1.8%) vs baseline: +5.5% ✅ resource-renamingTime: ✅ 3.344ms (SLO: <3.650ms -8.4%) vs baseline: ~same Memory: ✅ 56.493MB (SLO: <60.000MB -5.8%) vs baseline: +5.3% ✅ tracerTime: ✅ 3.351ms (SLO: <3.650ms -8.2%) vs baseline: ~same Memory: ✅ 56.512MB (SLO: <60.000MB -5.8%) vs baseline: +5.5% ✅ tracer-nativeTime: ✅ 3.340ms (SLO: <3.650ms -8.5%) vs baseline: ~same Memory: ✅ 56.515MB (SLO: <60.000MB -5.8%) vs baseline: +5.5% 🟡 recursivecomputation - 8/8✅ deepTime: ✅ 311.519ms (SLO: <320.950ms -2.9%) vs baseline: +0.2% Memory: ✅ 37.493MB (SLO: <38.750MB -3.2%) vs baseline: +5.7% ✅ deep-profiledTime: ✅ 328.398ms (SLO: <359.150ms -8.6%) vs baseline: -0.3% Memory: ✅ 43.785MB (SLO: <46.000MB -4.8%) vs baseline: +5.6% ✅ mediumTime: ✅ 7.260ms (SLO: <7.400ms 🟡 -1.9%) vs baseline: -0.5% Memory: ✅ 36.215MB (SLO: <38.000MB -4.7%) vs baseline: +5.5% ✅ shallowTime: ✅ 1.013ms (SLO: <1.050ms -3.5%) vs baseline: +1.3% Memory: ✅ 36.156MB (SLO: <38.000MB -4.9%) vs baseline: +5.5%
|
Summary
LLMObs._on_exit_signalwithregister_on_exit_signalso SIGTERM triggers graceful flushing of buffered LLMObs spans and eval metrics before process exit. Previously, pods receiving SIGTERM (e.g. during Kubernetes rolling deployments) would silently drop buffered spans.DD_LLMOBS_GRACEFUL_TERMINATION_PERIOD(default0s) — when non-zero, delays shutdown by that many seconds so the application can continue writing spans during a Kubernetes graceful termination window before flushing and stopping. Useful for apps with long-running LLM requests that need to complete before shutdown.LLMObs.SHUTDOWN_TIMEOUT = 5and joins writer threads with this timeout during_stop_service.Closes #17125
Supersedes #17130 (was targeting 4.6)
Notes on
DD_LLMOBS_GRACEFUL_TERMINATION_PERIODUses
threading.Timerso the main thread is not blocked and the application can continue serving during the grace period. This only works if the application framework keeps the process alive after SIGTERM (e.g. gunicorn/uvicorn draining connections). For standard deployments that drain traffic before SIGTERM, the default of0(flush immediately) is correct.Test plan
test_shutdown_timeout_default—SHUTDOWN_TIMEOUTis 5test_enable_registers_sigterm_handler—register_on_exit_signalcalled with_on_exit_signalduring enabletest_sigterm_disables_llmobs— invoking the registered handler disables LLMObs and stops writerstest_sigterm_joins_writers_with_shutdown_timeout—joincalled withSHUTDOWN_TIMEOUTtest_sigterm_with_grace_period_delays_disable— non-zero grace period schedules aTimerinstead of disabling immediatelytest_sigterm_with_zero_grace_period_disables_immediately— zero grace period disables immediately🤖 Generated with Claude Code