feat: native socket I/O tracking via PLT hooks (PROF-10637)

[jbachorik]

What does this PR do?

Intercepts libc send, recv, write, read via PLT hooking and records
datadog.NativeSocketEvent JFR events with time-weighted inverse-transform
sampling (Poisson process, PID rate control, target ~83 events/s / ~5000/min).

Motivation

Track blocking TCP socket I/O at the libc function level to surface socket
latency and throughput in the Datadog profiler. Netty with Java NIO transport
(the primary use case) goes through libc send/recv/write/read, making
PLT patching the right interception point. Feature is explicitly opt-in via
the natsock profiler argument. Implements PROF-10637.

Configuration

Argument	Effect
natsock	Enable native socket tracking with the default 1 ms initial period
natsock=100us	Enable with an explicit initial sampling period (any parseUnits unit: ns, us, ms, s)

The interval is the initial parameter of the time-weighted Poisson process;
the PID controller adapts it at runtime to hold the total event rate near
TARGET_EVENTS_PER_SECOND (= 83 events/s) across all four hooks combined.

Architecture

flowchart LR
    A["Java code<br/>OutputStream.write()"] --> B["JDK native<br/>libnio/libnet"]
    B --> C{"PLT entry<br/>for write()"}
    C -- patched --> D["NativeSocketSampler::write_hook"]
    D --> E{"fd is TCP<br/>socket?"}
    E -- no --> F["orig write()"]
    E -- yes --> G["orig write()"]
    G --> H{"shouldSample<br/>(duration, op)"}
    H -- no --> I["return"]
    H -- yes --> J["recordEvent"]
    J --> K["JFR buffer"]
    F --> I

Loading

Sampling pipeline

flowchart TD
    A["Hook fires with<br/>duration_ticks"] --> B["PoissonSampler<br/>(thread_local)"]
    B --> C{"interval budget<br/>consumed?"}
    C -- no --> D["reject"]
    C -- yes --> E["compute weight<br/>1 / (1 - exp(-d/interval))"]
    E --> F["RateLimiter::accept()"]
    F --> G{"PID budget<br/>for this epoch?"}
    G -- no --> D
    G -- yes --> H["record event<br/>operation / fd-addr / bytes / weight"]
    H --> I["RateLimiter<br/>maybeUpdateInterval<br/>(PID: P=31 I=511 D=3)"]
    I --> J["next epoch interval"]

Loading

Two thread_local PoissonSampler instances per thread: outbound (send+write)
and inbound (recv+read). Both share one RateLimiter so the ~83 events/s
target is a single budget across all directions.

Hook installation

flowchart TD
    A["NativeSocketSampler::start"] --> B["dlsym RTLD_NEXT<br/>(RTLD_DEFAULT fallback on musl)<br/>resolve _orig_send/recv/write/read"]
    B --> C["patch_socket_functions"]
    C --> D["walk every loaded CodeCache"]
    D --> E{"lib imports<br/>send/recv/write/read?"}
    E -- yes --> F["rewrite PLT entry<br/>to *_hook"]
    E -- no --> G["skip"]
    C --> H["also patch dlopen PLT"]
    H --> I["dlopen_hook wraps<br/>subsequent loads"]
    I --> J["updateSymbols +<br/>install_socket_hooks"]
    J --> E

Loading

Late-loaded libraries (e.g. HotSpot lazily loads libnet.so on the first
java.net.Socket use) are caught by the dlopen wrapper which re-runs
patch_socket_functions after each new library is registered.

JFR event fields (datadog.NativeSocketEvent)

Field	Type	Description
startTime	long	Event start (ticks)
eventThread	Thread	Calling Java thread (constant pool)
stackTrace	StackTrace	Captured stack (constant pool)
duration	long	Syscall duration (ticks)
operation	string	SEND or RECV
remoteAddress	string	ip:port (IPv4 or [ipv6]:port); empty if unknown
bytesTransferred	long	Bytes actually transferred
weight	float	Inverse sampling probability (1 / P)
spanId	long	Datadog span id from active context
localRootSpanId	long	Datadog local root span id
context attrs	strings	User context attributes

sum(weight × duration) is an unbiased estimator of total socket I/O time;
sum(weight × bytesTransferred) estimates total bytes.

Key design notes

• Scope: TCP blocking send/recv/write/read only. UDP (sendto/recvfrom)
  and Netty's native epoll / io_uring transports are explicitly out of scope.
• Fast path: write/read hooks short-circuit on non-socket fds via a
  lock-free std::atomic<uint8_t> _fd_type_cache[65536] — one relaxed atomic
  load per call on the non-socket path.
• Thread context: hook bodies run on the calling Java thread (not a signal
  handler), so malloc and std::mutex are safe.
• PLT re-patching on dlopen: install_socket_hooks() (called from
  dlopen_hook after updateSymbols) re-runs patch_socket_functions so
  lazily-loaded libraries (libnet.so, libnio.so, native Netty transports)
  are patched the moment they appear.
• fd-to-address cache: intentionally not cleared on JFR chunk boundaries;
  stale addresses for reused fds are a known, accepted trade-off to avoid
  racing with in-flight recordings.
• Original function pointer lifecycle: _orig_send/recv/write/read are
  assigned once (when _socket_size == 0) and are intentionally not nulled in
  unpatch_socket_functions to avoid a memory-ordering race with in-flight
  hook invocations that may still be executing.
• Platform scope: #if defined(__linux__) — compiled as no-op stubs on
  macOS. Runs on both glibc and musl Linux; on musl RTLD_NEXT returns NULL
  (libc precedes the profiler DSO in the link map), so RTLD_DEFAULT is used
  as fallback to locate the real libc symbols. Feature is additionally disabled
  in Java tests on J9/Zing via Platform.isJ9() / Platform.isZing() guards.
• NativeSocketEvent lives in nativeSocketSampler.h (not the shared
  event.h) since it is only used by NativeSocketSampler.

How to test the change?

• C++ unit tests — NativeSocketSamplerHookTest in
  ddprof-lib/src/test/cpp/nativeSocketSampler_ut.cpp — verifies that
  send_hook/recv_hook/write_hook/read_hook each delegate to the
  installed _orig_* pointer and return its value.

• JUnit integration tests in
  ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/:

  Test	What it verifies
  NativeSocketEnabledTest	Events produced when feature is enabled
  NativeSocketDisabledTest	No events when feature is not enabled
  NativeSocketEventFieldsTest	All required JFR fields present and valid
  NativeSocketEventThreadTest	eventThread populated with the calling thread
  NativeSocketStackTraceTest	Stack trace captured on events
  NativeSocketRemoteAddressTest	remoteAddress in ip:port format
  NativeSocketSendRecvSeparateTest	SEND and RECV tracked independently
  NativeSocketBytesAccuracyTest	sum(weight × duration) unbiased estimator within tolerance
  NativeSocketRateLimitTest	Event count ≪ operation count (PID rate control active); weight > 1 on sampled events
  NativeSocketRestartTest	Events produced after profiler stop/restart cycle
  NativeSocketUdpExcludedTest	UDP (sendto/recvfrom) produces zero events
  NativeSocketMacOsNoOpTest	No events on macOS (no-op stub)

Spec: #487

For Datadog employees

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.
• JIRA: PROF-10637

[dd-octo-sts]

CI Test Results

Run: #27286948682 | Commit: c878bb6 | Duration: 12m 20s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	✅	-	-
8-ibm	-	✅	-	-
8-j9	✅	✅	-	-
8-librca	-	-	✅	✅
8-orcl	-	✅	-	-
11	-	✅	-	-
11-j9	✅	✅	-	-
11-librca	-	-	✅	✅
17	✅	✅	-	-
17-graal	✅	✅	-	-
17-j9	✅	✅	-	-
17-librca	-	-	✅	✅
21	✅	✅	-	-
21-graal	✅	✅	-	-
21-librca	-	-	✅	✅
25	✅	✅	-	-
25-graal	✅	✅	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

---

Updated: 2026-06-10 15:43:02 UTC

[jbachorik]

Could not apply automatically — The reviewer comment (ID 4261434558) is an automated CI test-results report showing all 32 jobs passed with no failures. There is no code concern, requested change, or actionable finding to address with a patch.. Manual attention needed.

[jbachorik]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb894e92c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Adds an opt-in native socket I/O profiling feature on Linux by PLT-hooking libc send/recv/write/read and emitting a new datadog.NativeSocketEvent JFR event type with time-weighted Poisson sampling and PID-based global rate control.

Changes:

• Introduces NativeSocketSampler (Poisson sampler + PID RateLimiter) and integrates it into the profiler start/stop lifecycle and JFR writer/metadata.
• Extends Linux LibraryPatcher/CodeCache import scanning to patch GOT/PLT slots for send/recv/write/read, including repatching on library refresh.
• Adds C++ unit tests plus Java integration tests and a JMH benchmark for coverage and overhead evaluation.

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketUdpExcludedTest.java	Adds UDP negative test for zero NativeSocketEvent emission.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketTestBase.java	Shared TCP workload helper for nativesocket tests.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketStackTraceTest.java	Validates stack traces are present on socket events across cstack modes.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketSendRecvSeparateTest.java	Validates SEND/RECV directionality is tracked independently.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketRestartTest.java	Validates stop/restart lifecycle continues to produce events.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketRemoteAddressTest.java	Validates remoteAddress formatting and matching server port.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketRateLimitTest.java	Validates subsampling/rate limiting and weight behavior.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketMacOsNoOpTest.java	Validates macOS no-op behavior produces no events.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketEventThreadTest.java	Validates eventThread population.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketEventFieldsTest.java	Validates required event fields exist and are valid.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketEnabledTest.java	Smoke test: events emitted when enabled.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketDisabledTest.java	Smoke test: no events when not enabled.
ddprof-test/src/test/java/com/datadoghq/profiler/nativesocket/NativeSocketBytesAccuracyTest.java	Validates sum(weight × duration) is within reasonable bounds.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/NativeSocketOverheadBenchmark.java	Adds JMH benchmark to quantify hook overhead for file/socket/NIO writes.
ddprof-lib/src/test/cpp/nativeSocketSampler_ut.cpp	Adds gtest coverage for hooks, PoissonSampler, and args parsing.
ddprof-lib/src/main/cpp/vmEntry.h	Adds BCI_NATIVE_SOCKET frame type.
ddprof-lib/src/main/cpp/rateLimiter.h	Adds shared PID-based rate limiter used by socket sampler.
ddprof-lib/src/main/cpp/profiler.cpp	Integrates nativesocket engine into start/stop/check and refresher ordering.
ddprof-lib/src/main/cpp/poissonSampler.h	Adds thread-local Poisson-process sampler with inverse-probability weights.
ddprof-lib/src/main/cpp/nativeSocketSampler.h	Declares sampler engine, caches, and hook entry points.
ddprof-lib/src/main/cpp/nativeSocketSampler.cpp	Implements hooks, fd classification/address caching, sampling, and event recording.
ddprof-lib/src/main/cpp/livenessTracker.cpp	Fixes _record_heap_usage initialization on reused singleton.
ddprof-lib/src/main/cpp/libraryPatcher.h	Adds socket patching APIs and _socket_active state.
ddprof-lib/src/main/cpp/libraryPatcher_linux.cpp	Implements GOT/PLT patch/unpatch for send/recv/write/read with dlopen refresh support.
ddprof-lib/src/main/cpp/libraries.cpp	Triggers socket hook installation during library refresh.
ddprof-lib/src/main/cpp/jvmSupport.cpp	Allows OpenJ9/Zing stack walking for BCI_NATIVE_SOCKET.
ddprof-lib/src/main/cpp/jfrMetadata.h	Adds new JFR type id for native socket event.
ddprof-lib/src/main/cpp/jfrMetadata.cpp	Registers datadog.NativeSocketEvent fields in metadata.
ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp	Extends stack walking logic to handle BCI_NATIVE_SOCKET.
ddprof-lib/src/main/cpp/flightRecorder.h	Adds recording API for native socket samples.
ddprof-lib/src/main/cpp/flightRecorder.cpp	Serializes native socket event payload into JFR buffers.
ddprof-lib/src/main/cpp/event.h	Adds NativeSocketEvent struct used by recorder.
ddprof-lib/src/main/cpp/codeCache.h	Adds import IDs for send/recv/write/read.
ddprof-lib/src/main/cpp/codeCache.cpp	Detects and stores GOT imports for send/recv/write/read.
ddprof-lib/src/main/cpp/arguments.h	Adds _nativesocket + _nativesocket_interval and event mask bit.
ddprof-lib/src/main/cpp/arguments.cpp	Parses natsock[=<duration>] and hardens parseUnits overflow handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rkennke]

This is a nice feature! I have a few requests.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2a04e9ea9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[rkennke]

Thanks for addressing the issues. Looks good now!

[jbachorik]

@rkennke Thanks for the patient review :)