fork-safe agent lifecycle

[dwkang]

Summary

The agent is not safe to use across fork(). The most common Python (and many
C++/PHP) deployment models — gunicorn, uWSGI, multiprocessing(fork) —
initialize the agent in a master process and then fork worker processes. Because
the agent's gRPC worker threads and gRPC runtime do not survive fork(), forked
workers inherit a broken agent. Downstream bindings (e.g.
pinpoint-python-agent) currently can only work around this with fragile hacks.
This issue proposes first-class fork support in the C++ agent.

Background: what breaks after fork

1. Worker threads are dead in the child. AgentImpl's constructor starts
   init_thread_ (which spawns the ping/meta/span/stat/url-stat/command workers
   via init_grpc_workers()). Threads are not duplicated across fork(), so a
   child inherits an AgentImpl whose background threads no longer exist. Spans
   enqueued in the child are silently dropped.

2. Destroying the inherited agent crashes. ~AgentImpl() → do_shutdown()
   → close_grpc_workers() → wait_grpc_workers() joins the worker
   threads. In the child those std::thread handles are joinable but reference
   threads that don't exist → pthread_join fails / std::system_error escapes
   the joiner thread → std::terminate() → abort. This also fires via the
   atexit/Shutdown() path at worker exit.

3. CreateAgent() cannot rebuild. create_agent_helper() returns the
   existing global agent (with reloadConfig()) whenever
   global_agent() != nullptr. In a forked child the global is non-null
   (inherited), so CreateAgent() just hands back the same dead agent —
   there is no way to obtain a fresh, working one.

4. gRPC runtime is initialized at create time. GrpcClient::GrpcClient()
   (src/grpc.cpp:186-190) calls build_channel() →
   grpc::CreateCustomChannel(), and the Grpc* clients are constructed inside
   make_agent(). So grpc_init() and gRPC's own background threads are up as
   soon as CreateAgent() runs. Forking a process with an active gRPC stack is
   deadlock-prone unless gRPC's fork handlers are enabled.

5. agent_id is shared across children. generate_agent_id()
   (src/config.cpp, hostname[:18]-random5) runs once at config time. If a
   master generates the id and forks, all workers would register under the same
   id.

Why this can't be fixed in the binding/Python layer alone

• create_agent() never rebuilds while the global is set, and
  Shutdown()/destruction joins dead threads → crash. The only internal escape
  hatch, reset_global_agent(), is not declared in the public header
  (include/pinpoint/tracer.h), and even calling it risks destroying the agent
  (→ join → crash) unless the object is deliberately leaked.
• As a stopgap, a downstream binding currently must: forward-declare the
  internal reset_global_agent() symbol, permanently leak the inherited
  AgentImpl (a never-freed shared_ptr) so its destructor never joins, then
  create_agent() again, and separately set GRPC_ENABLE_FORK_SUPPORT=1 from
  Python. This works but depends on an unexported-by-contract symbol and manual
  leaking — it belongs in the agent.

Proposed functionality

1. (Recommended) Split construction from startup — "cold create + Start()"

Make CreateAgent() cold: parse config and allocate the object only — no
threads, no gRPC channel / grpc_init, no config-file watcher. Add an
explicit startup entry point that brings the agent online in the current
process:

// Cold: config + object allocation only. Safe to call in a master that
// will fork; the returned agent holds no threads and has not touched gRPC.
AgentPtr CreateAgent(int32_t app_type, /* ...server metadata... */);

// Bring the agent online in THIS process: build gRPC channels (grpc_init),
// spawn worker threads, start the config-file watcher, and (re)generate a
// process-unique agent id. Idempotent and non-blocking (async connect).
void Agent::Start();

This enables the industry-standard "initialize per worker after fork" model
(cf. OpenTelemetry's post_fork guidance): the master calls CreateAgent(),
and each child calls Start() from its fork hook. The parent never holds live
threads or an open gRPC runtime at the fork point, which removes failure modes
1–4 at the root and makes the inherited object directly reusable (no reset/leak
needed).

Note: deferring only the pinpoint worker threads is insufficient —
grpc_init must also move into Start() (see failure mode 4), otherwise the
gRPC runtime is still up at fork time.

2. PID-guarded teardown (safety net)

Store owner_pid_ = getpid() in AgentImpl and, in do_shutdown()/
~AgentImpl(), detach instead of join when getpid() != owner_pid_. This
guarantees no crash even if an agent that was already started in the master gets
destroyed in a child (preload/master-tracing cases).

3. Agent owns gRPC fork support

Enable gRPC's fork handlers from within the agent before the first channel is
created (equivalent to GRPC_ENABLE_FORK_SUPPORT=1, which grpcio implements via
pthread_atfork). Since the agent owns gRPC initialization, this is the correct
layer — bindings shouldn't have to set a global env var.

4. Process-unique agent_id on start

Generate/refresh the agent_id at Start() time so each forked worker gets a
unique identity (random when unset; a pid suffix or regeneration when an
explicit id is pinned).

Acceptance criteria

• A master process can create an agent, fork(), and have each child obtain
  a fully working agent (live workers, spans delivered) without crashing or
  deadlocking.
• Destroying/shutting down an inherited agent in a child never joins dead
  threads (no abort).
• Forked workers register with distinct agent_ids.
• No global env var or unexported symbol is required from the binding.
• Behavior on non-fork platforms is unchanged.