Skip to content

feat(otlp): callback-based HTTP exporter with in-flight limit - #103

Merged
LimiNode merged 8 commits into
mainfrom
feat/callback-otlp-exporter
May 21, 2026
Merged

feat(otlp): callback-based HTTP exporter with in-flight limit#103
LimiNode merged 8 commits into
mainfrom
feat/callback-otlp-exporter

Conversation

@LimiNode

@LimiNode LimiNode commented May 21, 2026

Copy link
Copy Markdown
Owner

Summary

Replaces the blocking future.wait_for() HTTP export loop in OtlpHttpLogger with kurlyk callback API, enabling non-blocking batched exports up to a configurable max_in_flight_requests limit. Also introduces a new OtlpPayloadLogger backend that forms OTLP JSON payloads and passes them to a user-provided callback instead of sending HTTP itself.

Motivation

The previous implementation blocked the dedicated worker thread on every HTTP round-trip (future.wait_for(...) -> future.get()). This serialized all batch exports and prevented the worker from dequeuing new logs while a slow collector endpoint was being processed. Under backpressure the queue filled up and triggered drop_on_overflow even though the worker was idle.

Changes

OtlpHttpLogger.hpp

  • Introduces OtlpHttpLoggerState (shared_ptr-held) with mutex, cv, http_in_flight and failed_exports counters.
  • HTTP callback captures weak_ptr<OtlpHttpLoggerState> instead of this, preventing use-after-free if the logger is destroyed before the callback fires.
  • worker_loop() no longer blocks inside export_batch(). It increments http_in_flight, calls submit_batch_async(), and immediately loops back to dequeue the next batch while http_in_flight < max_in_flight_requests.
  • wait() now waits on http_in_flight == 0 && m_queue.empty() via the shared-state condition variable.
  • stop() uses m_client.wait_requests() for graceful drain, or m_client.cancel_requests() when cancel_on_shutdown = true.
  • Config moved to nested OtlpHttpLogger::Config with OtlpJsonFormatConfig format sub-member.

OtlpPayloadLogger.hpp (new)

  • New callback-only backend that serializes batches to OTLP JSON and passes the payload string to a user-provided std::function<void(std::string)>.
  • Supports sync and async modes, queue overflow drop policy, and FailedExportCount / DroppedLogCount diagnostics.
  • Callback exceptions are caught (both sync and async) to prevent killing the worker thread.
  • Config is nested OtlpPayloadLogger::Config sharing OtlpJsonFormatConfig format.

OtlpJsonFormatConfig.hpp (new)

  • Extracted shared serialization settings (service/resource attributes, include_* flags, args_prefix) from the old standalone OtlpHttpLoggerConfig.
  • Used by both OtlpHttpLogger, OtlpPayloadLogger, and build_otlp_logs_json_payload().

Deleted

  • otlp/OtlpHttpLoggerConfig.hpp — replaced by OtlpHttpLogger::Config.
  • otlp/OtlpPayloadLoggerConfig.hpp — replaced by OtlpPayloadLogger::Config.

CMakeLists.txt / tests/CMakeLists.txt

  • Guards install(EXPORT) when OTLP is enabled (kurlyk target not exported).
  • Raises C++ standard to C++17 when LOGIT_WITH_OTLP=ON (required by kurlyk).
  • Adds target_include_directories for Simple-Web-Server headers used by OTLP HTTP integration tests.
  • Adds otlp_payload_logger_test to the test list (guarded by LOGIT_WITH_OTLP).

New tests

otlp_http_logger_callback_test.cpp — 7 scenarios:

  1. Single batch callback export — payload received and JSON structure validated.
  2. max_in_flight=1 blocking — second batch is not submitted until the first callback completes.
  3. max_in_flight=2 parallelism — two batches are submitted concurrently.
  4. HTTP 500 failure counting — FailedExportCount increments correctly.
  5. wait() blocks until callback completes — measured elapsed time >= server delay.
  6. Graceful shutdown without UAF — slow server, cancel_on_shutdown=false, shutdown waits for callback and completes cleanly.
  7. cancel_on_shutdown=true fast path — slow server, shutdown completes in < 1s.

otlp_payload_logger_test.cpp — 8 scenarios:

  1. Sync mode callback receives valid JSON with "resourceLogs".
  2. Sync throwing callback increments FailedExportCount without escaping from log().
  3. Async batching — 5 messages coalesced into 1 payload.
  4. Async queue overflow with drop_on_overflow=trueDroppedLogCount > 0.
  5. wait() blocks until queue drain.
  6. shutdown() stops worker cleanly without deadlock.
  7. Slow async callback — wait() blocks until callback finishes (>= 400 ms).
  8. Throwing async callback — worker survives, FailedExportCount increments.

Verification

  • All 55 tests pass (0 failures).
  • otlp_http_logger_callback_test PASS in ~3.9s.
  • otlp_payload_logger_test PASS.
  • No regressions in existing OTLP integration tests.

Scope risk

Moderate — the threading model changes from "worker thread blocks on HTTP" to "worker thread submits and moves on". The shared-state + weak_ptr pattern ensures safety, but concurrent callback delivery from kurlyk's NetworkWorker thread now decrements http_in_flight asynchronously.

LimiNode and others added 8 commits May 21, 2026 09:10
Replace the blocking future.wait_for() export loop with kurlyk callback
API. This lets the worker thread submit multiple HTTP requests without
waiting for each round-trip, controlled by max_in_flight_requests.

Key changes:
- OtlpHttpLoggerState holds shared mutex, cv, http_in_flight and
  failed_exports counters. The HTTP callback captures a weak_ptr to
  this state, eliminating use-after-free on logger destruction.
- worker_loop no longer blocks on export_batch(); it submits via
  submit_batch_async() and loops immediately while http_in_flight
  stays below max_in_flight_requests.
- wait() now waits on http_in_flight == 0 and queue empty via the
  shared state condition variable.
- stop() uses m_client.wait_requests() for graceful drain, or
  m_client.cancel_requests() when cancel_on_shutdown is true.
- OtlpHttpLoggerConfig gains max_in_flight_requests (default 1) and
  cancel_on_shutdown (default false) to preserve prior semantics.

Tests added in otlp_http_logger_callback_test.cpp cover:
- single batch callback export
- max_in_flight=1 blocking second batch
- max_in_flight=2 parallel export
- HTTP 500 failure counting
- wait() blocks until callback completes
- graceful shutdown without UAF
- cancel_on_shutdown fast path

Constraint: callback must not capture `this`; weak_ptr<State> only.
Rejected: kurlyk future-based blocking API | blocks worker thread, serializes all batches.
Confidence: high
Scope-risk: moderate

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add .claude/ agents, rules, commands, and .cbmignore project config.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Blocker 1 (stop ordering): move m_client.cancel_requests() before
worker.join() so cancel_on_shutdown actually fast-paths. Graceful path
now calls wait_requests() after join().

Blocker 2 (worker wakeup): unify queue + in-flight state under single
OtlpHttpLoggerState with one mutex and one cv. The HTTP callback wakes
worker via state->cv without capturing this, eliminating stalled worker
between export_interval_ms timeouts.

Blocker 3 (CMake install): replace blanket if(NOT LOGIT_WITH_OTLP) guard
with IMPORTED-target-aware conditional. install(EXPORT) is skipped only
when kurlyk is a non-IMPORTED submodule; it still works when kurlyk is
pre-installed, keeping the package usable.

Also:
- Normalize max_in_flight_requests==0 to 1 in constructor.
- Move http_in_flight under state mutex (non-atomic) to avoid mixed
  synchronization model.
- Add .gitignore to exclude local agent configs.
- Remove .claude/proxy-config.json and settings.local.json from repo.

Rejected: per-field atomics + separate mutexes — mixed model is
harder to reason about and callback could not wake worker safely.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… bundled kurlyk

Blocker 1: cancel_on_shutdown=true could still drain and submit queued
batches after cancel_requests() because worker_loop() only checked
stopping + empty queue. Now stop() clears the queue under state mutex
and increments m_dropped; worker_loop() returns immediately when
stopping && cancel_on_shutdown, preventing new submissions.

Blocker 2: CMake install/export with bundled kurlyk now hard-fails at
install time (install(CODE FATAL_ERROR)) instead of producing a broken
installed package. install(EXPORT) is conditional: included only when
kurlyk is pre-installed (IMPORTED) or absent. Development builds with
submodules still configure and compile normally.

Also: removed unused #include <future>.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add OtlpPayloadLogger backend that forms OTLP JSON payloads and passes
them to a user-provided std::function<void(std::string)> callback.
This allows applications with their own telemetry pipeline to reuse
LogIt++'s OTLP formatting and batching without duplicating HTTP
infrastructure.

Includes OtlpPayloadLoggerConfig, queue/worker batching, sync/async
modes, overflow drop policy, and integration tests.

Constraint: OtlpPayloadLogger.hpp and tests guarded by LOGIT_WITH_OTLP.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wrap on_payload in try/catch in both sync and async paths. Add
FailedExportCount counter and param getter. Fix m_idle so it is set
true only after callback completes, preventing wait()/shutdown() hang
if callback throws. Add tests for slow callback and throwing callback.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extract serialization settings into shared OtlpJsonFormatConfig.
Convert OtlpHttpLogger and OtlpPayloadLogger to use nested Config
structs (OtlpHttpLogger::Config, OtlpPayloadLogger::Config) following
the library pattern used by other backends. Serialization fields live
under a `format` sub-member.

Delete standalone OtlpHttpLoggerConfig.hpp and
OtlpPayloadLoggerConfig.hpp. Update all tests, example, and docs.

All 55 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Update tool-priority.md to elevate Codebase Memory to first-pass
codebase discovery before Grep/Read/LSP. Add explicit Codebase Memory
Usage Policy section. Update CLAUDE.md specific overrides to reflect
Codebase Memory as the lightest path for non-trivial codebase
understanding.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@LimiNode
LimiNode merged commit 2ef7a7c into main May 21, 2026
12 checks passed
@LimiNode
LimiNode deleted the feat/callback-otlp-exporter branch May 21, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant