Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ on:

env:
CARGO_TERM_COLOR: always
RUSTFLAGS: --cfg tokio_unstable

jobs:
fmt:
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

## [Unreleased]

## 0.1.0 - 2026-04-19
## 0.1.0 - 2026-04-26

Initial release.
135 changes: 53 additions & 82 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,7 @@

Opinionated telemetry setup for Rust services.

## What it does

`telemetry` provides a small builder for the telemetry setup we want by default:

- formatted local `tracing` logs to stdout
- optional OTLP export for traces, logs, and metrics
- optional `journald` output
- optional localhost-only log-level control API
- optional Tokio runtime metrics exported through the OpenTelemetry global meter pipeline

The API is intentionally small and opinionated.

## Prerequisites

If you enable the `tokio-metrics` feature, compile the consuming process with:

```bash
RUSTFLAGS="--cfg tokio_unstable"
```

Tokio exposes the runtime metrics APIs used by this crate only behind that cfg.

## Usage
## Quick start

```rust
use telemetry_setup::TelemetryBuilder;
Expand All @@ -40,39 +18,42 @@ fn main() -> Result<(), telemetry_setup::TelemetryError> {
}
```

Keep the returned `TelemetryGuard` alive for the process lifetime.
This initialization is process-global; calling `init()` more than once in the
same process is unsupported.
Keep the returned `TelemetryGuard` alive for the lifetime of the process:

- call `TelemetryGuard::shutdown().await` during teardown for a clean flush
- initialize telemetry once per process; calling `init()` more than once is unsupported
- dropping the guard without calling `shutdown()` first is only a best-effort fallback
- on multi-thread Tokio runtimes, drop attempts a final OTLP flush
- on `current_thread` runtimes, OTLP flush-on-drop is unavailable because that path relies on `tokio::task::block_in_place`, so explicit shutdown is strongly preferred

For graceful shutdown, call `TelemetryGuard::shutdown().await` during teardown.
Dropping the guard without calling `shutdown` first performs a best-effort fallback.
When OTLP is enabled, drop attempts provider shutdown through
`tokio::task::block_in_place` on a multi-thread Tokio runtime. This crate enables
Tokio's `rt-multi-thread` feature to support that drop path. Consumers running a
`current_thread`-only runtime should call `TelemetryGuard::shutdown().await`
explicitly rather than relying on `Drop`. On a `current_thread` runtime the drop
fallback is unavailable, so it emits a stderr warning and may skip the final
OTLP flush. Explicit shutdown during teardown is strongly preferred so
background tasks can finish and final telemetry can flush in a predictable
order.
This crate enables Tokio's `rt-multi-thread` feature to support the multi-thread drop path.

## Optional features
## Prerequisites

Tokio metrics require the `tokio-metrics` feature and an installed OpenTelemetry
meter provider. With OTLP enabled, this crate installs one automatically.
Without OTLP, consumers must install their own meter provider or Tokio metrics
will record into OpenTelemetry's default no-op global meter.

## Features

- formatted `tracing` logs to stdout
- optional OTLP export for traces, logs, and metrics
- optional `journald` output
- optional localhost-only log-level control API
- optional Tokio runtime metrics via OpenTelemetry

- `otlp`: OTLP trace/log/metric export; exposes `OtlpConfig`,
`OtlpHeadersConfig`, and `TelemetryBuilder::with_otlp_config(...)`
- `journald`: `tracing-journald` output; use `TelemetryBuilder::enable_journald()`
- `log-control`: HTTP endpoints on `127.0.0.1` for runtime filter changes;
exposes `LogControlConfig` and `TelemetryBuilder::with_log_control(...)`
- `tokio-metrics`: Tokio runtime gauges exported through OpenTelemetry; use `TelemetryBuilder::enable_tokio_metrics()`
Cargo features:

GreptimeDB export does not require a dedicated crate feature. Configure GreptimeDB
OTLP headers explicitly through `OtlpConfig::headers`; see
`examples/greptime_otlp.toml` and `examples/greptime_otlp.rs`.
- `otlp` — enables OTLP trace, log, and metric export and the OTLP configuration types
- `journald` — enables `tracing-journald` output
- `log-control` — enables localhost-only HTTP endpoints for runtime filter changes
- `tokio-metrics` — enables Tokio runtime gauges via OpenTelemetry

## Common configuration
## Full configuration example

This example assumes the `otlp`, `log-control`, `journald`, and
`tokio-metrics` crate features are enabled.
This example assumes the `otlp`, `log-control`, `journald`, and `tokio-metrics`
crate features are enabled.

```rust
# use std::collections::HashMap;
Expand Down Expand Up @@ -109,48 +90,38 @@ let _telemetry = TelemetryBuilder::new("controller")

Defaults:

- local filter comes from `RUST_LOG`, falling back to `info`
- log-control port defaults to `6669`
- OTLP defaults are local-first and target `http://localhost:4318` for OTLP/HTTP
- OTLP metric export defaults to a 5 second interval and can be overridden with `OtlpConfig::metrics_interval` between 1 second and 1 day
- Tokio runtime metric collection defaults to a 5 second interval and can be overridden with `TelemetryBuilder::with_tokio_metrics_interval(...)`
- stdout filter: `RUST_LOG`, falling back to `info`
- log-control port: `6669`
- OTLP endpoint: `http://localhost:4318`
- OTLP metric interval: 5 seconds, configurable from 1 second to 1 day
- Tokio metrics interval: 5 seconds

## Documentation and examples
Use `TelemetryBuilder::without_env_var()` to ignore `RUST_LOG`.

API documentation is generated by `cargo doc` from crate and module rustdoc. The
crate is configured explicitly by the consuming service; it does not discover or
load files from the repository automatically.
## Log control API

Compilable example applications live in `examples/`:
When `log-control` is enabled, the crate binds an HTTP server only on `127.0.0.1`.
There is no authentication, so any local process can inspect and change runtime filters.
Invalid filters return `400`.

- `stdout_only.rs` for local stdout logging with no exporters or control API
- `stdout_custom_filter.rs` for an explicit fallback stdout filter
- `greptime_otlp.rs` and `greptime_otlp.toml` for GreptimeDB OTLP/HTTP traces,
logs, and metrics configuration
- `GET /filters` returns `{ "stdout": "...", "otlp": "..." | null }`
- `PUT /filters/stdout` accepts `{ "filter": "..." }` and returns the updated filter state
- `PUT /filters/otlp` accepts `{ "filter": "..." }`, returns the updated filter state, and returns `404` when OTLP is unavailable

Notes:
## Examples

- Tokio metrics require the `tokio-metrics` crate feature and any installed OpenTelemetry meter provider
- Tokio metrics also require compiling the process with `RUSTFLAGS="--cfg tokio_unstable"`
- call `TelemetryGuard::shutdown().await` to gracefully stop background tasks before OTLP providers are shut down so final telemetry can flush cleanly
- dropping `TelemetryGuard` without calling `shutdown` first falls back to best-effort teardown, aborts any tasks that are still running, and then attempts provider shutdown via `tokio::task::block_in_place` on multi-thread runtimes
- the OTLP log rate limit is intentionally approximate at Unix-second boundaries under contention; it is a best-effort overload guard, not exact accounting
- OTLP rate-limit warnings use at most one helper thread at a time to avoid spawning a new thread on every overflow transition during sustained overload
- use `TelemetryBuilder::without_env_var()` when the process environment must not override the fallback stdout filter
See `examples/`:

## Log control API
- `stdout_only.rs` — local stdout logging with no exporters or control API
- `stdout_custom_filter.rs` — stdout logging with an explicit fallback filter
- `greptime_otlp.rs` — GreptimeDB OTLP/HTTP traces, logs, and metrics configuration

When `log-control` is enabled, the crate binds an HTTP server only on `127.0.0.1`.
There is no authentication, so any local process can inspect and change runtime
filters.

Endpoints:
Companion configuration:

- `GET /filters` returns `{ "stdout": "...", "otlp": "..." | null }`
- `PUT /filters/stdout` accepts `{ "filter": "..." }` and returns the updated filter state or `400`
- `PUT /filters/otlp` accepts `{ "filter": "..." }` and returns the updated filter state, `400`, or `404` when OTLP is unavailable
- `greptime_otlp.toml` — configuration for the GreptimeDB example

`PUT /filters/otlp` only works when OTLP is enabled for the process.
GreptimeDB export does not require a dedicated feature; configure headers with
`OtlpConfig::headers`.

## License

Expand Down
9 changes: 5 additions & 4 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,14 @@
//! - `otlp`: OTLP trace, log, and metric export.
//! - `journald`: `tracing-journald` output.
//! - `log-control`: localhost-only runtime filter update endpoints.
//! - `tokio-metrics`: Tokio runtime gauges exported through OpenTelemetry.
//! - `tokio-metrics`: Tokio runtime gauges recorded through the OpenTelemetry global meter.
//!
//! # Prerequisites
//!
//! When enabling the `tokio-metrics` feature, compile the consuming process
//! with `RUSTFLAGS="--cfg tokio_unstable"`. Tokio exposes the runtime metrics
//! used by this crate only behind that cfg.
//! Tokio runtime metrics are recorded through the OpenTelemetry global meter.
//! This crate installs a global meter provider when `otlp` is enabled. Without
//! `otlp`, consumers must install their own global meter provider or the metrics
//! will be recorded into OpenTelemetry's default no-op meter.
//!
//! # Examples
//!
Expand Down
3 changes: 2 additions & 1 deletion src/otlp/endpoint.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ pub(super) fn signal_endpoint(
) -> Result<String, TelemetryError> {
let parsed =
Url::parse(base_url).map_err(|error| TelemetryError::otlp_endpoint(signal, error))?;
if parsed.path().ends_with(suffix) {
let expected_suffix = format!("/{suffix}");
if parsed.path().ends_with(&expected_suffix) {
return Ok(base_url.to_string());
}

Expand Down
13 changes: 2 additions & 11 deletions src/tokio_metrics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,7 @@ struct TokioRuntimeMetrics {
/// Starts a background task that records Tokio runtime gauges at the provided interval.
///
/// This monitor must be started from within an active Tokio runtime because it
/// uses [`tokio::runtime::Handle::current()`]. The underlying Tokio runtime
/// metrics APIs also require compiling the process with
/// `RUSTFLAGS="--cfg tokio_unstable"` when the `tokio-metrics` crate feature is
/// enabled.
/// uses [`tokio::runtime::Handle::current()`].
///
/// The task runs until `cancel_token` is cancelled. The returned join handle
/// tracks the task lifetime.
Expand All @@ -43,7 +40,7 @@ pub(crate) fn start_tokio_metrics_monitoring(

/// Starts a Tokio metrics task with a caller-provided recording interval.
///
/// This helper has the same runtime and `tokio_unstable` requirements as
/// This helper has the same runtime requirements as
/// [`start_tokio_metrics_monitoring`].
///
/// # Arguments
Expand Down Expand Up @@ -75,9 +72,6 @@ fn start_tokio_metrics_monitoring_with_interval(

/// Runs the Tokio metrics collection loop until cancelled.
///
/// This loop assumes it is executing on a Tokio runtime whose metrics are
/// available via `tokio_unstable`.
///
/// # Arguments
///
/// * `cancel_token` - Token that signals the loop to stop collecting metrics.
Expand Down Expand Up @@ -110,9 +104,6 @@ async fn run_tokio_metrics_monitoring<F>(

/// Collects the current Tokio runtime metrics from `handle`.
///
/// Tokio exposes these runtime metrics only when the process is compiled with
/// `RUSTFLAGS="--cfg tokio_unstable"`.
///
/// # Arguments
///
/// * `handle` - Runtime handle whose metrics should be sampled.
Expand Down
9 changes: 5 additions & 4 deletions tests/init_log_control.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@ use std::net::{TcpListener, TcpStream};

use telemetry_setup::{LogControlConfig, TelemetryBuilder};

fn free_port() -> u16 {
fn reserve_port() -> (TcpListener, u16) {
let listener = TcpListener::bind("127.0.0.1:0").expect("bind ephemeral port");
let port = listener.local_addr().expect("read local addr").port();
drop(listener);
port
(listener, port)
}

async fn raw_http_request(port: u16, request: String) -> String {
Expand Down Expand Up @@ -47,7 +46,9 @@ async fn raw_http_put_json(port: u16, path: &str, body: &str) -> String {

#[tokio::test]
async fn log_control_server_accepts_filter_update_after_init() {
let port = free_port();
let (reserved_listener, port) = reserve_port();
drop(reserved_listener);

let mut guard = TelemetryBuilder::new("test-lc")
.without_env_var()
.with_stdout_filter("info")
Expand Down
9 changes: 5 additions & 4 deletions tests/init_otlp_log_control.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,10 @@ use std::time::Duration;
use telemetry_setup::{LogControlConfig, OtlpConfig, TelemetryBuilder};
use tracing::Level;

fn free_port() -> u16 {
fn reserve_port() -> (TcpListener, u16) {
let listener = TcpListener::bind("127.0.0.1:0").expect("bind ephemeral port");
let port = listener.local_addr().expect("read local addr").port();
drop(listener);
port
(listener, port)
}

fn unused_local_url() -> String {
Expand Down Expand Up @@ -59,7 +58,9 @@ async fn raw_http_put_json(port: u16, path: &str, body: &str) -> String {

#[tokio::test]
async fn otlp_filter_update_reloads_live_filter_behavior() {
let port = free_port();
let (reserved_listener, port) = reserve_port();
drop(reserved_listener);

let mut guard = TelemetryBuilder::new("test-otlp-log-control")
.without_env_var()
.with_stdout_filter("error")
Expand Down
Loading