Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@ otlp = [
]
journald = ["dep:tracing-journald"]
log-control = ["dep:axum"]
tokio-metrics = ["otlp"]
tokio-metrics = ["dep:opentelemetry"]

[dependencies]
tracing = "0.1"
tracing-core = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt", "registry"] }
thiserror = "2.0"
serde = { version = "1.0", features = ["derive"] }
tokio = { version = "1.52", features = ["net", "rt", "sync", "time"] }
tokio = { version = "1.52", features = ["net", "rt", "rt-multi-thread", "sync", "time"] }
opentelemetry = { version = "0.31", optional = true }
opentelemetry-otlp = { version = "0.31", features = ["http-proto", "trace", "metrics", "logs"], optional = true }
opentelemetry_sdk = { version = "0.31", features = ["rt-tokio", "metrics", "logs"], optional = true }
Expand All @@ -61,3 +61,19 @@ name = "stdout_custom_filter"
toml = "1.1"
tokio = { version = "1.52", features = ["macros", "rt-multi-thread", "time"] }
tower = { version = "0.5", features = ["util"] }

[[test]]
name = "init_otlp"
required-features = ["otlp"]

[[test]]
name = "init_otlp_idempotent"
required-features = ["otlp"]

[[test]]
name = "init_log_control"
required-features = ["log-control"]

[[test]]
name = "init_otlp_log_control"
required-features = ["otlp", "log-control"]
31 changes: 20 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Opinionated shared telemetry setup for workspace Rust services.
- optional OTLP export for traces, logs, and metrics
- optional `journald` output
- optional localhost-only log-level control API
- optional Tokio runtime metrics exported through the OTLP/OpenTelemetry pipeline
- optional Tokio runtime metrics exported through the OpenTelemetry global meter pipeline

The API is intentionally small and opinionated.

Expand Down Expand Up @@ -46,9 +46,15 @@ same process is unsupported.

For graceful shutdown, call `TelemetryGuard::shutdown().await` during teardown.
Dropping the guard without calling `shutdown` first performs a best-effort fallback.
When OTLP is enabled, provider shutdown is blocking, so drop avoids running that
blocking shutdown path on an active Tokio runtime thread and may skip the final
OTLP flush. Explicit shutdown during teardown is strongly preferred.
When OTLP is enabled, drop attempts provider shutdown through
`tokio::task::block_in_place` on a multi-thread Tokio runtime. This crate enables
Tokio's `rt-multi-thread` feature to support that drop path. Consumers running a
`current_thread`-only runtime should call `TelemetryGuard::shutdown().await`
explicitly rather than relying on `Drop`. On a `current_thread` runtime the drop
fallback is unavailable, so it emits a stderr warning and may skip the final
OTLP flush. Explicit shutdown during teardown is strongly preferred so
background tasks can finish and final telemetry can flush in a predictable
order.

## Optional features

Expand All @@ -57,7 +63,7 @@ OTLP flush. Explicit shutdown during teardown is strongly preferred.
- `journald`: `tracing-journald` output; use `TelemetryBuilder::enable_journald()`
- `log-control`: HTTP endpoints on `127.0.0.1` for runtime filter changes;
exposes `LogControlConfig` and `TelemetryBuilder::with_log_control(...)`
- `tokio-metrics`: Tokio runtime gauges exported through OTLP/OpenTelemetry; implies `otlp`; use `TelemetryBuilder::enable_tokio_metrics()`
- `tokio-metrics`: Tokio runtime gauges exported through OpenTelemetry; use `TelemetryBuilder::enable_tokio_metrics()`

GreptimeDB export does not require a dedicated crate feature. Configure GreptimeDB
OTLP headers explicitly through `OtlpConfig::headers`; see
Expand All @@ -69,9 +75,9 @@ This example assumes the `otlp`, `log-control`, `journald`, and
`tokio-metrics` crate features are enabled.

```rust
use std::collections::HashMap;
use telemetry_setup::{LogControlConfig, OtlpConfig, OtlpHeadersConfig, TelemetryBuilder};

# use std::collections::HashMap;
# use telemetry_setup::{LogControlConfig, OtlpConfig, OtlpHeadersConfig, TelemetryBuilder};
# fn example() -> Result<(), telemetry_setup::TelemetryError> {
let _telemetry = TelemetryBuilder::new("controller")
.with_stdout_filter("info")
.with_otlp_config(OtlpConfig {
Expand All @@ -97,14 +103,17 @@ let _telemetry = TelemetryBuilder::new("controller")
.enable_journald()
.enable_tokio_metrics()
.init()?;
# Ok(())
# }
```

Defaults:

- local filter comes from `RUST_LOG`, falling back to `info`
- log-control port defaults to `6669`
- OTLP defaults are local-first and target `http://localhost:4318` for OTLP/HTTP
- OTLP metric export and Tokio runtime metric collection default to a 5 second interval and can be overridden with `OtlpConfig::metrics_interval`
- OTLP metric export defaults to a 5 second interval and can be overridden with `OtlpConfig::metrics_interval` between 1 second and 1 day
- Tokio runtime metric collection defaults to a 5 second interval and can be overridden with `TelemetryBuilder::with_tokio_metrics_interval(...)`

## Documentation and examples

Expand All @@ -121,10 +130,10 @@ Compilable example applications live in `examples/`:

Notes:

- Tokio metrics are OTLP-only and require both the `tokio-metrics` crate feature, the `otlp` feature, and an OTLP configuration at runtime
- Tokio metrics require the `tokio-metrics` crate feature and any installed OpenTelemetry meter provider
- Tokio metrics also require compiling the process with `RUSTFLAGS="--cfg tokio_unstable"`
- call `TelemetryGuard::shutdown().await` to gracefully stop background tasks before OTLP providers are shut down so final telemetry can flush cleanly
- dropping `TelemetryGuard` without calling `shutdown` first falls back to best-effort teardown and aborts any tasks that are still running
- dropping `TelemetryGuard` without calling `shutdown` first falls back to best-effort teardown, aborts any tasks that are still running, and then attempts provider shutdown via `tokio::task::block_in_place` on multi-thread runtimes
- the OTLP log rate limit is intentionally approximate at Unix-second boundaries under contention; it is a best-effort overload guard, not exact accounting
- OTLP rate-limit warnings use at most one helper thread at a time to avoid spawning a new thread on every overflow transition during sustained overload
- use `TelemetryBuilder::without_env_var()` when the process environment must not override the fallback stdout filter
Expand Down
2 changes: 1 addition & 1 deletion examples/greptime_otlp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use telemetry_setup::{OtlpConfig, TelemetryBuilder};
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let otlp_config: OtlpConfig = toml::from_str(include_str!("greptime_otlp.toml"))?;

let telemetry = TelemetryBuilder::new("controller")
let mut telemetry = TelemetryBuilder::new("controller")
.with_otlp_config(otlp_config)
.init()?;

Expand Down
33 changes: 0 additions & 33 deletions src/builder/feature_checks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,33 +50,6 @@ pub(crate) fn reject_tokio_metrics_without_feature(enabled: bool) -> Result<(),
Ok(())
}

/// Rejects Tokio metrics when OTLP export cannot be configured.
///
/// # Arguments
///
/// * `enabled` - Whether the builder requested Tokio runtime metric collection.
/// * `otlp_configured` - Whether the builder has an OTLP configuration.
///
/// # Returns
///
/// `Ok(())` when Tokio metrics are disabled or OTLP is configured.
///
/// # Errors
///
/// Returns [`TelemetryError::TokioMetricsRequiresOtlp`] when Tokio metrics are enabled
/// without a runtime OTLP configuration.
#[cfg(feature = "tokio-metrics")]
pub(crate) fn reject_tokio_metrics_without_otlp_config(
enabled: bool,
otlp_configured: bool,
) -> Result<(), TelemetryError> {
if enabled && !otlp_configured {
return Err(TelemetryError::TokioMetricsRequiresOtlp);
}

Ok(())
}

#[cfg(test)]
mod tests {
use super::{reject_journald_without_feature, reject_tokio_metrics_without_feature};
Expand All @@ -86,10 +59,4 @@ mod tests {
assert!(reject_journald_without_feature(false).is_ok());
assert!(reject_tokio_metrics_without_feature(false).is_ok());
}

#[cfg(feature = "tokio-metrics")]
#[test]
fn tokio_metrics_with_otlp_config_is_accepted() {
assert!(super::reject_tokio_metrics_without_otlp_config(true, true).is_ok());
}
}
34 changes: 27 additions & 7 deletions src/builder/filter.rs
Original file line number Diff line number Diff line change
@@ -1,27 +1,47 @@
// SPDX-License-Identifier: MIT

use std::sync::Arc;

pub(crate) type EnvLookup = Arc<dyn Fn(&str) -> Option<String> + Send + Sync>;

/// Resolves the local filter from the configured environment variable or fallback string.
///
/// # Arguments
///
/// * `env_var_name` - Optional environment variable name to query first.
/// * `env_lookup` - Lookup callback used to read environment values.
/// * `stdout_filter` - Optional explicit fallback filter.
///
/// # Returns
///
/// The environment-provided filter, explicit fallback filter, or `info` default.
pub(crate) fn resolve_stdout_filter(
env_var_name: &Option<String>,
env_lookup: &EnvLookup,
stdout_filter: &Option<String>,
) -> String {
if let Some(env_var_name) = env_var_name
&& let Some(filter) = env_lookup(env_var_name)
&& let Ok(filter) = std::env::var(env_var_name)
{
return filter;
}

stdout_filter.clone().unwrap_or_else(|| "info".to_string())
}

/// Resolves the local filter using a caller-provided lookup function.
///
/// # Arguments
///
/// * `env_var_name` - Optional environment variable name to query first.
/// * `lookup` - Function used to resolve environment values for tests.
/// * `stdout_filter` - Optional explicit fallback filter.
///
/// # Returns
///
/// The lookup-provided filter, explicit fallback filter, or `info` default.
#[cfg(test)]
pub(crate) fn resolve_stdout_filter_with_lookup(
env_var_name: &Option<String>,
lookup: impl Fn(&str) -> Option<String>,
stdout_filter: &Option<String>,
) -> String {
if let Some(env_var_name) = env_var_name
&& let Some(filter) = lookup(env_var_name)
{
return filter;
}
Expand Down
Loading
Loading