From 1c5e4265ed1482b61a68444a3200a30660b81984 Mon Sep 17 00:00:00 2001 From: Alexander Yevsyukov Date: Sat, 27 Jun 2026 21:16:42 +0100 Subject: [PATCH 01/20] Add the report from Claude Chat and implementation plan from Claude Ultracode --- .agents/tasks/otel-backend-implementation.md | 239 +++++++ .agents/tasks/otel-backend-report.md | 642 +++++++++++++++++++ 2 files changed, 881 insertions(+) create mode 100644 .agents/tasks/otel-backend-implementation.md create mode 100644 .agents/tasks/otel-backend-report.md diff --git a/.agents/tasks/otel-backend-implementation.md b/.agents/tasks/otel-backend-implementation.md new file mode 100644 index 00000000..9f388370 --- /dev/null +++ b/.agents/tasks/otel-backend-implementation.md @@ -0,0 +1,239 @@ +--- +slug: otel-backend-implementation +branch: otel +owner: claude +status: draft +started: 2026-06-27 +related-memories: [] +--- + +## Goal + +Ship an OpenTelemetry logger backend for Spine Logging that maps `LogData` to OTel +log records (`Logger.emit(...)`) and, on top of the same call path, a log-based +*events* surface. Success = an app can route Spine logs to an OTel pipeline by +adding one backend artifact and pointing it at an `OpenTelemetry` instance; the +record carries severity, message, log-site, throwable, Spine scope/log-site +metadata as attributes, and active-span trace correlation; all proven with tests. + +The companion research spec is [`otel-backend-report.md`](otel-backend-report.md) +(referred to below as "the report"). **This plan corrects several factual errors +in the report** (see *Context → Corrections*); where the two disagree, this plan +wins. + +## Context + +The plan is grounded in three sources, all read on 2026-06-27: + +1. **The Spine SPI** (verified `confirmed` against source) — the report describes it + accurately: + - [`BackendFactory.kt:81`](logging/src/commonMain/kotlin/io/spine/logging/backend/BackendFactory.kt) — `abstract fun create(loggingClass: String): LoggerBackend` + - [`LoggerBackend.kt`](logging/src/commonMain/kotlin/io/spine/logging/backend/LoggerBackend.kt) — `loggerName: String?` (65), `isLoggable(Level)` (72), `log(LogData)` (80), `handleError(RuntimeException, LogData)` (118) + - [`LogData.kt`](logging/src/commonMain/kotlin/io/spine/logging/backend/LogData.kt) — `level`, `timestampNanos: Long`, `loggerName`, `logSite`, `metadata`, `wasForced()`, `literalArgument` + - [`Level.kt:56`](logging/src/commonMain/kotlin/io/spine/logging/Level.kt) — `data class Level(name, value)`; FATAL=2000, ERROR/SEVERE=1000, WARNING=900, INFO=800, CONFIG=700, DEBUG/FINE=500, FINER/TRACE=400, FINEST=300 + - [`MetadataProcessor.kt`](logging/src/commonMain/kotlin/io/spine/logging/backend/MetadataProcessor.kt) — `forScopeAndLogSite(scope, logged)` (83), `process(handler, ctx)` (133), `getSingleValue(key)` (151), `keySet()` (168) + - [`MetadataHandler.kt`](logging/src/commonMain/kotlin/io/spine/logging/backend/MetadataHandler.kt) — `handle(key, value, ctx)` (62) and the repeated-key hook **`handleRepeated(key, values: Iterator, ctx)`** (78); build via `MetadataHandler.builder()` + - [`SimpleMessageFormatter.kt:167`](logging/src/commonMain/kotlin/io/spine/logging/backend/SimpleMessageFormatter.kt) — `getLiteralLogMessage(logData): String` (message text without the `[CONTEXT …]` suffix) + - [`Platform.kt:182`](logging/src/commonMain/kotlin/io/spine/logging/backend/Platform.kt) — `getInjectedMetadata(): Metadata` + - [`LogContext.kt:585`](logging/src/commonMain/kotlin/io/spine/logging/LogContext.kt) — `Key.LOG_CAUSE: MetadataKey` + +2. **The reference backends to mirror**: + - [`log4j2-backend`](backends/log4j2-backend) — the mapping template ([`LogEvents.kt`](backends/log4j2-backend/src/main/kotlin/io/spine/logging/backend/log4j2/LogEvents.kt): `MetadataProcessor.forScopeAndLogSite` → format → `getSingleValue(LOG_CAUSE)` → `MetadataHandler.builder()`, with `Tags` special-cased and `ValueQueue` for repeated keys). + - [`probe-backend`](backends/probe-backend) — the `@AutoService` + KSP registration pattern ([`build.gradle.kts`](backends/probe-backend/build.gradle.kts): `plugins { ksp }`, `implementation(AutoService.annotations)`, `ksp(AutoServiceKsp.processor)`) and the memoizing test backend. + - [`logging/build.gradle.kts`](logging/build.gradle.kts) — the **`kmp-module`** shape: `kotlin { sourceSets { commonMain {…}; jvmMain { runtimeOnly(project(":jvm-default-platform")) }; jvmTest {…} } }`, JVM target only. + +3. **The proven OTel-Kotlin precedent — the `core-jvm` repo's `server-otel` module** + (sibling checkout at `/Users/sanders/Projects/Spine/core-jvm`). It already uses + the native Kotlin SDK successfully and gives us copy-ready artifacts: + - `core-jvm/buildSrc/src/main/kotlin/io/spine/dependency/lib/OpenTelemetryKotlin.kt` — a `Dependency()` object pinning **0.4.0**, group `io.opentelemetry.kotlin`, exposing `api`, `noop`, `core`, `implementation`, `compat`. **Copy it verbatim into this repo's `buildSrc/.../lib/`.** + - `core-jvm/server-otel/build.gradle.kts` — production depends on **`api` only**; tests add `core` + `implementation` (the native SDK). + - `core-jvm/.../given/TestOtel.kt` — the SDK is built with `createOpenTelemetry { tracerProvider { export { processor } } }` and a hand-rolled recording processor. **The logs test path mirrors this with a recording `LogRecordProcessor` — no in-memory-exporter artifact required.** + + The actual **0.4.0 Logs API** was read from the api-discovery cache + (`io.opentelemetry.kotlin:api:0.4.0`) and matches the report's skeletons: + `Logger.emit(body, eventName, timestamp: Long?, observedTimestamp, context, severityNumber, severityText, exception, attributes)`, + `Logger.enabled(context, severityNumber, eventName)`, + `LoggerProvider.getLogger(name, version, schemaUrl, attributes)`, + `OpenTelemetry.loggerProvider`, and the `SeverityNumber` enum (1–24; `DEBUG4=8`). + +### Corrections to the report (verified 2026-06-27) + +| # | Report says | Reality | Consequence | +|---|-------------|---------|-------------| +| C1 | `opentelemetry-kotlin` **0.5.0** | Latest is **0.4.0** (2026-05-20); 0.5.0 does not exist | Pin `0.4.0` (matches `core-jvm`). | +| C2 | Version catalog `gradle/libs.versions.toml` | Repo uses **buildSrc Kotlin objects** | Copy `core-jvm`'s `OpenTelemetryKotlin` object into `buildSrc/.../lib/`, not a TOML entry. | +| C3 | Service file in `jvmMain/resources`, hand-written | Repo standard is **`@AutoService` + KSP** (probe-backend) | Annotate the JVM factory `@AutoService(BackendFactory::class)`; KSP generates the service file. | +| C4 | Test with `exporters-in-memory` (Kotlin); compat bridge for the SDK | Kotlin in-memory exporter artifact unconfirmed; `core-jvm` uses the **native Kotlin SDK** + a hand-rolled recording processor | Use the **native SDK** (`core`+`implementation`); tests capture records with a recording `LogRecordProcessor`. No compat, no in-memory artifact. | +| C5 | A `std` backend exists | Only `log4j2`, `jul`, `probe` exist | Ignore; mirror `log4j2`. | +| C6 | `OtelBackendSettings.use(...)` reuses an existing pattern | No backend has programmatic injection today; JVM precedent is the `spine.logging.backend_factory` system-property override ([`DefaultPlatform.kt`](platforms/jvm-default-platform/src/main/kotlin/io/spine/logging/backend/system/DefaultPlatform.kt)) | `OtelBackendSettings` is a **new** convention; model its `@Volatile`/no-op-default shape on `DefaultPlatform`'s loader and document it. | + +Two API refinements (vs the report's prose): +- `AttributesMutator` has **no `Int`/`Float` setters** — only `setLongAttribute`/`setDoubleAttribute` (+ `*List*`, `setByteArrayAttribute`, `setAnyValueAttribute`). The attribute handler must widen `Int`/`Short`/`Byte` → `Long` and `Float` → `Double`. +- `Logger.emit(timestamp: Long?)` — the param exists, but **the unit (epoch nanos vs millis) is not documented in the API**; it is resolved by the SDK. Confirm against `:implementation` 0.4.0 in Phase 0 before wiring `data.timestampNanos`. + +### Decisions (resolved) + +| Topic | Decision | +|-------|----------| +| **JVM SDK strategy** | **Native Kotlin SDK** (`core`+`implementation`) from the start — proven in `core-jvm/server-otel`. No compat bridge. | +| **Module platform** | **`kmp-module`** with the JVM target only (mirrors core `logging`). Mapping in `commonMain` (only touches OTel `:api` + Spine commonMain SPI); JVM-only registration in `jvmMain`. Aims at KMP up front to cut future migration, even though only JVM is wired today. | +| **Registration** | **`@AutoService(BackendFactory::class)` + KSP** (`kspJvm`), as probe-backend does. | +| **Events / OTEP 4430** | Events emit through the Logs API (`emit(eventName = …)`); never `Span.AddEvent`. | +| **Body vs attributes** | Body = `getLiteralLogMessage(...)` (no `[CONTEXT]` suffix); metadata → attributes (no double-encoding). | +| **CONFIG severity** | `DEBUG4` (judgment call, documented in KDoc) — open to `INFO` if preferred. | + +## Architecture + +``` +backends/otel-backend/ ← kmp-module, JVM target only (for now) + src/commonMain/kotlin/io/spine/logging/backend/otel/ + OtelLoggerBackend.kt ← LoggerBackend; LogData → Logger.emit(...) [api only] + OtelBackendSettings.kt ← @Volatile holder; default NoopOpenTelemetry; use(otel) + SeverityMapping.kt ← Level.toSeverityNumber() + AttributeMapping.kt ← MetadataHandler + LogSite → code.* semconv + src/jvmMain/kotlin/io/spine/logging/backend/otel/ + OtelBackendFactory.kt ← @AutoService(BackendFactory::class); resolves OpenTelemetry from settings + src/jvmTest/kotlin/... ← native SDK + recording LogRecordProcessor + Kotest/JUnit5 specs + +backends/otel-backend-bootstrap/ ← OPTIONAL, Phase 2 (jvmMain SDK init, turnkey wiring) +``` + +- `commonMain` depends on `OpenTelemetryKotlin.api` + `.noop` + `project(":logging")`. +- `jvmMain` adds `AutoService.annotations` (+ `kspJvm(AutoServiceKsp.processor)`), and + the factory carries `@AutoService` directly — it is a no-arg `class`, so unlike + probe-backend's `object` it needs no adapter shim. +- `jvmTest` adds `OpenTelemetryKotlin.core` + `.implementation`, `logging-testlib`, and + `runtimeOnly(project(":jvm-default-platform"))` so `DefaultPlatform` discovers the + `@AutoService` factory end-to-end. +- **Build wrinkle to validate (Phase 0):** KSP on a `kmp-module` uses `kspJvm(...)`, + not the plain `ksp(...)` probe-backend uses on its `jvm-module`. If KSP-on-KMP + misbehaves with the `kmp-module` convention plugin, fall back to a hand-written + `src/jvmMain/resources/META-INF/services/io.spine.logging.backend.BackendFactory`. + +## Open decisions + +1. **CONFIG severity** — `DEBUG4` (default) vs `INFO`. Cosmetic; documented either way. +2. **Bootstrap module SDK config surface** (Phase 2) — env-var driven vs explicit DSL. + Defer until Phase 1 lands. + +## Plan + +### Phase 0 — spike: native SDK end-to-end on JVM (de-risk) +- [ ] Copy `core-jvm`'s `OpenTelemetryKotlin.kt` into + `buildSrc/src/main/kotlin/io/spine/dependency/lib/`; confirm the shape against + the `dependency-audit` skill. Add `AutoService`/`AutoServiceKsp` usage check — + both already exist ([`Auto.kt`](buildSrc/src/main/kotlin/io/spine/dependency/lib/Auto.kt), [`Ksp.kt`](buildSrc/src/main/kotlin/io/spine/dependency/build/Ksp.kt)). +- [ ] Register the module: add `"otel-backend"` to `includeBackend(...)` in + [`settings.gradle.kts`](settings.gradle.kts). +- [ ] Create `backends/otel-backend/build.gradle.kts`: `plugins { kmp-module; ksp }`; + `commonMain` → `api(OpenTelemetryKotlin.api)` + `noop` + `project(":logging")`; + `jvmMain` → `AutoService.annotations`; `kspJvm(AutoServiceKsp.processor)`; + `jvmTest` → `OpenTelemetryKotlin.core` + `.implementation` + `logging-testlib` + + `runtimeOnly(project(":jvm-default-platform"))`. Add `@OptIn(ExperimentalApi::class)`. +- [ ] **Verify the logs SDK DSL with `api-discovery` on `:implementation` 0.4.0** — + confirm the `createOpenTelemetry { loggerProvider { export { … } } }` shape and the + `LogRecordProcessor`/exporter type (the api cache only has `:api`, not the SDK). + Also confirm the **`emit(timestamp)` unit** (epoch nanos vs millis). +- [ ] Build a `RecordingLogRecordProcessor` (mirror `core-jvm`'s `RecordingSpanProcessor`), + wire `createOpenTelemetry { loggerProvider { export { it } } }`, call + `OtelBackendSettings.use(...)`, log once through Spine, assert the record round-trips + (severity + body). This throwaway proof becomes the first real test. + +### Phase 1 — production backend +- [ ] `commonMain/OtelBackendSettings.kt`: `@Volatile private var instance = NoopOpenTelemetry`; + `fun use(openTelemetry: OpenTelemetry)`; `internal fun current()`. No-op default, + drop-before-`use`. KDoc it as a new convention (correction C6). +- [ ] `commonMain/SeverityMapping.kt`: numeric-threshold `Level.toSeverityNumber()` + (report §5.1) → `FATAL/ERROR/WARN/INFO/DEBUG4(CONFIG)/DEBUG/TRACE2/TRACE`. +- [ ] `commonMain/OtelLoggerBackend.kt`: + - `isLoggable(level)` → `logger.enabled(severityNumber = level.toSeverityNumber())`. + - `log(data)` → `MetadataProcessor.forScopeAndLogSite(Platform.getInjectedMetadata(), data.metadata)`; + `body = getLiteralLogMessage(data)`; `severityNumber`/`severityText = data.level.name`; + `exception = metadata.getSingleValue(LogContext.Key.LOG_CAUSE)`; + `timestamp = data.timestampNanos` (**unit confirmed in Phase 0**); + `context = null` (⇒ implicit/active context, per the `emit` KDoc); + `attributes = { applyAttributes(data, metadata) }`. + - `handleError(error, badData)` → emit an ERROR record carrying `badData.toString()`. +- [ ] `commonMain/AttributeMapping.kt`: a `MetadataHandler` via + `MetadataHandler.builder()`: + - LogSite → `code.namespace`/`code.function`/`code.filepath`/`code.lineno`. + - `handle(...)`: skip `LOG_CAUSE`; namespace keys `spine.