Skip to content

fix: enable DB instrumentation metrics via Elysia OTel plugin#452

Open
turisanapo wants to merge 18 commits into
mainfrom
fix/otel-db-instrumentation
Open

fix: enable DB instrumentation metrics via Elysia OTel plugin#452
turisanapo wants to merge 18 commits into
mainfrom
fix/otel-db-instrumentation

Conversation

@turisanapo
Copy link
Copy Markdown
Contributor

@turisanapo turisanapo commented Apr 30, 2026

Summary

Fixes #449 — DB operations were invisible in telemetry (no db_* metric tables, no @opentelemetry/instrumentation-pg spans).

  • Move PgInstrumentation and BunSqlInstrumentation from a standalone registerInstrumentations() call into the instrumentations field of getOtelConfig()
  • Defer Prisma client initialization so @opentelemetry/instrumentation-pg patches the pg driver before the first connection
  • Simplify Greptime client setup and deduplicate GREPTIME_HOST resolution
  • Defer @prisma/adapter-pg import to avoid loading pg before OTel patches it

Known limitation

The fix works locally with bun dev, but does not work when bundling with bun build.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Refactor
    • Enhanced system initialization sequence for database connections to ensure proper startup ordering and resource management.
    • Optimized OpenTelemetry instrumentation configuration to improve observability and system monitoring capabilities.
    • Improved database client initialization to better coordinate with system instrumentation setup requirements.

Review Change Stack

…rumentation

Closes #449

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 30, 2026

Warning

Rate limit exceeded

@turisanapo has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 8 minutes and 3 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ccc3f9d9-6d41-4e19-9cba-c935412c9556

📥 Commits

Reviewing files that changed from the base of the PR and between ce8656b and 0445637.

📒 Files selected for processing (2)
  • packages/shared-api/db/postgres.ts
  • packages/shared-api/lib/otel.ts
📝 Walkthrough

Walkthrough

This PR refactors database initialization and OpenTelemetry instrumentation coordination. GREPTIME_HOST becomes a shared constant, OTel instrumentations move from import-time registration into getOtelConfig, the Prisma adapter switches to runtime instantiation, and Prisma client creation defers to first request via getPrisma() to ensure OTel registers before the database driver loads.

Changes

OTel instrumentation coordination and eager-to-lazy database initialization

Layer / File(s) Summary
Shared database environment and Greptime client setup
packages/shared-api/db/greptime.ts
GREPTIME_HOST is exported as a module-level constant resolved from secrets with a "localhost" fallback, and greptimeSqlClient is eagerly instantiated using the synchronized host value instead of lazy helpers.
OpenTelemetry instrumentation refactoring with synchronized OTLP endpoint
packages/shared-api/lib/otel.ts
Import-time instrumentation registration is removed, GREPTIME_OTLP_ENDPOINT becomes synchronous using the shared GREPTIME_HOST constant, and PgInstrumentation and BunSqlInstrumentation are moved from a module-level registerInstrumentations call into the instrumentations array returned by getOtelConfig.
Prisma adapter runtime instantiation
packages/shared-api/db/postgres.ts
PrismaPg instantiation switches from static import to runtime require() with dynamic module access, and the explicit return type annotation is removed from createPrismaAdapter.
Lazy Prisma client initialization for instrumentation coordination
apps/api/src/db/prisma.ts
getPrisma() lazy accessor defers Prisma client instantiation until first use to ensure OTel instrumentation registers before the driver loads; extension wiring and provider-config queries are updated to use the lazy client.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

  • #449: This PR directly addresses the production issue where DB spans from PgInstrumentation and BunSqlInstrumentation are not being captured by moving instrumentation setup into getOtelConfig and ensuring OTel registers before Prisma driver initialization.

Possibly related PRs

  • 8monkey-ai/hebo-platform#388: Both PRs modify packages/shared-api/lib/otel.ts to change how OpenTelemetry database instrumentation is wired up (retrieved PR switches Prisma instrumentation to @opentelemetry/instrumentation-pg, while this PR refactors the registration approach by removing top-level registerInstrumentations and supplying instrumentations via getOtelConfig).
  • 8monkey-ai/hebo-platform#378: Both PRs modify packages/shared-api/lib/otel.ts to include/configure BunSqlInstrumentation for OpenTelemetry query tracing, but they implement it via different registration approaches.
  • 8monkey-ai/hebo-platform#169: Both PRs change packages/shared-api/lib/otel.ts and Prisma initialization (apps/api/src/db/prisma.ts) to coordinate OpenTelemetry/Prisma instrumentation timing.

Suggested reviewers

  • heiwen

Poem

🐰 Instrumentation hops before the driver wakes,
GREPTIME_HOST shared, no async secrets it takes,
PgSpans and BunSQL now synchronous flow,
Lazy Prisma blooms when requests say go!
From DB to GreptimeDB, telemetry glows.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main change: refactoring OpenTelemetry setup to integrate database instrumentation into the Elysia OTel plugin, which is the core objective of this PR.
Linked Issues check ✅ Passed The PR successfully addresses all code requirements from #449: instrumentations moved into getOtelConfig(), Prisma client deferred for early OTel patching, adapter imports deferred, Greptime setup simplified, and client initialization coordinated to enable DB instrumentation.
Out of Scope Changes check ✅ Passed All changes are directly scoped to resolving the DB instrumentation visibility issue: OTel config refactoring, Prisma/Greptime client lazy initialization, and adapter import deferral—no extraneous modifications detected.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/otel-db-instrumentation

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@turisanapo turisanapo marked this pull request as draft April 30, 2026 02:49
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/shared-api/lib/otel.ts (1)

90-100: Debug logging would aid troubleshooting when pg manual patching fails silently.

The workaround is well-documented and addresses a real Bun limitation (GitHub issue #3775). The version is already pinned to 0.66.0, which mitigates the risk of API breakage. However, the silent catch {} block makes it hard to debug if the patching fails. Consider adding a debug log:

   try {
     // oxlint-disable no-unsafe-assignment no-unsafe-call no-unsafe-member-access
     const { createRequire } = require("module");
     const pg = createRequire(require.resolve("@prisma/adapter-pg"))("pg");
     // `@ts-expect-error` _patchPgClient is a private method on PgInstrumentation
     pgInstrumentation._patchPgClient(pg.Client);
     // oxlint-enable no-unsafe-assignment no-unsafe-call no-unsafe-member-access
-  } catch {}
+  } catch (err) {
+    if (!IS_PRODUCTION) console.debug("[otel] pg manual patching failed:", err);
+  }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/shared-api/lib/otel.ts` around lines 90 - 100, The empty catch
around the manual pg patching hides failures; update the try/catch in the otel
patch block so the catch logs debug-level details including the caught error and
context (e.g., that createRequire(require.resolve("@prisma/adapter-pg"))("pg")
and pgInstrumentation._patchPgClient(pg.Client) failed), using the existing
logger (or console.debug if none) so failures to patch pg.Client are visible for
troubleshooting.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@packages/shared-api/lib/otel.ts`:
- Around line 90-100: The empty catch around the manual pg patching hides
failures; update the try/catch in the otel patch block so the catch logs
debug-level details including the caught error and context (e.g., that
createRequire(require.resolve("@prisma/adapter-pg"))("pg") and
pgInstrumentation._patchPgClient(pg.Client) failed), using the existing logger
(or console.debug if none) so failures to patch pg.Client are visible for
troubleshooting.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8e845411-5f16-4cc9-b3d8-046d88cd626e

📥 Commits

Reviewing files that changed from the base of the PR and between e727c36 and c667f24.

⛔ Files ignored due to path filters (1)
  • bun.lock is excluded by !**/*.lock
📒 Files selected for processing (1)
  • packages/shared-api/lib/otel.ts

@turisanapo turisanapo marked this pull request as ready for review April 30, 2026 02:58
The bun:sql client was created at module-eval time, before
BunSqlInstrumentation could wrap the SQL constructor. Make it lazy
so the first access goes through the instrumented constructor.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@turisanapo turisanapo force-pushed the fix/otel-db-instrumentation branch from 26b0c42 to ad05bd3 Compare April 30, 2026 03:14
@turisanapo turisanapo force-pushed the fix/otel-db-instrumentation branch from 20cc603 to ca30fc1 Compare April 30, 2026 03:20
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@turisanapo turisanapo force-pushed the fix/otel-db-instrumentation branch from 1dee971 to 1501084 Compare April 30, 2026 03:27
@turisanapo turisanapo force-pushed the fix/otel-db-instrumentation branch from 62caa11 to a61932d Compare April 30, 2026 03:39
@turisanapo turisanapo requested a review from heiwen April 30, 2026 03:56
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/shared-api/lib/otel.ts (1)

85-100: Empty catch block pattern is intentional for resilience—logging suggestion should align with structured logger pattern.

The empty catch {} at line 100 mirrors similar graceful fallbacks elsewhere in the codebase (e.g., secret.ts, postgres.ts), confirming this is intentional. If added, observability logging should use the codebase's structured logger (already imported in this file and used consistently with logger.info/error/warn) rather than console.debug.

Additionally, _patchPgClient is a private API that may change between @opentelemetry/instrumentation-pg versions, though this is already mitigated by the @ts-expect-error comment.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/shared-api/lib/otel.ts` around lines 85 - 100, The empty catch
swallowing errors when attempting to patch pg.Client should emit a structured
log via the existing logger instead of doing nothing; update the try/catch
around the createRequire + pgInstrumentation._patchPgClient call to catch the
error (e.g., catch (err)) and call logger.debug or logger.warn with a short
message like "failed to patch pg.Client for Bun: falling back" and include the
caught error object, referencing the pgInstrumentation,
createRequire(require.resolve("@prisma/adapter-pg"))("pg"), and _patchPgClient
call so maintainers can locate the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@packages/shared-api/lib/otel.ts`:
- Around line 85-100: The empty catch swallowing errors when attempting to patch
pg.Client should emit a structured log via the existing logger instead of doing
nothing; update the try/catch around the createRequire +
pgInstrumentation._patchPgClient call to catch the error (e.g., catch (err)) and
call logger.debug or logger.warn with a short message like "failed to patch
pg.Client for Bun: falling back" and include the caught error object,
referencing the pgInstrumentation,
createRequire(require.resolve("@prisma/adapter-pg"))("pg"), and _patchPgClient
call so maintainers can locate the change.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d858a7dd-35e9-4aaf-b6ca-9db395cbd51e

📥 Commits

Reviewing files that changed from the base of the PR and between c667f24 and d853188.

📒 Files selected for processing (3)
  • apps/api/src/middlewares/greptime.ts
  • packages/shared-api/db/greptime.ts
  • packages/shared-api/lib/otel.ts

Comment thread apps/api/src/middlewares/greptime.ts Outdated
.as("scoped");

export type GreptimeDb = BunSqlClient;
export type GreptimeDb = Bun.SQL;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't know this, its completely encapsulated by the shared-api layer. Please revert.

Comment thread packages/shared-api/db/greptime.ts Outdated
url,
let _client: Bun.SQL;

/** Lazily created so BunSqlInstrumentation wraps `bun:sql` before the first `new SQL()` call. */
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous createBunSqlClient was lazy as well, what exactly is the change here?

Comment thread packages/shared-api/lib/otel.ts Outdated
Comment on lines +105 to +111
pgInstrumentation,
new BunSqlInstrumentation({
requireParentSpan: true,
ignoreConnectionSpans: true,
// FUTURE: set to true to avoid leaking sensitive information
maskStatement: false,
}),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Align coding pattern across pg and bun instrumentation.

Comment thread packages/shared-api/lib/otel.ts Outdated
Comment on lines +90 to +100
// Pg is already loaded via @prisma/adapter-pg before OTel's module-load
// hooks can patch it. Patch pg.Client directly as a workaround, while still
// passing the instrumentation to NodeSDK so providers are configured correctly.
try {
// oxlint-disable no-unsafe-assignment no-unsafe-call no-unsafe-member-access
const { createRequire } = require("module");
const pg = createRequire(require.resolve("@prisma/adapter-pg"))("pg");
// @ts-expect-error _patchPgClient is a private method on PgInstrumentation
pgInstrumentation._patchPgClient(pg.Client);
// oxlint-enable no-unsafe-assignment no-unsafe-call no-unsafe-member-access
} catch {}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a big workaround / very hacky. Is this either (a) something that needs to be fixed upstream in @prisma/adapter-pg or otel-instrumentation-otel or (b) something where the initialisation needs to follow a similar lazy pattern like we do with our greptime client?

turisanapo and others added 2 commits May 4, 2026 17:36
- pg: opaque require prevents Bun bundler from inlining @prisma/adapter-pg,
  so OTel RITM hooks intercept pg at runtime
- bun:sql: lazy getGreptimeSqlClient() defers new SQL() to after
  BunSqlInstrumentation wraps the constructor (Proxy-based, per-instance)
- otel: instrumentations passed to NodeSDK via getOtelConfig() so
  sdk.start() calls setMeterProvider(), fixing the NoopMeter issue

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@turisanapo turisanapo marked this pull request as draft May 5, 2026 03:42
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/shared-api/lib/otel.ts (1)

95-96: 💤 Low value

Consider enabling maskStatement before production traffic carries PII.

The FUTURE comment acknowledges this, but maskStatement: false will log raw SQL statements including any embedded literals (user IDs, emails, etc.) to your telemetry backend. If PII flows through these queries, enable masking or ensure Greptime access is appropriately restricted.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shared-api/lib/otel.ts` around lines 95 - 96, The telemetry
configuration currently sets maskStatement: false which will log raw SQL
(possibly containing PII); update the configuration in
packages/shared-api/lib/otel.ts to enable masking (set maskStatement to true) or
make it configurable via an environment flag (e.g., OTEL_MASK_STATEMENT) and
ensure the code reads that flag to default to true in production; locate the
maskStatement property in the DB/tracing config object in this file and change
it to true or wire it to the env toggle accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/shared-api/lib/otel.ts`:
- Around line 95-96: The telemetry configuration currently sets maskStatement:
false which will log raw SQL (possibly containing PII); update the configuration
in packages/shared-api/lib/otel.ts to enable masking (set maskStatement to true)
or make it configurable via an environment flag (e.g., OTEL_MASK_STATEMENT) and
ensure the code reads that flag to default to true in production; locate the
maskStatement property in the DB/tracing config object in this file and change
it to true or wire it to the env toggle accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 97c511ad-6639-409d-9c18-4fbc7d0fc528

📥 Commits

Reviewing files that changed from the base of the PR and between d853188 and 566fac4.

📒 Files selected for processing (4)
  • apps/api/src/middlewares/greptime.ts
  • packages/shared-api/db/greptime.ts
  • packages/shared-api/db/postgres.ts
  • packages/shared-api/lib/otel.ts

@turisanapo turisanapo force-pushed the fix/otel-db-instrumentation branch 2 times, most recently from 6a4a950 to 8696ed3 Compare May 13, 2026 02:31
@turisanapo turisanapo changed the title fix: enable DB spans and metrics in telemetry fix: enable DB instrumentation metrics via Elysia OTel plugin May 14, 2026
Deferring is unnecessary for metrics — the fix is solely the
instrumentations field in getOtelConfig(). Reverts to eager client init.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@turisanapo turisanapo marked this pull request as ready for review May 14, 2026 04:07
@turisanapo turisanapo requested a review from heiwen May 14, 2026 04:07
@turisanapo
Copy link
Copy Markdown
Contributor Author

@heiwen, the solution works in dev, and it correctly generates spans and metrics for all DB operations. However, the solution does not work when bundling with bun run build.

Do you have any suggestions on how to continue working on the fix?

The changes include a simplification of how we work with GREPTIME_HOST, unnecessary for the fix. Let me know if you prefer it to be removed.

turisanapo and others added 2 commits May 14, 2026 12:34
…l.ts

The only change needed for metrics is passing instrumentations via
getOtelConfig() instead of registerInstrumentations(). No changes to the
Greptime client or middleware are required.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DB operations only partially visible in telemetry — no DB spans or metric tables in production

2 participants