Integrate Kubeflow SDK with OpenTelemetry

[andreyvelich]

What you would like to be added?

We would like to explore opportunities to integrate Kubeflow SDK with Open Telemetry for better observability and telemetry of AI Workloads.
For example, we can take a look at OTel and GenAI integrations: https://opentelemetry.io/blog/2024/otel-generative-ai/

This was one of the user asks during Kubeflow Summit at KubeCon + CloudNativeCon 2025 NA.

cc @kubeflow/kubeflow-sdk-team @kubeflow/kubeflow-trainer-team @Fiona-Waters @austinlparker @alolita

Why is this needed?

Improve observability experience for Kubeflow SDK users.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[kramaranya]

/assign

[dhanishaphadate]

/assign

Created project proposal for this kubeflow/website#4294 (comment)

[1Ayush-Petwal]

Hi @dhanishaphadate, @kramaranya, @jaiakash

I had read the OTel and GenAI integrations: https://opentelemetry.io/blog/2024/otel-generative-ai/ & the sdk docs https://sdk.kubeflow.org/en/latest/index.html. Using those I developed a POC to demonstrate my technical approach for this integration:
(https://github.com/1Ayush-Petwal/SDK_OTEL_POC)

The PoC demonstrates the following core features:

1. Library-Native Instrumentation: Manual trace/metric capture for TrainerClient and PipelinesClient using the opentelemetry-api to ensure zero overhead for non-users.

2. Distributed Tracing: Implementation of W3C context propagation (traceparent) to ensure traces survive the hand-off from the SDK to remote Kubernetes executors.

3. AI/ML Observability: Integration of GenAI semantic conventions to track token usage, model metadata, and operation durations.

4. Configurability: A functional setup including an OTel Collector, Jaeger, and Prometheus with clear documentation and examples.

I am ready to align this with the existing roadmap. As soon as I am assigned, I will proceed with the detailed implementation planning and formal design for the main repository.

[jaiakash]

Hi @1Ayush-Petwal I will review this over weekend.

[Mahimasai]

Hi @kramaranya @dhanishaphadate @jaiakash,

I’ve been going through the Kubeflow SDK documentation and reviewing the TrainerClient and PipelinesClient implementations to better understand where OpenTelemetry instrumentation could fit naturally.

From my current understanding, public SDK entry points like train() and run_pipeline() seem like good candidates for top-level spans, while internal API interactions (e.g., CRD creation and status polling) could form child spans. I’m also exploring whether introducing a shared instrumentation utility in kubeflow/common would help avoid duplication across clients.

I’ve started experimenting locally with manual OpenTelemetry instrumentation in Python to better understand span hierarchy and context propagation.

Before diving deeper, I’d appreciate guidance on preferred instrumentation granularity — should tracing primarily focus on SDK method boundaries, or also include lower-level Kubernetes API interactions?

Looking forward to contributing and refining this approach.

[prabindersinghh]

Hi @kramaranya, @dhanishaphadate, @kramaranya -

I’ve gone through the issue discussion, the GSoC proposal PR in kubeflow/website (#4294), and the current SDK documentation.
Given that the SDK’s core promise is a Pythonic abstraction over Kubernetes, I believe any OpenTelemetry integration should:

• Remain fully optional and zero-overhead for non-users
• Avoid leaking Kubernetes-level complexity into user-facing APIs
• Follow a consistent span model across clients

To start in a focused way, I propose beginning with a minimal reference implementation in TrainerClient:

1. Instrument the top-level train() entry point
2. Add child spans for CRD creation, submission, and status polling

Establish a consistent naming pattern such as:

1. kubeflow.sdk.trainer.train
2. kubeflow.sdk.trainer.submit
3. kubeflow.sdk.trainer.poll

This would help validate:

1. Span hierarchy design
2. Context propagation behavior
3. Attribute conventions

Before proceeding, I’d appreciate clarification on:

• Whether there is any active SDK branch or ongoing OTel integration I should align with
• Whether instrumentation should live within each client or be centralized in kubeflow/common

My plan is to validate this pattern with TrainerClient first, then extend consistently across other clients.
Looking forward to guidance on preferred direction!

[priyansh-saxena1]

Great to join the call yesterday, the roadmap discussion was really helpful context.
i have started reading through the Trainerclient source and have a quick question about the data collection discussion #170 would otel metrics like kubeflow.sdk.trainjob.create.duration (histogram) and kubeflow.sdk.trainjob.create.count (counter) cover the kind of usage data the team is looking to collect? Or is the intent more around anonymous telemetry similar to what Ray and Mlflow do (as referenced in the meeting notes)
trying to understand the right scope for the Otel integration before next week meet.

[1Ayush-Petwal]

Hi @1Ayush-Petwal I will review this over weekend.

Hi @jaiakash, @dhanishaphadate, @kramaranya — I just submitted a small fix (#348) while further exploring the codebase. I missed last week's call, but I caught the recording and got the gist of the 2026 SDK Roadmap and the importance of OpenTelemetry integration within it.

Let me know if there's anything you'd like me to revise in the PoC. Does the direction I’m presenting align with the team's vision for this project?