Skip to content

feat: add complete MLflow UC tracing to all agent templates - #260

Open
adamgurary wants to merge 39 commits into
databricks:mainfrom
adamgurary:feat/mlflow-uc-agent-tracing
Open

feat: add complete MLflow UC tracing to all agent templates#260
adamgurary wants to merge 39 commits into
databricks:mainfrom
adamgurary:feat/mlflow-uc-agent-tracing

Conversation

@adamgurary

Copy link
Copy Markdown

Summary

Depends on: databricks/appkit#535

This instruments every executable agent surface discovered in app-templates with MLflow tracing and Unity Catalog setup, then adds behavior-derived conformance tests so future agent templates cannot be added without equivalent coverage.

  • Traces complete agent execution trees across LangChain/LangGraph, OpenAI Agents SDK (including multi-agent handoffs), non-conversational and migration agents, support-agent pipelines, RAG/chat applications, MCP servers, and AppKit-generated agent templates.
  • Captures bounded step inputs/outputs, errors, tool and retrieval calls, sub-agent handoffs, model metadata, token/cache usage, estimated cost when available, latency, time to first token, and streaming duration.
  • Propagates W3C context and stable MLflow trace identity across framework, process, API, and AppKit boundaries.
  • Adds UC experiment/table resources, authenticated OTLP configuration, quickstart/preflight setup, and template documentation.
  • Adds executable-template discovery plus a shared trace contract/normalizer. The conformance suite derives its population from actual agent behavior, preventing a manually maintained allowlist from hiding omissions.

Verification

  • LangChain TypeScript focused suites: 61 passed, plus tsc
  • Multi-agent tracing suite: 9 passed
  • Targeted Ruff correctness and format checks: passed
  • Contract/discovery/sync suites: 55 passed
  • Behavior-derived conformance: 24 passed in 212.62s
  • All 39 contribution commits include DCO sign-off
  • Pre/post-rebase tree hashes are identical (f8b0ee1a3050dbabeee540a30cb82bb6f2e8c020)
  • OSS scrub: no internal URLs, workspace identifiers, warehouse identifiers, experiment identifiers, or credentials; remaining IDs are synthetic fixtures

Broad repository-wide Ruff output includes pre-existing template findings and was not used as the contribution gate; the targeted correctness and format checks for changed Python paths passed.

Live-environment limitations

  • Managed Supervisor still needs live validation that it continues supplied W3C context or returns a persistently linkable remote trace identity.
  • No live Databricks credential/profile was available for this verification run.
  • The Apps Unity Catalog experiment resource used during earlier investigation returned HTTP 403.
  • Therefore, this PR does not claim live Unity Catalog persistence was validated; framework instrumentation and behavior-derived conformance are covered locally.

Adam Gurary added 30 commits August 13, 2026 11:07
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Adam Gurary added 9 commits August 13, 2026 11:07
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
Signed-off-by: Adam Gurary <adam.gurary@databricks.com>
@adamgurary

Copy link
Copy Markdown
Author

@MarioCadenas Could you help route/review this companion to databricks/appkit#535? This repository has no CODEOWNERS routing for the touched paths, and fork contributors cannot request reviewers directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant