feat(tracing): add memory read-path spans

[Kairo-J]

Which issue or RFC does this PR close?

Closes #1241

Rationale for this change

Memory read requests currently expose only top-level application spans, making it difficult to determine whether latency comes from retrieval, embeddings, reranking, Experience recall, or context assembly.

This change adds stage-level tracing while keeping OpenTelemetry owned by the Server layer. The Runtime remains framework-neutral, and tracing failures do not affect search behavior.

What changes are included in this PR?

• Add internal, framework-neutral RuntimeSpan and RuntimeTracing protocols.
• Inject optional Runtime tracing through BuiltinRuntime.
• Implement ServerTracing.stage() using OpenTelemetry INTERNAL spans.
• Record success, failure, and cancelled outcomes while preserving original exceptions.
• Add the following read-path spans:
  • memory.search
  • memory.rerank
  • experience.search
  • context.prepare
• Emit stable zero-result spans when Memory or Experience configuration is absent.
• Wrap actual reranker calls without changing reranking behavior or policy_id.
• Export only bounded booleans, enums, and counts.
• Do not record queries, content, vectors, IDs, or exception messages.
• Add unit and end-to-end coverage for span hierarchy, privacy, cancellation, disabled tracing, and tracing failure isolation.

Validated span trees:

powercontext search_memory
└── memory.search
    ├── embeddings <model>
    └── memory.rerank
        └── invoke_agent memory_rerank

powercontext prepare_context
├── memory.search
├── experience.search
└── context.prepare

Are there any user-facing changes?

Operators with tracing enabled will see the additional internal spans and bounded attributes.

Search results and error behavior remain unchanged. This PR does not change public APIs, OpenAPI, CLI behavior, database schemas, persisted formats, or dependencies.

How was this change tested?

make test
486 passed, 7 skipped

make check
Lock-file validation, pre-commit checks, Ruff formatting/linting, and ty type checking passed.

The tests also cover missing Memory, unconfigured Experience recall, empty results, reranker fallback, embedding and agent span nesting, sensitive-data exclusion, cancellation, and tracing failure isolation.

AI usage statement

OpenAI Codex (GPT-5.6-Sol) was used to help analyze the issue, assist with making changes, write tests.

[CLAassistant]

All committers have signed the CLA.

[Ethan-Xingyue]

This adds a fifth value for powercontext.operation.unit, but the docs that define the vocabulary aren't updated: the unit table in docs/{en,zh}/rfcs/0046_observability_foundations.md
and the span table in docs/{en,zh}/docs/how-to/trace-with-phoenix.md
.

[Kairo-J]

Get, it was an oversight on my part, and I will add it later.

[Ethan-Xingyue]

Just a thought, not a blocker: _context() (which awaits self._provider.get() and opens a DB session) and the wait on _lock(scope_id) both happen before the first stage span opens, so they don't show up in the trace. Since the goal here is locating latency in the read path, and test_same_scope_read_only_searches_do_not_serialize_reranking suggests lock contention is already on your radar, those might be interesting to cover eventually.

[Ethan-Xingyue]

Those are my thoughts so far; let's wait for @PsiACE's review before settling on the specifics.

[PsiACE]

Please focus the tests on exported tracing behavior and real regressions: drop the dynamic policy_id, fake _RecordingTracing, and synthetic SystemExit cases, and cover the actual vector-search span tree plus the readiness probe's no-root-span behavior.

[Kairo-J]

Just a thought, not a blocker: _context() (which awaits self._provider.get() and opens a DB session) and the wait on _lock(scope_id) both happen before the first stage span opens, so they don't show up in the trace. Since the goal here is locating latency in the read path, and test_same_scope_read_only_searches_do_not_serialize_reranking suggests lock contention is already on your radar, those might be interesting to cover eventually.

@Ethan-Xingyue
Thanks, that’s indeed a great angle. Currently, the lock wait time during context resolution and the prepare_context phase appears in the external application Span merely as "unattributed exclusive time."

I noticed a detail here: the default RelationalContexts.get() (_context() line 1173) simply invokes the Provider to resolve and cache scoped services (RelationalContexts.get() line 434) without initiating a database session; the actual database read operation still occurs within memory.search. However, the skew caused by delayed attribution is indeed an issue.

Given that context retrieval and lock waiting are runtime concerns spanning multiple stages—whereas the scope of this PR is limited to the memory read phase—I intend to keep the PR focused.

We can add context.resolve and scope.lock.wait as sibling Spans in future work without altering the semantics of the Span introduced here. Thanks for pointing this out.

[Kairo-J]

@Ethan-Xingyue @PsiACE When adding ready regression coverage, I encountered an additional special case that I felt was worth mentioning.

The previous suppression mechanism relied on attaching an unsampled synthetic parent model during ready testing. This worked for the default ParentBased sampler, but injected embedding models using an effective ALWAYS_ON sampler ignored the parent model's sampling decisions and still exported the embeddings<model> span. Since the synthetic parent model was never exported, this resulted in incomplete/isolated ready tracking.

I fixed this by providing a separate, uninstrumented ready copy of the injected PydanticAIEmbeddingModel instance:

• The adapter and its Embedder are shallow-copied;

• instrument=False is applied only to the ready copy;

• The original adapter still retains the instrumentation used for manipulating the search;

• Subclass behavior and existing providers/settings/state are preserved;

• Since no shared objects are temporarily modified, concurrent ready state calls and search calls do not disable each other's tracing.

Regression tests now use TracerProvider(sampler=ALWAYS_ON) via the public create_server_app() injection path. It first verifies whether the ready state derives any inference span, then performs a real HTTP vector search and verifies that the operation trace still produces the following result:

powercontext search_memory
└── memory.search
    └── embeddings test

The entire test suite passed 484 tests, skipped 7 tests, and both make check and make docs-test passed.

[PsiACE]

Readiness should stay outside tracing, but attaching a synthetic parent makes tracing setup part of the readiness path and can abort server startup. Please make it best-effort.