Skip to content

Implement Phase 1H non-stream safety parity across OpenLLMetry fork packages - #4

Merged
manas-fortifyroot merged 1 commit into
fr-v0.52.6.xfrom
feature/phase-1h-nonstream-safety-parity-20260317
Mar 17, 2026
Merged

Implement Phase 1H non-stream safety parity across OpenLLMetry fork packages#4
manas-fortifyroot merged 1 commit into
fr-v0.52.6.xfrom
feature/phase-1h-nonstream-safety-parity-20260317

Conversation

@manas-fortifyroot

Copy link
Copy Markdown

Summary

This PR completes the agreed Phase 1H non-stream safety scope in the FR OpenLLMetry fork.

It adds non-stream prompt/completion safety handling for the remaining in-scope leaf invocation packages, the required framework suppression layers, and LiteLLM without requiring a LiteLLM fork.

What Changed

  • Added non-stream safety coverage for these leaf invocation packages:
    • alephalpha
    • bedrock
    • groq
    • mistralai
    • ollama
    • replicate
    • sagemaker
    • together
    • transformers
    • vertexai
    • watsonx
    • writer
  • Added framework-layer safety coverage for:
    • langchain
    • llamaindex
  • Added a new external package:
    • opentelemetry-instrumentation-litellm
  • Wired LiteLLM into traceloop-sdk instrument discovery and initialization.
  • Followed the minimal-delta fork pattern:
    • substantive new safety logic lives in package-local safety.py
    • existing upstream-tracked files only have narrow helper-call wiring where possible

Why

Provider/leaf-level safety from the original 4 packages was not sufficient for the rest of the agreed non-stream scope.

In particular:

  • several remaining leaf SDK instrumentations had no safety handling yet
  • langchain and llamaindex can create framework spans that bypass downstream provider-level hooks
  • LiteLLM had to be handled without modifying the LiteLLM repo

Fixes Included From Critical Review

This PR also includes the follow-up fixes and regressions discovered during critical self-review, including:

  • Groq async non-metrics completion masking gap
  • LlamaIndex inherited BaseLLM wrapper coverage gap
  • Transformers keyword/nested batch masking gaps
  • Watsonx multi-result masking gap
  • LiteLLM direct text_completion / atext_completion
  • Replicate raw-string completion leakage and broader prompt-key traversal
  • Together mixed-content prompt/completion masking
  • Writer mixed-content prompt masking
  • AlephAlpha multi-candidate completion masking and prompt-structure preservation

Testing

Validated with:

  • fr-openllmetry-py/scripts/run-tests.sh --safety

Latest clean run result:

  • Packages: total=36 passed=21 failed=0 install_failed=0 skipped=15

Also revalidated the affected package slices during the review/fix passes.

@manas-fortifyroot
manas-fortifyroot merged commit 9724c8c into fr-v0.52.6.x Mar 17, 2026
1 check passed
@manas-fortifyroot
manas-fortifyroot deleted the feature/phase-1h-nonstream-safety-parity-20260317 branch March 17, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant