Skip to content

fix(openai): copy cache tokens onto retry attempts - #42

Closed
sayonfortify wants to merge 1 commit into
mainfrom
fix/openai-retry-cache-token-attrs
Closed

fix(openai): copy cache tokens onto retry attempts#42
sayonfortify wants to merge 1 commit into
mainfrom
fix/openai-retry-cache-token-attrs

Conversation

@sayonfortify

Copy link
Copy Markdown
Contributor

Summary

  • copy OpenAI response-body prompt_tokens_details.cached_tokens onto the canonical llm_attempt span as gen_ai.usage.cache_read_input_tokens
  • also copies total_tokens onto retry-attempt spans via the existing semantic constant
  • adds a focused OpenAI unit test for cache-token extraction from retry-attempt response bodies

Why

Backend v0.0.80 is deployed and can store cache tokens, but live validation still showed zero cache reads for OpenAI. RDS trace projection showed the cache attr on the skipped parent openai.chat span, while the canonical fortifyroot.openai.attempt_1 span had token usage but no cache attr. Because attempts can be exported/processed without parent attrs available, the durable fix is to write cache-token usage directly onto the canonical retry-attempt span in Ocelle.

Validation

  • PYTHONPATH=src pytest tests/openai/test_instrumentation.py -q → 21 passed
  • PYTHONPATH=src pytest tests/test_ocelle_imports.py -q → 3 passed
  • PYTHONPATH=src python -m compileall -q src/fortifyroot/_vendor/opentelemetry/instrumentation/openai/retry_handler.py tests/openai/test_instrumentation.py
  • git diff --check
  • live dev validation with local patched SDK + backend v0.0.80:
    • probe app: openai-cache-token-rds-probe-17062026-sdkfix-local
    • OpenAI returned cached tokens on 7/8 calls (cached=2688)
    • RDS llm_usage_events stored cache_read_sum=18816, max_cache_read=2688

Notes

  • The broader vendored-import smoke has an environment-specific failure in local py3.12: top-level traceloop is installed, so test_no_traceloop_in_top_level fails. This is unrelated to the change; CI clean env should remain authoritative.
  • No backend/proto/schema changes in this PR.

@sayonfortify

Copy link
Copy Markdown
Contributor Author

Closing this direct-vendored PR because vendored OpenLLMetry changes must land in FortifyRoot/openllmetry-py first per internal-sop/openllmetry/VENDORING_SOP.md. Source fix is now in FortifyRoot/openllmetry-py#32; after that merges and a new fr-v0.52.6.N tag is cut, I will re-vendor into ocelle-py and open/update the SDK PR from the tag.

@manas-fortifyroot
manas-fortifyroot deleted the fix/openai-retry-cache-token-attrs branch June 30, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants