TRACY-87: Automatically collect test fixtures for OpenAI tests#224
Open
Vladislav0Art wants to merge 98 commits intomainfrom
Open
TRACY-87: Automatically collect test fixtures for OpenAI tests#224Vladislav0Art wants to merge 98 commits intomainfrom
Vladislav0Art wants to merge 98 commits intomainfrom
Conversation
Turn `DROPPED_ATTRIBUTES_COUNT_ATTRIBUTE_KEY` to an instance of `AttributeKey` (not plain string).
Collaborator
Author
|
Tasks
|
…s (via litellm endpoint)
This comment was marked as duplicate.
This comment was marked as duplicate.
Agent-Logs-Url: https://github.com/JetBrains/tracy/sessions/ac7acff7-cc9b-40d3-a9ec-1ec141c2c5ef Co-authored-by: Vladislav0Art <47076189+Vladislav0Art@users.noreply.github.com>
This comment was marked as duplicate.
This comment was marked as duplicate.
Collaborator
Author
|
@georgiizorabov @slawa4s please read the description before review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
The diff is big because OpenAI test fixtures are committed in this PR! The file changes are way less in size. This PR requires #227 to be reviewed and merged first, as this PR uses its commits!
Description
Closes
https://youtrack.jetbrains.com/issue/TRACY-87Implements an automated fixture recording pipeline for OpenAI provider tests, enabling tests to run offline in
MOCKmode using pre-recorded HTTP fixtures, and to be re-recorded from a real LLM provider inRECORDmode. Only OpenAI tests are supported right now.Core fixture infrastructure (
tracing/test-utils)TestMode— defines two test execution modes:MOCK(default): tests run against a localMockWebServerusing pre-recorded fixture files. Controlled via-Dtracy.test.mode=mockorTRACY_TEST_MODE=mock.RECORD: tests hit a real LLM endpoint and record sanitized responses as fixture files. Controlled via-Dtracy.test.mode=recordorTRACY_TEST_MODE=record.RecordingInterceptor— an OkHttp interceptor that wraps response bodies and, on close, saves each response as a fixture file.FixtureRecorder— serialises recorded HTTP responses to JSON fixture files. Responses small enough to store inline are embedded in the JSON; larger or non-JSON bodies (SSE streams, images, etc.) are saved to separate external files.FixtureMockServer— loads fixture JSON files from the resources directory and serves them viaMockWebServer, matching requests by method + path + fixture tag + sequential index.OpenAISanitizer— removes or normalises non-deterministic fields from recorded OpenAI responses (IDs, timestamps, AI-generated content, rate-limit headers, LiteLLM proxy headers, etc.) so fixtures are stable across re-recordings.FixtureNaming— helpers for deriving deterministic fixture file/folder names from JUnit test names, including parameterised test support.OpenAI test integration (
tracing/openai)BaseOpenAITracingTest— shared base class for all OpenAI provider tests. InMOCKmode it starts aFixtureMockServerand injects fixture tag headers. InRECORDmode it attaches aRecordingInterceptorto the client.tracing/openai/src/jvmTest/resources/fixtures/— one folder per test class, one sub-folder per test case, one JSON fixture per HTTP round-trip.Gradle task & CI
This PR is big enough; thus, the Gradle task and CI workflow YAML config are added in #229