Skip to content

TRACY-87: Automatically collect test fixtures for OpenAI tests#224

Open
Vladislav0Art wants to merge 98 commits intomainfrom
vartiukhov/feature/automate-test-fixtures-collection
Open

TRACY-87: Automatically collect test fixtures for OpenAI tests#224
Vladislav0Art wants to merge 98 commits intomainfrom
vartiukhov/feature/automate-test-fixtures-collection

Conversation

@Vladislav0Art
Copy link
Copy Markdown
Collaborator

@Vladislav0Art Vladislav0Art commented Mar 17, 2026

TLDR

The diff is big because OpenAI test fixtures are committed in this PR! The file changes are way less in size. This PR requires #227 to be reviewed and merged first, as this PR uses its commits!


Description

Closes https://youtrack.jetbrains.com/issue/TRACY-87

Implements an automated fixture recording pipeline for OpenAI provider tests, enabling tests to run offline in MOCK mode using pre-recorded HTTP fixtures, and to be re-recorded from a real LLM provider in RECORD mode. Only OpenAI tests are supported right now.

Core fixture infrastructure (tracing/test-utils)

  1. TestMode — defines two test execution modes:
    • MOCK (default): tests run against a local MockWebServer using pre-recorded fixture files. Controlled via -Dtracy.test.mode=mock or TRACY_TEST_MODE=mock.
    • RECORD: tests hit a real LLM endpoint and record sanitized responses as fixture files. Controlled via -Dtracy.test.mode=record or TRACY_TEST_MODE=record.
  2. RecordingInterceptor — an OkHttp interceptor that wraps response bodies and, on close, saves each response as a fixture file.
  3. FixtureRecorder — serialises recorded HTTP responses to JSON fixture files. Responses small enough to store inline are embedded in the JSON; larger or non-JSON bodies (SSE streams, images, etc.) are saved to separate external files.
  4. FixtureMockServer — loads fixture JSON files from the resources directory and serves them via MockWebServer, matching requests by method + path + fixture tag + sequential index.
  5. OpenAISanitizer — removes or normalises non-deterministic fields from recorded OpenAI responses (IDs, timestamps, AI-generated content, rate-limit headers, LiteLLM proxy headers, etc.) so fixtures are stable across re-recordings.
  6. FixtureNaming — helpers for deriving deterministic fixture file/folder names from JUnit test names, including parameterised test support.

OpenAI test integration (tracing/openai)

  1. BaseOpenAITracingTest — shared base class for all OpenAI provider tests. In MOCK mode it starts a FixtureMockServer and injects fixture tag headers. In RECORD mode it attaches a RecordingInterceptor to the client.
  2. Pre-recorded fixtures committed under tracing/openai/src/jvmTest/resources/fixtures/ — one folder per test class, one sub-folder per test case, one JSON fixture per HTTP round-trip.

Gradle task & CI

This PR is big enough; thus, the Gradle task and CI workflow YAML config are added in #229

Turn `DROPPED_ATTRIBUTES_COUNT_ATTRIBUTE_KEY` to an instance of `AttributeKey` (not plain string).
@Vladislav0Art
Copy link
Copy Markdown
Collaborator Author

Vladislav0Art commented Apr 2, 2026

Tasks

  1. Fix left TODOs
  2. Refactor (remove prints) and review the implementation
  3. Implement OpenAI sanitizer (at least basic fields)
  4. Write PR description and open it for review
  5. [MAJOR] Run the Gradle task for fixture recording (should be in stash somewhere) + restore the GitHub actions yaml config -> opened TRACY-87: Introduce Gradle Task for fixture collection (OpenAI) #229

@Vladislav0Art

This comment was marked as duplicate.

This comment was marked as duplicate.

@Vladislav0Art
Copy link
Copy Markdown
Collaborator Author

@georgiizorabov @slawa4s please read the description before review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants