test(nextjs): Add AI provider orchestrion instrumentations to e2e app - #22550
Conversation
Add e2e coverage for openai, anthropic-ai and google-genai orchestrion instrumentations in the nextjs-16-orchestrion app. A shared node:http mock server stands in for the three provider APIs so the real SDK clients emit gen_ai spans without live credentials. These libraries are bundled (internalized) and instrumented by the orchestrion build-time loader. Fixes #22506 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
buglitzer run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit e8be434. Configure here.
Replace the `/\/v1beta\/models\/.+:generateContent$/` match with linear startsWith/endsWith string checks to resolve the CodeQL js/polynomial-redos alert. Equivalent matching, no backtracking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| id: 'msg_mock123', | ||
| type: 'message', | ||
| model, | ||
| role: 'assistant', |
There was a problem hiding this comment.
Bug: The mock AI server returns finishReason: 'stop' in lowercase, which is inconsistent with the real API and existing integration tests that expect the uppercase 'STOP'.
Severity: LOW
Suggested Fix
In ai-mock-server.mjs, change the finishReason value from 'stop' to 'STOP' to match the behavior of the actual Google GenAI API and align with existing test expectations. This change should be applied to all mock responses within the file.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location:
dev-packages/e2e-tests/test-applications/nextjs-16-orchestrion/ai-mock-server.mjs#L58
Potential issue: The mock server introduced for end-to-end tests returns a lowercase
`finishReason: 'stop'`. This is inconsistent with the actual Google GenAI API, which
returns uppercase enum values like `'STOP'`. While the current tests do not assert on
this attribute, existing integration tests expect the uppercase value. This discrepancy
in the test infrastructure could cause future tests that validate `finishReason` to fail
or could mask potential bugs in the instrumentation code if it doesn't handle different
casings correctly.
Did we get this right? 👍 / 👎 to inform future reviews.
Adds e2e coverage for
openai,anthropic-aiandgoogle-genaiorchestrion instrumentations in thenextjs-16-orchestrionapp. A sharednode:httpmock server stands in for the three provider APIs so the real SDK clients emit gen_ai spans without live credentials. These libraries are bundled (internalized) and instrumented by the orchestrion build-time loader.Closes #22506