feat(e2e): enable gemini tests#1503
Conversation
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-cli-automation
left a comment
There was a problem hiding this comment.
Change itself is straightforward and correct — removing skipInvoke: true will let the invoke tests run. One thing to flag for the harness suite though.
| @@ -4,5 +4,4 @@ createHarnessE2ESuite({ | |||
| modelProvider: 'gemini', | |||
| apiKeyEnvVar: 'GEMINI_API_KEY_ARN', | |||
There was a problem hiding this comment.
harness-gemini.test.ts reads GEMINI_API_KEY_ARN, but that env var isn't set anywhere in CI — only GEMINI_API_KEY is exposed (e2e-tests.yml line 178, e2e-tests-full.yml line 89, canary.yml line 89). With skipInvoke removed, the harness suite's canRun gate (harness-e2e-helper.ts line 23-24) will now evaluate to false because hasRequiredVar is false, so beforeAll/it blocks all skip — meaning the entire harness Gemini suite (including invoke) silently skips in CI rather than actually running.
In other words, the smoke test you ran locally (with GEMINI_API_KEY_ARN set) doesn't reflect what happens in CI for this file. Some options:
- Wire up a
GEMINI_API_KEY_ARNsecret and export it in the harness step ofe2e-tests.yml(around line 173-183),e2e-tests-full.yml, andcanary.yml. This is what makes the test actually run. - Change
harness-gemini.test.tsto useGEMINI_API_KEYlike the other three Gemini tests, if the harness path can accept a raw key (or if the harness e2e helper can be taught to upload the key into a secret as part of setup). - Drop this file from the PR and re-enable harness Gemini in a follow-up once the ARN secret is set up.
The other three test files (googleadk/langgraph/strands) use GEMINI_API_KEY which is wired up correctly, so those will run as intended.
There was a problem hiding this comment.
This is a broader issue where the entire harness tests are skipped. filing an issue and will look into as a follow-up.
Problem
#1435
Billing issue has since been fixed, we can re-enable.
Solution
Testing
Ran a smoke test with the key: https://ai.google.dev/api/models#models_list-SHELL, and its now working.