Skip to content

feat(e2e): enable gemini tests#1503

Merged
jariy17 merged 1 commit into
aws:mainfrom
Hweinstock:fix/enable-gemini
Jun 10, 2026
Merged

feat(e2e): enable gemini tests#1503
jariy17 merged 1 commit into
aws:mainfrom
Hweinstock:fix/enable-gemini

Conversation

@Hweinstock

Copy link
Copy Markdown
Contributor

Problem

#1435
Billing issue has since been fixed, we can re-enable.

Solution

  • re-enable gemini invokes by removing the skipInvoke flag.

Testing

Ran a smoke test with the key: https://ai.google.dev/api/models#models_list-SHELL, and its now working.

@github-actions github-actions Bot added the size/xs PR size: XS label Jun 10, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 10, 2026
@github-actions github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Jun 10, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 10, 2026

@agentcore-cli-automation agentcore-cli-automation left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change itself is straightforward and correct — removing skipInvoke: true will let the invoke tests run. One thing to flag for the harness suite though.

@@ -4,5 +4,4 @@ createHarnessE2ESuite({
modelProvider: 'gemini',
apiKeyEnvVar: 'GEMINI_API_KEY_ARN',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

harness-gemini.test.ts reads GEMINI_API_KEY_ARN, but that env var isn't set anywhere in CI — only GEMINI_API_KEY is exposed (e2e-tests.yml line 178, e2e-tests-full.yml line 89, canary.yml line 89). With skipInvoke removed, the harness suite's canRun gate (harness-e2e-helper.ts line 23-24) will now evaluate to false because hasRequiredVar is false, so beforeAll/it blocks all skip — meaning the entire harness Gemini suite (including invoke) silently skips in CI rather than actually running.

In other words, the smoke test you ran locally (with GEMINI_API_KEY_ARN set) doesn't reflect what happens in CI for this file. Some options:

  1. Wire up a GEMINI_API_KEY_ARN secret and export it in the harness step of e2e-tests.yml (around line 173-183), e2e-tests-full.yml, and canary.yml. This is what makes the test actually run.
  2. Change harness-gemini.test.ts to use GEMINI_API_KEY like the other three Gemini tests, if the harness path can accept a raw key (or if the harness e2e helper can be taught to upload the key into a secret as part of setup).
  3. Drop this file from the PR and re-enable harness Gemini in a follow-up once the ARN secret is set up.

The other three test files (googleadk/langgraph/strands) use GEMINI_API_KEY which is wired up correctly, so those will run as intended.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a broader issue where the entire harness tests are skipped. filing an issue and will look into as a follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Hweinstock Hweinstock marked this pull request as ready for review June 10, 2026 16:17
@Hweinstock Hweinstock requested a review from a team June 10, 2026 16:17
@jariy17 jariy17 merged commit c3d7565 into aws:main Jun 10, 2026
33 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentcore-harness-reviewing AgentCore Harness review in progress size/xs PR size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants