test(e2e): add harness E2E coverage for lite_llm, tools, and CUSTOM_JWT#1609
test(e2e): add harness E2E coverage for lite_llm, tools, and CUSTOM_JWT#1609tejaskash wants to merge 2 commits into
Conversation
… CUSTOM_JWT Extends the harness E2E surface with three real-AWS scenarios that the provider matrix (bedrock/open_ai/gemini) doesn't cover: - harness-litellm.test.ts — lite_llm provider routed at a Bedrock model (no third-party key); deploy-only (skipInvoke) to prove the model config is accepted by CloudFormation. Extends harness-e2e-helper with modelId/apiBase/ additionalParams support. - harness-with-tool.test.ts — bedrock harness + agentcore_code_interpreter tool via 'add tool'; proves tool wiring survives synth/deploy and the harness still invokes. - harness-custom-jwt.test.ts — harness with a CUSTOM_JWT authorizer backed by a Cognito pool; asserts AuthorizerConfiguration in the CFN template, SigV4 rejection, and bearer-token invoke (mirrors byo-custom-jwt.test.ts). All self-skip without AWS creds. The per-PR e2e workflow auto-runs changed harness-*.test.ts files; the full suite shards everything.
github-actions
Bot
added
the
agentcore-harness-reviewing
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
Package TarballHow to installgh release download pr-1609-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.20.2.tgz |
agentcore-cli-automation
left a comment
agentcore-cli-automation
left a comment
There was a problem hiding this comment.
LGTM — closely mirrors the existing byo-custom-jwt.test.ts pattern (which is good) and exercises three meaningful gaps in harness E2E coverage (lite_llm provider, tool wiring at CFN synth, CUSTOM_JWT authorizer). The helper extension for modelId/apiBase/additionalParams is appropriately scoped. Self-gating on AWS creds means no impact on contributor PRs. No blocking issues found.
github-actions
Bot
removed
the
agentcore-harness-reviewing
Coverage Report
|
Set the CUSTOM_JWT harness up via 'add harness' with the JWT + OAuth-credential flags (--authorizer-type/--discovery-url/--allowed-audience/--client-id/ --client-secret) instead of patching harness.json directly. This registers the managed OAuth credential and .env.local secret — the real user flow — which are the prerequisites for fetch access to mint a token. Adds a step asserting 'fetch access --type harness' returns a CUSTOM_JWT bearer token and that the JWT's issuer/client_id claims match the Cognito pool. Depends on the fetch-access-harness feature (PR #1611); until that merges, this step exercises a command not yet on main. The e2e suite is manual/full-suite only, so this does not gate per-PR CI.
|
Updated and marked ready for review. Added a Verified against real AWS (us-west-2): the bedrock, litellm, and with-tool harness e2e tests pass end-to-end. The fetch-access flow itself was also verified live (CLI + TUI) on a deployed CUSTOM_JWT harness. Dependency: the |
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
2 participants
Summary
Adds three real-AWS E2E scenarios for harness that the existing provider matrix (
harness-bedrock/harness-openai/harness-gemini) doesn't cover. Companion to #1608 (integration-layer coverage); these prove behaviors that only surface at CloudFormation synth/deploy or runtime.What's added
harness-litellm.test.ts— thelite_llmprovider, routed at a Bedrock model so the deploy needs no third-party API key (LiteLLM's bedrock backend uses the runtime execution role's IAM). Deploy-only (skipInvoke) — the bedrock suite already proves invoke; this proves the lite_llm model config is accepted by CloudFormation.harness-with-tool.test.ts— a bedrock harness with anagentcore_code_interpretertool attached viaadd tool. Tool wiring only fails at synth/deploy, so this is the layer that catches it; then invokes to confirm the harness still works. Code interpreter chosen because it needs no external ARN.harness-custom-jwt.test.ts— a harness with aCUSTOM_JWTauthorizer backed by a Cognito user pool. AssertsAuthorizerConfigurationin the CFN template, SigV4 rejection, and bearer-token invoke. Mirrors the existingbyo-custom-jwt.test.tspattern.Also extends
harness-e2e-helper.tswithmodelId/apiBase/additionalParamsconfig (for the lite_llm case).Notes
e2e-tests.ymlworkflow auto-runs any changedharness-*.test.ts, so these run once on this PR; afterward they only run in the sharded full suite (e2e-tests-full.yml).createhas no harness authorizer flags, so the JWT test patchesharness.jsonafter create — same approachbyo-custom-jwt.test.tsuses foragentcore.json.fetch accessdoes not support harness (FetchResourceType = 'gateway' | 'agent'); the harness JWT-token path lives ininvokeauto-fetch. The JWT test therefore exercises bearer-token invoke, notfetch access.Testing
vitest list --project e2eenumerates all cases correctly (lite_llm's invoke step correctly absent due toskipInvoke).tsc --noEmit, eslint, prettier, secretlint all clean.