Skip to content

test(integ): expand harness config-shape coverage#1608

Merged
tejaskash merged 2 commits into
mainfrom
test/harness-integ-coverage
Jun 23, 2026
Merged

test(integ): expand harness config-shape coverage#1608
tejaskash merged 2 commits into
mainfrom
test/harness-integ-coverage

Conversation

@tejaskash

Copy link
Copy Markdown
Contributor

Summary

Expands the harness integration test suite (integ-tests/add-remove-harness.test.ts) to cover the harness configuration surface that was previously untested at the integration layer. Follow-up to #1598, which made harness the default create path and brought the harness flows into the GA build.

All new cases are local — no AWS. They exercise the CLI flag → action → validate → spec-write wiring that schema unit tests don't reach.

Coverage added

  • model: lite_llm (+--api-base/--additional-params), bedrock --api-format, plus api-format provider-gating rejections (open_ai/gemini)
  • tools (add tool): remote_mcp, agentcore_code_interpreter, agentcore_gateway (awsIam), inline_function, plus required-config rejections
  • skills (add skill): path / s3 / git (+auth credential)
  • memory modes (gated): managed + strategies, existing-by-name + tuning, plus by-arn / no-ref / gated-off rejections
  • truncation: summarization
  • VPC network-mode validation (no deploy)
  • CUSTOM_JWT inbound authorizer + missing-config rejection

Notes

  • Fast by design. All cases share a single project (one create spawn instead of one per block) and rejection groups use it.each tables. Each runCLI is a real ~1.3s process spawn, so the suite is kept tight.
  • No duplication. Pure schema refinements already covered by src/schema/schemas/primitives/__tests__/harness.test.ts (top-k / api-base provider gating, skill URI-scheme rejection) are intentionally not re-tested here.
  • Out of scope: E2E additions (lite_llm deploy, harness-with-tool deploy, CUSTOM_JWT + fetch access) — tracked separately.

Testing

  • vitest run --project integ integ-tests/add-remove-harness.test.ts35 passed (16 existing + 19 new).
  • eslint, prettier, secretlint, and tsc --noEmit clean.

Add an integration suite covering the harness config surface the existing
tests miss — all local (no AWS), exercising CLI flag → action → validate →
spec-write wiring that schema unit tests don't reach:

- model: lite_llm (+api-base/additional-params), bedrock api-format, plus
  api-format provider-gating rejections (open_ai/gemini)
- tools (`add tool`): remote_mcp, code_interpreter, gateway (awsIam),
  inline_function, plus required-config rejections
- skills (`add skill`): path / s3 / git (+auth credential)
- memory modes (gated): managed + strategies, existing-by-name + tuning,
  plus by-arn / no-ref / gated-off rejections
- truncation: summarization
- VPC network-mode validation (no deploy)
- CUSTOM_JWT inbound authorizer + missing-config rejection

Cases share one project (single `create` spawn) and use it.each tables for
rejection groups to keep the suite fast. Pure schema refinements already
covered by harness.test.ts unit tests are intentionally not duplicated.

All 35 tests in the file pass.
@tejaskash tejaskash requested a review from a team June 22, 2026 19:20
@github-actions github-actions Bot added size/m PR size: M agentcore-harness-reviewing AgentCore Harness review in progress labels Jun 22, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 22, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Package Tarball

aws-agentcore-0.20.2.tgz

How to install

gh release download pr-1608-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.20.2.tgz

@agentcore-cli-automation agentcore-cli-automation left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are well-structured and pragmatically scoped — sharing one project, real CLI spawns, and it.each rejection tables strikes a good balance. One concern below worth addressing before merge.

Comment thread integ-tests/add-remove-harness.test.ts
@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Jun 22, 2026
@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 36.98% 13516 / 36546
🔵 Statements 36.25% 14369 / 39631
🔵 Functions 31.59% 2317 / 7334
🔵 Branches 30.76% 8905 / 28944
Generated in workflow #3755 for commit 26bb9e9 by the Vitest Coverage Report Action

avi-alpert
avi-alpert previously approved these changes Jun 22, 2026
…FEATURES

The 'rejects --memory-mode when gated off' case relied on the gate being
unset in the environment. cleanSpawnEnv inherits the host process.env and
does not strip ENABLE_GATED_FEATURES, so a shell/CI with it exported as '1'
would flip the case: the CLI accepts the flag, exits 0, and the rejection
assertion fails for an environment-dependent reason.

Force the gate off explicitly via env { ENABLE_GATED_FEATURES: '' } ('' is
off since isGatedFeaturesEnabled checks === '1'). Verified the block passes
both with the gate unset and with ENABLE_GATED_FEATURES=1 exported.
@github-actions github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Jun 22, 2026
@tejaskash

Copy link
Copy Markdown
Contributor Author

Good catch — verified and fixed in 26bb9e9.

You're right on all three points: isGatedFeaturesEnabled() is a strict === '1', cleanSpawnEnv only strips AGENTCORE_TELEMETRY_DISABLED (not ENABLE_GATED_FEATURES), so the gated-off case inherited the host value and would flip if a shell/CI had it exported.

I went with option 1 (force the gate off for that one case via env: { ENABLE_GATED_FEATURES: '' }) rather than stripping it globally in cleanSpawnEnv. Reason: add-remove-config-bundle.test.ts:113 deliberately reads process.env.ENABLE_GATED_FEATURES to predict the spawned CLI's behavior — a global strip would make the CLI always-off while that test still computed expectedBranch='feature-branch' in a gated-on shell, breaking it. Option 1 is self-contained and touches no shared infra.

Verified the memory-modes block passes both with the gate unset and with ENABLE_GATED_FEATURES=1 exported (before the fix, the gated-off case failed in the latter).

@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 22, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 22, 2026
@tejaskash tejaskash merged commit 5fd591a into main Jun 23, 2026
32 checks passed
@tejaskash tejaskash deleted the test/harness-integ-coverage branch June 23, 2026 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants