From 02a420c3def77481bd6ec413c004ec5592de3069 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9s=20Aguilar?= Date: Mon, 18 May 2026 16:08:38 -0500 Subject: [PATCH 1/2] docs: update CLI command names and table formatting - Replace `batch-eval` with `batch-evaluation` in PERMISSIONS.md for consistency - Replace `recommend` with `recommendation` in PERMISSIONS.md for consistency - Update table column alignment in PERMISSIONS.md logging section - Standardize table formatting across commands.md and evals.md documentation - Align CLI command naming conventions throughout documentation --- docs/PERMISSIONS.md | 48 +++--- docs/commands.md | 381 ++++++++++++++++++++++++++++++++++---------- docs/evals.md | 52 ++++-- 3 files changed, 352 insertions(+), 129 deletions(-) diff --git a/docs/PERMISSIONS.md b/docs/PERMISSIONS.md index 6b6204ebe..bde70564d 100644 --- a/docs/PERMISSIONS.md +++ b/docs/PERMISSIONS.md @@ -337,17 +337,17 @@ Required for all deployment operations (`deploy`, `status`, `diff`). ### Batch evaluation and recommendations -| Action | CLI Commands | Purpose | -| ----------------------------------------- | ---------------- | ------------------------------ | -| `bedrock-agentcore:StartBatchEvaluation` | `run batch-eval` | Start a batch evaluation job | -| `bedrock-agentcore:GetBatchEvaluation` | `run batch-eval` | Poll batch evaluation status | -| `bedrock-agentcore:ListBatchEvaluations` | `evals history` | List past batch evaluations | -| `bedrock-agentcore:StopBatchEvaluation` | `run batch-eval` | Stop an in-progress batch eval | -| `bedrock-agentcore:DeleteBatchEvaluation` | `run batch-eval` | Delete a batch evaluation | -| `bedrock-agentcore:StartRecommendation` | `run recommend` | Start a recommendation job | -| `bedrock-agentcore:GetRecommendation` | `run recommend` | Poll recommendation status | -| `bedrock-agentcore:ListRecommendations` | `run recommend` | List past recommendations | -| `bedrock-agentcore:DeleteRecommendation` | `run recommend` | Stop/delete a recommendation | +| Action | CLI Commands | Purpose | +| ----------------------------------------- | ---------------------- | ------------------------------ | +| `bedrock-agentcore:StartBatchEvaluation` | `run batch-evaluation` | Start a batch evaluation job | +| `bedrock-agentcore:GetBatchEvaluation` | `run batch-evaluation` | Poll batch evaluation status | +| `bedrock-agentcore:ListBatchEvaluations` | `evals history` | List past batch evaluations | +| `bedrock-agentcore:StopBatchEvaluation` | `run batch-evaluation` | Stop an in-progress batch eval | +| `bedrock-agentcore:DeleteBatchEvaluation` | `run batch-evaluation` | Delete a batch evaluation | +| `bedrock-agentcore:StartRecommendation` | `run recommendation` | Start a recommendation job | +| `bedrock-agentcore:GetRecommendation` | `run recommendation` | Poll recommendation status | +| `bedrock-agentcore:ListRecommendations` | `run recommendation` | List past recommendations | +| `bedrock-agentcore:DeleteRecommendation` | `run recommendation` | Stop/delete a recommendation | ### Identity and credential management @@ -375,19 +375,19 @@ Required for all deployment operations (`deploy`, `status`, `diff`). ### Logging, traces, and observability -| Action | CLI Commands | Purpose | -| ------------------------------- | ---------------------------------------- | ---------------------------------------------------------- | -| `logs:StartLiveTail` | `logs` | Stream agent logs in real-time | -| `logs:FilterLogEvents` | `logs` | Search agent logs | -| `logs:StartQuery` | `traces list`, `traces get`, `run evals` | Run CloudWatch Logs Insights queries | -| `logs:GetQueryResults` | `traces list`, `traces get`, `run evals` | Retrieve query results | -| `logs:DescribeResourcePolicies` | `deploy` | Check for X-Ray log resource policy | -| `logs:PutResourcePolicy` | `deploy` | Create resource policy for X-Ray trace access | -| `logs:DescribeLogGroups` | `run batch-eval`, `run recommend` | Discover runtime log groups for evaluation data sources | -| `logs:CreateLogGroup` | `run batch-eval` | Create log group for batch evaluation results output | -| `logs:CreateLogStream` | `run batch-eval` | Create log stream for batch evaluation results | -| `logs:PutLogEvents` | `run batch-eval` | Write batch evaluation results to CloudWatch Logs | -| `logs:PutRetentionPolicy` | `run batch-eval` | Set retention policy on batch evaluation results log group | +| Action | CLI Commands | Purpose | +| ------------------------------- | -------------------------------------------- | ---------------------------------------------------------- | +| `logs:StartLiveTail` | `logs` | Stream agent logs in real-time | +| `logs:FilterLogEvents` | `logs` | Search agent logs | +| `logs:StartQuery` | `traces list`, `traces get`, `run evals` | Run CloudWatch Logs Insights queries | +| `logs:GetQueryResults` | `traces list`, `traces get`, `run evals` | Retrieve query results | +| `logs:DescribeResourcePolicies` | `deploy` | Check for X-Ray log resource policy | +| `logs:PutResourcePolicy` | `deploy` | Create resource policy for X-Ray trace access | +| `logs:DescribeLogGroups` | `run batch-evaluation`, `run recommendation` | Discover runtime log groups for evaluation data sources | +| `logs:CreateLogGroup` | `run batch-evaluation` | Create log group for batch evaluation results output | +| `logs:CreateLogStream` | `run batch-evaluation` | Create log stream for batch evaluation results | +| `logs:PutLogEvents` | `run batch-evaluation` | Write batch evaluation results to CloudWatch Logs | +| `logs:PutRetentionPolicy` | `run batch-evaluation` | Set retention policy on batch evaluation results log group | ### Transaction search setup diff --git a/docs/commands.md b/docs/commands.md index b5a304a70..877990518 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -72,33 +72,36 @@ agentcore create \ --memory none ``` -| Flag | Description | -| -------------------------- | -------------------------------------------------------------------------------------------------------------- | -| `--name ` | Project name (alphanumeric, starts with letter, max 23 chars) | -| `--defaults` | Use defaults (Python, Strands, Bedrock, no memory) | -| `--no-agent` | Skip agent creation | -| `--type ` | `create` (default) or `import` | -| `--language ` | `Python` (default) or `TypeScript` (Strands-only; see [Frameworks](frameworks.md#supported-languages)) | -| `--framework ` | `Strands`, `LangChain_LangGraph`, `GoogleADK`, `OpenAIAgents`, `VercelAI` | -| `--model-provider

` | `Bedrock`, `Anthropic`, `OpenAI`, `Gemini` | -| `--build ` | `CodeZip` (default) or `Container` (see [Container Builds](container-builds.md)) | -| `--api-key ` | API key for non-Bedrock providers | -| `--memory ` | `none`, `shortTerm`, `longAndShortTerm` (see [Memory Shorthand Mapping](memory.md#--memory-shorthand-mapping)) | -| `--protocol ` | `HTTP` (default), `MCP`, `A2A` | -| `--network-mode ` | `PUBLIC` (default) or `VPC` | -| `--subnets ` | Comma-separated subnet IDs (required for VPC mode) | -| `--security-groups ` | Comma-separated security group IDs (required for VPC mode) | -| `--agent-id ` | Bedrock Agent ID (import only) | -| `--agent-alias-id ` | Bedrock Agent Alias ID (import only) | -| `--region ` | AWS region for Bedrock Agent (import only) | -| `--idle-timeout ` | Idle session timeout in seconds | -| `--max-lifetime ` | Max instance lifetime in seconds | -| `--output-dir

` | Output directory | -| `--skip-git` | Skip git initialization | -| `--skip-python-setup` | Skip venv setup | -| `--skip-install` | Skip all dependency installation (npm install, uv sync) | -| `--dry-run` | Preview without creating | -| `--json` | JSON output | +| Flag | Description | +| ------------------------------------- | -------------------------------------------------------------------------------------------------------------- | +| `--name ` | Agent (resource) name; also used as project directory name when `--project-name` is omitted | +| `--project-name ` | Project directory name (alphanumeric, starts with letter, max 23 chars) | +| `--defaults` | Use defaults (Python, Strands, Bedrock, no memory) | +| `--no-agent` | Skip agent creation | +| `--type ` | `create` (default) or `import` | +| `--language ` | `Python` (default) or `TypeScript` (Strands-only; see [Frameworks](frameworks.md#supported-languages)) | +| `--framework ` | `Strands`, `LangChain_LangGraph`, `GoogleADK`, `OpenAIAgents`, `VercelAI` | +| `--model-provider

` | `Bedrock`, `Anthropic`, `OpenAI`, `Gemini` | +| `--build ` | `CodeZip` (default) or `Container` (see [Container Builds](container-builds.md)) | +| `--api-key ` | API key for non-Bedrock providers | +| `--memory ` | `none`, `shortTerm`, `longAndShortTerm` (see [Memory Shorthand Mapping](memory.md#--memory-shorthand-mapping)) | +| `--protocol ` | `HTTP` (default), `MCP`, `A2A`, `AGUI` | +| `--network-mode ` | `PUBLIC` (default) or `VPC` | +| `--subnets ` | Comma-separated subnet IDs (required for VPC mode) | +| `--security-groups ` | Comma-separated security group IDs (required for VPC mode) | +| `--agent-id ` | Bedrock Agent ID (import only) | +| `--agent-alias-id ` | Bedrock Agent Alias ID (import only) | +| `--region ` | AWS region for Bedrock Agent (import only) | +| `--idle-timeout ` | Idle session timeout in seconds | +| `--max-lifetime ` | Max instance lifetime in seconds | +| `--session-storage-mount-path ` | Absolute mount path for session filesystem storage under `/mnt` (e.g. `/mnt/data`) | +| `--with-config-bundle` | [preview] Create a config bundle wired into the generated agent template | +| `--output-dir

` | Output directory | +| `--skip-git` | Skip git initialization | +| `--skip-python-setup` | Skip venv setup | +| `--skip-install` | Skip all dependency installation (npm install, uv sync) | +| `--dry-run` | Preview without creating | +| `--json` | JSON output | ### deploy @@ -158,6 +161,84 @@ agentcore validate -d ./my-project | ------------------------ | ----------------- | | `-d, --directory ` | Project directory | +### import + +Import existing AgentCore resources from your AWS account into the project, or migrate from a Bedrock AgentCore Starter +Toolkit project. + +```bash +# Import a runtime by ARN +agentcore import runtime \ + --arn arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/my-runtime \ + --code ./app/MyAgent \ + --entrypoint main.py \ + --name MyAgent + +# Import a memory resource +agentcore import memory --arn arn:aws:bedrock-agentcore:us-east-1:123456789012:memory/abc123 --name SharedMemory + +# Import an evaluator +agentcore import evaluator --arn arn:aws:bedrock-agentcore:us-east-1:123456789012:evaluator/eval123 --name ResponseQuality + +# Import an online eval config +agentcore import online-eval --arn arn:aws:bedrock-agentcore:us-east-1:123456789012:online-evaluation-config/oec123 --name QualityMonitor + +# Import a gateway (with all its targets) +agentcore import gateway --arn arn:aws:bedrock-agentcore:us-east-1:123456789012:gateway/gw123 + +# Import from a Starter Toolkit project (auto-detects .bedrock_agentcore.yaml in cwd) +agentcore import +agentcore import --source ./path/to/.bedrock_agentcore.yaml -y +``` + +Top-level flags (apply when running `agentcore import` without a subcommand to migrate a Starter Toolkit project): + +| Flag | Description | +| ------------------- | -------------------------------------------------------- | +| `--source ` | Path to the `.bedrock_agentcore.yaml` configuration file | +| `--target ` | Deployment target name (only when project has multiple) | +| `-y, --yes` | Auto-confirm prompts | + +Subcommand: `import runtime` + +| Flag | Description | +| --------------------- | ---------------------------------------------------------------------- | +| `--arn ` | Runtime ARN to import | +| `--code ` | Path to directory containing the entrypoint (e.g. folder with main.py) | +| `--entrypoint ` | Entrypoint file (auto-detected from runtime, e.g. `main.py`) | +| `--name ` | Local name for the imported runtime | +| `-y, --yes` | Auto-confirm prompts | + +Subcommand: `import memory` + +| Flag | Description | +| ------------------- | ---------------------------------- | +| `--arn ` | Memory ARN to import | +| `--name ` | Local name for the imported memory | +| `-y, --yes` | Auto-confirm prompts | + +Subcommand: `import evaluator` + +| Flag | Description | +| ---------------------- | ------------------------------------- | +| `--arn ` | Evaluator ARN to import | +| `--name ` | Local name for the imported evaluator | +| `-y, --yes` | Auto-confirm prompts | + +Subcommand: `import online-eval` + +| Flag | Description | +| ------------------- | --------------------------------------- | +| `--arn ` | Online evaluation config ARN to import | +| `--name ` | Local name for the imported online eval | +| `-y, --yes` | Auto-confirm prompts | + +Subcommand: `import gateway` + +| Flag | Description | +| -------------------- | ---------------------------------------- | +| `--arn ` | Gateway ARN to import (with all targets) | + --- ## Resource Management @@ -203,36 +284,39 @@ agentcore add agent \ --memory none ``` -| Flag | Description | -| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -| `--name ` | Agent name (alphanumeric + underscores, starts with letter, max 48 chars) | -| `--type ` | `create` (default), `byo`, or `import` | -| `--build ` | `CodeZip` (default) or `Container` (see [Container Builds](container-builds.md)) | -| `--language ` | `Python` (create); `Python`, `TypeScript`, `Other` (BYO) | -| `--framework ` | `Strands`, `LangChain_LangGraph`, `GoogleADK`, `OpenAIAgents`, `VercelAI` | -| `--model-provider

` | `Bedrock`, `Anthropic`, `OpenAI`, `Gemini` | -| `--api-key ` | API key for non-Bedrock providers | -| `--memory ` | `none`, `shortTerm`, `longAndShortTerm` (create and import; see [Memory Shorthand Mapping](memory.md#--memory-shorthand-mapping)) | -| `--protocol ` | `HTTP` (default), `MCP`, `A2A` | -| `--code-location ` | Path to existing code (BYO only) | -| `--entrypoint ` | Entry file relative to code-location (BYO, default: `main.py`) | -| `--network-mode ` | `PUBLIC` (default) or `VPC` | -| `--subnets ` | Comma-separated subnet IDs (required for VPC mode) | -| `--security-groups ` | Comma-separated security group IDs (required for VPC mode) | -| `--agent-id ` | Bedrock Agent ID (import only) | -| `--agent-alias-id ` | Bedrock Agent Alias ID (import only) | -| `--region ` | AWS region for Bedrock Agent (import only) | -| `--authorizer-type ` | Inbound auth: `AWS_IAM` or `CUSTOM_JWT` | -| `--discovery-url ` | OIDC discovery URL (for CUSTOM_JWT) | -| `--allowed-audience ` | Comma-separated allowed audiences (for CUSTOM_JWT) | -| `--allowed-clients ` | Comma-separated allowed client IDs (for CUSTOM_JWT) | -| `--allowed-scopes ` | Comma-separated allowed scopes (for CUSTOM_JWT) | -| `--custom-claims ` | Custom claim validations as JSON array (for CUSTOM_JWT) | -| `--client-id ` | OAuth client ID for agent bearer token | -| `--client-secret ` | OAuth client secret | -| `--idle-timeout ` | Idle session timeout in seconds | -| `--max-lifetime ` | Max instance lifetime in seconds | -| `--json` | JSON output | +| Flag | Description | +| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--name ` | Agent name (alphanumeric + underscores, starts with letter, max 48 chars) | +| `--type ` | `create` (default), `byo`, or `import` | +| `--build ` | `CodeZip` (default) or `Container` (see [Container Builds](container-builds.md)) | +| `--language ` | `Python` (create); `Python`, `TypeScript`, `Other` (BYO) | +| `--framework ` | `Strands`, `LangChain_LangGraph`, `GoogleADK`, `OpenAIAgents`, `VercelAI` | +| `--model-provider

` | `Bedrock`, `Anthropic`, `OpenAI`, `Gemini` | +| `--api-key ` | API key for non-Bedrock providers | +| `--memory ` | `none`, `shortTerm`, `longAndShortTerm` (create and import; see [Memory Shorthand Mapping](memory.md#--memory-shorthand-mapping)) | +| `--protocol ` | `HTTP` (default), `MCP`, `A2A`, `AGUI` | +| `--code-location ` | Path to existing code (BYO only) | +| `--entrypoint ` | Entry file relative to code-location (BYO, default: `main.py`) | +| `--network-mode ` | `PUBLIC` (default) or `VPC` | +| `--subnets ` | Comma-separated subnet IDs (required for VPC mode) | +| `--security-groups ` | Comma-separated security group IDs (required for VPC mode) | +| `--agent-id ` | Bedrock Agent ID (import only) | +| `--agent-alias-id ` | Bedrock Agent Alias ID (import only) | +| `--region ` | AWS region for Bedrock Agent (import only) | +| `--authorizer-type ` | Inbound auth: `AWS_IAM` or `CUSTOM_JWT` | +| `--discovery-url ` | OIDC discovery URL (for CUSTOM_JWT) | +| `--allowed-audience ` | Comma-separated allowed audiences (for CUSTOM_JWT) | +| `--allowed-clients ` | Comma-separated allowed client IDs (for CUSTOM_JWT) | +| `--allowed-scopes ` | Comma-separated allowed scopes (for CUSTOM_JWT) | +| `--custom-claims ` | Custom claim validations as JSON array (for CUSTOM_JWT) | +| `--client-id ` | OAuth client ID for agent bearer token | +| `--client-secret ` | OAuth client secret | +| `--request-header-allowlist ` | Comma-separated list of inbound header names to forward to the agent (X-prefixed names pass through; bare names are auto-prefixed with `X-Custom-`) | +| `--session-storage-mount-path ` | Absolute mount path for session filesystem storage (e.g. `/mnt/session-storage`) | +| `--with-config-bundle` | [preview] Wire a config bundle into the generated agent template | +| `--idle-timeout ` | Idle session timeout in seconds | +| `--max-lifetime ` | Max instance lifetime in seconds | +| `--json` | JSON output | ### add memory @@ -421,26 +505,40 @@ agentcore add credential \ ### add evaluator -Add a custom LLM-as-a-Judge evaluator. See [Evaluations](evals.md) for full details. +Add a custom evaluator. Two types are supported: `llm-as-a-judge` (default) and `code-based` (Lambda). See +[Evaluations](evals.md) for full details. ```bash +# LLM-as-a-Judge agentcore add evaluator \ --name ResponseQuality \ --level SESSION \ --model us.anthropic.claude-sonnet-4-5-20250514-v1:0 \ --instructions "Evaluate the response quality. Context: {context}" \ --rating-scale 1-5-quality + +# Code-based (existing Lambda) +agentcore add evaluator \ + --name LatencyCheck \ + --type code-based \ + --level TRACE \ + --lambda-arn arn:aws:lambda:us-east-1:123456789012:function:my-evaluator \ + --timeout 60 ``` -| Flag | Description | -| ------------------------- | -------------------------------------------------------------------------- | -| `--name ` | Evaluator name | -| `--level ` | `SESSION`, `TRACE`, or `TOOL_CALL` | -| `--model ` | Bedrock model ID for the LLM judge | -| `--instructions ` | Evaluation prompt with placeholders (e.g. `{context}`) | -| `--rating-scale ` | `1-5-quality`, `1-3-simple`, `pass-fail`, `good-neutral-bad`, or custom | -| `--config ` | Config JSON file (overrides `--model`, `--instructions`, `--rating-scale`) | -| `--json` | JSON output | +| Flag | Description | +| ------------------------- | ----------------------------------------------------------------------------- | +| `--name ` | Evaluator name | +| `--type ` | `llm-as-a-judge` (default) or `code-based` | +| `--level ` | `SESSION`, `TRACE`, or `TOOL_CALL` | +| `--model ` | [LLM] Bedrock model ID for the LLM judge | +| `--instructions ` | [LLM] Evaluation prompt with placeholders (e.g. `{context}`) | +| `--rating-scale ` | [LLM] `1-5-quality`, `1-3-simple`, `pass-fail`, `good-neutral-bad`, or custom | +| `--lambda-arn ` | [Code-based] Existing Lambda function ARN | +| `--timeout ` | [Code-based] Lambda timeout in seconds (1–300) | +| `--kms-key-arn ` | KMS key ARN for evaluator encryption (optional) | +| `--config ` | Config JSON file (overrides `--model`, `--instructions`, `--rating-scale`) | +| `--json` | JSON output | ### add online-eval @@ -461,9 +559,91 @@ agentcore add online-eval \ | `-e, --evaluator ` | Evaluator name(s), `Builtin.*` IDs, or ARNs | | `--evaluator-arn ` | Evaluator ARN(s) | | `--sampling-rate ` | Percentage of requests to evaluate (0.01–100) | +| `--endpoint ` | Runtime endpoint name to scope monitoring | | `--enable-on-create` | Enable immediately after deploy | | `--json` | JSON output | +### add policy-engine + +Add a Cedar policy engine to the project. Policy engines provide authorization for gateway requests using Cedar +policies. + +```bash +agentcore add policy-engine \ + --name MyPolicyEngine \ + --description "Authorization for production gateways" \ + --attach-to-gateways MyGateway,OtherGateway \ + --attach-mode ENFORCE +``` + +| Flag | Description | +| --------------------------------- | --------------------------------------------------------------- | +| `--name ` | Policy engine name | +| `--description ` | Policy engine description | +| `--encryption-key-arn ` | KMS encryption key ARN | +| `--attach-to-gateways ` | Comma-separated gateway names to attach this engine to | +| `--attach-mode ` | Enforcement mode for attached gateways: `LOG_ONLY` or `ENFORCE` | +| `--json` | JSON output | + +### add policy + +Add a Cedar policy to a policy engine. Policies can be authored inline, loaded from a file, or generated from a natural +language description. + +```bash +# From a Cedar policy file +agentcore add policy \ + --name AdminAccess \ + --engine MyPolicyEngine \ + --source ./policies/admin.cedar + +# Inline statement +agentcore add policy \ + --name DenyDelete \ + --engine MyPolicyEngine \ + --statement 'forbid(principal, action == Action::"Delete", resource);' + +# Generate from natural language (uses a deployed gateway as context) +agentcore add policy \ + --name ReadOnlyForGuests \ + --engine MyPolicyEngine \ + --generate "Allow guests to read but never write or delete" \ + --gateway MyGateway +``` + +| Flag | Description | +| -------------------------- | -------------------------------------------------------------------- | +| `--name ` | Policy name | +| `--engine ` | Policy engine name (must already exist) | +| `--description ` | Policy description | +| `--source ` | Path to a Cedar policy file | +| `--statement ` | Cedar policy statement (inline) | +| `-g, --generate ` | Generate Cedar policy from natural language description | +| `--gateway ` | Deployed gateway name for policy generation (used with `--generate`) | +| `--validation-mode ` | Validation mode: `FAIL_ON_ANY_FINDINGS` or `IGNORE_ALL_FINDINGS` | +| `--json` | JSON output | + +### add runtime-endpoint + +Add a named endpoint (version alias) to a deployed runtime. Endpoints let you address specific runtime versions by name +(e.g. `prod`, `staging`). + +```bash +agentcore add runtime-endpoint \ + --runtime MyAgent \ + --endpoint prod \ + --version 3 \ + --description "Production endpoint pinned to version 3" +``` + +| Flag | Description | +| ---------------------- | -------------------------------------- | +| `--runtime ` | Runtime to add the endpoint to | +| `--endpoint ` | Endpoint name (e.g. `prod`, `staging`) | +| `--version ` | Version number to alias (default: `1`) | +| `--description ` | Description of the endpoint | +| `--json` | JSON output | + ### remove Remove resources from project. @@ -476,18 +656,22 @@ agentcore remove evaluator --name ResponseQuality agentcore remove online-eval --name QualityMonitor agentcore remove gateway --name MyGateway agentcore remove gateway-target --name WeatherTools +agentcore remove policy-engine --name MyPolicyEngine +agentcore remove policy --name AdminAccess --engine MyPolicyEngine +agentcore remove runtime-endpoint --name prod # Reset everything agentcore remove all -y agentcore remove all --dry-run # Preview ``` -| Flag | Description | -| --------------- | ------------------------- | -| `--name ` | Resource name | -| `-y, --yes` | Skip confirmation | -| `--dry-run` | Preview (remove all only) | -| `--json` | JSON output | +| Flag | Description | +| ------------------- | ------------------------------------------------- | +| `--name ` | Resource name | +| `--engine ` | Policy engine name (required for `remove policy`) | +| `-y, --yes` | Skip confirmation | +| `--dry-run` | Preview (`remove all` only) | +| `--json` | JSON output | --- @@ -520,6 +704,8 @@ agentcore dev call-tool --tool myTool --input '{"arg": "value"}' | `--input ` | MCP tool arguments as JSON (with `--tool`) | | `-H, --header ` | Custom header (`"Name: Value"`, repeatable) | | `--exec` | Execute a shell command in the running dev container (Container only) | +| `-b, --no-browser` | Use terminal TUI instead of web-based chat UI | +| `--no-traces` | Disable local OTEL trace collection | ### invoke @@ -650,18 +836,22 @@ agentcore run eval \ --region us-east-1 ``` -| Flag | Description | -| ---------------------------- | ------------------------------------------- | -| `-r, --runtime ` | Runtime name from project | -| `--runtime-arn ` | Runtime ARN (standalone mode) | -| `-e, --evaluator ` | Evaluator name(s) or `Builtin.*` IDs | -| `--evaluator-arn ` | Evaluator ARN(s) (use with `--runtime-arn`) | -| `--region ` | AWS region (required with `--runtime-arn`) | -| `-s, --session-id ` | Evaluate a specific session | -| `-t, --trace-id ` | Evaluate a specific trace | -| `--days ` | Lookback window in days (default: 7) | -| `--output ` | Custom output file path | -| `--json` | JSON output | +| Flag | Description | +| ------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| `-r, --runtime ` | Runtime name from project | +| `--runtime-arn ` | Runtime ARN (standalone mode) | +| `-e, --evaluator ` | Evaluator name(s) or `Builtin.*` IDs | +| `--evaluator-arn ` | Evaluator ARN(s) (use with `--runtime-arn`) | +| `--region ` | AWS region (required with `--runtime-arn`) | +| `-s, --session-id ` | Evaluate a specific session | +| `-t, --trace-id ` | Evaluate a specific trace | +| `--endpoint ` | Runtime endpoint name (e.g. `PROMPT_V1`); defaults to `AGENTCORE_RUNTIME_ENDPOINT` env var, then `DEFAULT` | +| `--days ` | Lookback window in days (default: 7) | +| `-A, --assertion ` | Ground truth assertion the agent response must satisfy (repeatable) | +| `--expected-trajectory ` | Ground truth: expected tool call names in order (comma-separated) | +| `--expected-response ` | Ground truth: expected agent response text to compare against | +| `--output ` | Custom output file path | +| `--json` | JSON output | ### evals history @@ -767,17 +957,32 @@ agentcore package -d ./my-project ### update -Check for and install CLI updates. +Check for and install CLI updates. Equivalent to `agentcore update cli`. ```bash -agentcore update # Check and install -agentcore update --check # Check only, don't install +agentcore update # Check and install +agentcore update --check # Check only, don't install +agentcore update cli # Same as `agentcore update` +agentcore update cli --check # Same as `agentcore update --check` ``` | Flag | Description | | ------------- | ------------------------------------ | | `-c, --check` | Check for updates without installing | +### telemetry + +Manage anonymous usage analytics preferences. Telemetry is opt-in and used to improve the CLI. + +```bash +agentcore telemetry status # Show current preference and where it was set +agentcore telemetry enable # Opt in +agentcore telemetry disable # Opt out +``` + +`enable`, `disable`, and `status` take no flags beyond `-h, --help`. The preference is stored in your global CLI config +and persists across projects. + ### help Display help topics. diff --git a/docs/evals.md b/docs/evals.md index d12684a2a..d660d310d 100644 --- a/docs/evals.md +++ b/docs/evals.md @@ -33,26 +33,39 @@ a 1–5 numerical scale produces a normalized score of `0.60`. # Interactive (TUI wizard) agentcore add evaluator -# Non-interactive +# Non-interactive — LLM-as-a-Judge (default type) agentcore add evaluator \ --name ResponseQuality \ --level SESSION \ --model us.anthropic.claude-sonnet-4-5-20250514-v1:0 \ --instructions "Evaluate the agent response quality. Context: {context}" \ --rating-scale 1-5-quality + +# Non-interactive — Code-based (existing Lambda) +agentcore add evaluator \ + --name LatencyCheck \ + --type code-based \ + --level TRACE \ + --lambda-arn arn:aws:lambda:us-east-1:123456789012:function:my-evaluator \ + --timeout 60 ``` | Flag | Description | | ------------------------- | --------------------------------------------------------------------------------------- | | `--name ` | Evaluator name (alphanumeric + underscore, max 48 chars) | +| `--type ` | Evaluator type: `llm-as-a-judge` (default) or `code-based` | | `--level ` | Evaluation level: `SESSION`, `TRACE`, `TOOL_CALL` | -| `--model ` | Bedrock model ID for the LLM judge | -| `--instructions ` | Evaluation prompt (must include level-appropriate placeholders — see below) | -| `--rating-scale ` | Rating scale preset or custom format (default: `1-5-quality`) | +| `--model ` | [LLM] Bedrock model ID for the LLM judge | +| `--instructions ` | [LLM] Evaluation prompt (must include level-appropriate placeholders — see below) | +| `--rating-scale ` | [LLM] Rating scale preset or custom format (default: `1-5-quality`) | +| `--lambda-arn ` | [Code-based] Existing Lambda function ARN | +| `--timeout ` | [Code-based] Lambda timeout in seconds, 1–300 (default: 60) | +| `--kms-key-arn ` | KMS key ARN for evaluator encryption (optional) | | `--config ` | Path to evaluator config JSON (overrides `--model`, `--instructions`, `--rating-scale`) | | `--json` | JSON output | -> **Note**: `--instructions` is required in non-interactive mode unless `--config` is provided. +> **Note**: For LLM-as-a-Judge, `--instructions` is required in non-interactive mode unless `--config` is provided. For +> code-based evaluators, `--lambda-arn` is required. ### Instruction Placeholders @@ -173,18 +186,22 @@ agentcore run eval \ --days 7 ``` -| Flag | Description | -| ---------------------------- | -------------------------------------------------- | -| `-r, --runtime ` | Runtime name from project config | -| `--runtime-arn ` | Runtime ARN (standalone mode, no project required) | -| `-e, --evaluator ` | Evaluator name(s) from project or `Builtin.*` IDs | -| `--evaluator-arn ` | Evaluator ARN(s) (use with `--runtime-arn`) | -| `--region ` | AWS region (required with `--runtime-arn`) | -| `-s, --session-id ` | Evaluate a specific session only | -| `-t, --trace-id ` | Evaluate a specific trace only | -| `--days ` | Lookback window in days (default: 7) | -| `--output ` | Custom output file path | -| `--json` | JSON output | +| Flag | Description | +| ------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| `-r, --runtime ` | Runtime name from project config | +| `--runtime-arn ` | Runtime ARN (standalone mode, no project required) | +| `-e, --evaluator ` | Evaluator name(s) from project or `Builtin.*` IDs | +| `--evaluator-arn ` | Evaluator ARN(s) (use with `--runtime-arn`) | +| `--region ` | AWS region (required with `--runtime-arn`) | +| `-s, --session-id ` | Evaluate a specific session only | +| `-t, --trace-id ` | Evaluate a specific trace only | +| `--endpoint ` | Runtime endpoint name (e.g. `PROMPT_V1`); defaults to `AGENTCORE_RUNTIME_ENDPOINT` env var, then `DEFAULT` | +| `--days ` | Lookback window in days (default: 7) | +| `-A, --assertion ` | Ground truth assertion the agent response must satisfy (repeatable) | +| `--expected-trajectory ` | Ground truth: expected tool call names in order (comma-separated) | +| `--expected-response ` | Ground truth: expected agent response text to compare against | +| `--output ` | Custom output file path | +| `--json` | JSON output | > **Note**: Traces may take 5–10 minutes to appear after agent invocations. If a run returns no sessions, try increasing > `--days` or waiting for traces to propagate. @@ -249,6 +266,7 @@ agentcore add online-eval \ | `-e, --evaluator ` | Evaluator name(s), `Builtin.*` IDs, or ARNs | | `--evaluator-arn ` | Evaluator ARN(s) | | `--sampling-rate ` | Percentage of requests to evaluate (0.01–100) | +| `--endpoint ` | Runtime endpoint name to scope monitoring | | `--enable-on-create` | Enable immediately after deploy | | `--json` | JSON output | From 92838b88c4e1cc00450e14f10a37f4e00efa334c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9s=20Aguilar?= Date: Mon, 18 May 2026 19:14:49 -0500 Subject: [PATCH 2/2] docs: fix --request-header-allowlist auto-prefix value Per review feedback (#1296), the auto-prefix for bare header names without 'X-' is the legacy 'X-Amzn-Bedrock-AgentCore-Runtime-Custom-', not 'X-Custom-' as the table previously stated. The X-* pass-through half was already correct. Confirmed against normalizeHeaderName in src/cli/commands/shared/header-utils.ts and the HEADER_ALLOWLIST_PREFIX constant in src/schema/schemas/agent-env.ts. --- docs/commands.md | 66 ++++++++++++++++++++++++------------------------ 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/docs/commands.md b/docs/commands.md index 877990518..6d22b6bf0 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -284,39 +284,39 @@ agentcore add agent \ --memory none ``` -| Flag | Description | -| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--name ` | Agent name (alphanumeric + underscores, starts with letter, max 48 chars) | -| `--type ` | `create` (default), `byo`, or `import` | -| `--build ` | `CodeZip` (default) or `Container` (see [Container Builds](container-builds.md)) | -| `--language ` | `Python` (create); `Python`, `TypeScript`, `Other` (BYO) | -| `--framework ` | `Strands`, `LangChain_LangGraph`, `GoogleADK`, `OpenAIAgents`, `VercelAI` | -| `--model-provider

` | `Bedrock`, `Anthropic`, `OpenAI`, `Gemini` | -| `--api-key ` | API key for non-Bedrock providers | -| `--memory ` | `none`, `shortTerm`, `longAndShortTerm` (create and import; see [Memory Shorthand Mapping](memory.md#--memory-shorthand-mapping)) | -| `--protocol ` | `HTTP` (default), `MCP`, `A2A`, `AGUI` | -| `--code-location ` | Path to existing code (BYO only) | -| `--entrypoint ` | Entry file relative to code-location (BYO, default: `main.py`) | -| `--network-mode ` | `PUBLIC` (default) or `VPC` | -| `--subnets ` | Comma-separated subnet IDs (required for VPC mode) | -| `--security-groups ` | Comma-separated security group IDs (required for VPC mode) | -| `--agent-id ` | Bedrock Agent ID (import only) | -| `--agent-alias-id ` | Bedrock Agent Alias ID (import only) | -| `--region ` | AWS region for Bedrock Agent (import only) | -| `--authorizer-type ` | Inbound auth: `AWS_IAM` or `CUSTOM_JWT` | -| `--discovery-url ` | OIDC discovery URL (for CUSTOM_JWT) | -| `--allowed-audience ` | Comma-separated allowed audiences (for CUSTOM_JWT) | -| `--allowed-clients ` | Comma-separated allowed client IDs (for CUSTOM_JWT) | -| `--allowed-scopes ` | Comma-separated allowed scopes (for CUSTOM_JWT) | -| `--custom-claims ` | Custom claim validations as JSON array (for CUSTOM_JWT) | -| `--client-id ` | OAuth client ID for agent bearer token | -| `--client-secret ` | OAuth client secret | -| `--request-header-allowlist ` | Comma-separated list of inbound header names to forward to the agent (X-prefixed names pass through; bare names are auto-prefixed with `X-Custom-`) | -| `--session-storage-mount-path ` | Absolute mount path for session filesystem storage (e.g. `/mnt/session-storage`) | -| `--with-config-bundle` | [preview] Wire a config bundle into the generated agent template | -| `--idle-timeout ` | Idle session timeout in seconds | -| `--max-lifetime ` | Max instance lifetime in seconds | -| `--json` | JSON output | +| Flag | Description | +| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--name ` | Agent name (alphanumeric + underscores, starts with letter, max 48 chars) | +| `--type ` | `create` (default), `byo`, or `import` | +| `--build ` | `CodeZip` (default) or `Container` (see [Container Builds](container-builds.md)) | +| `--language ` | `Python` (create); `Python`, `TypeScript`, `Other` (BYO) | +| `--framework ` | `Strands`, `LangChain_LangGraph`, `GoogleADK`, `OpenAIAgents`, `VercelAI` | +| `--model-provider

` | `Bedrock`, `Anthropic`, `OpenAI`, `Gemini` | +| `--api-key ` | API key for non-Bedrock providers | +| `--memory ` | `none`, `shortTerm`, `longAndShortTerm` (create and import; see [Memory Shorthand Mapping](memory.md#--memory-shorthand-mapping)) | +| `--protocol ` | `HTTP` (default), `MCP`, `A2A`, `AGUI` | +| `--code-location ` | Path to existing code (BYO only) | +| `--entrypoint ` | Entry file relative to code-location (BYO, default: `main.py`) | +| `--network-mode ` | `PUBLIC` (default) or `VPC` | +| `--subnets ` | Comma-separated subnet IDs (required for VPC mode) | +| `--security-groups ` | Comma-separated security group IDs (required for VPC mode) | +| `--agent-id ` | Bedrock Agent ID (import only) | +| `--agent-alias-id ` | Bedrock Agent Alias ID (import only) | +| `--region ` | AWS region for Bedrock Agent (import only) | +| `--authorizer-type ` | Inbound auth: `AWS_IAM` or `CUSTOM_JWT` | +| `--discovery-url ` | OIDC discovery URL (for CUSTOM_JWT) | +| `--allowed-audience ` | Comma-separated allowed audiences (for CUSTOM_JWT) | +| `--allowed-clients ` | Comma-separated allowed client IDs (for CUSTOM_JWT) | +| `--allowed-scopes ` | Comma-separated allowed scopes (for CUSTOM_JWT) | +| `--custom-claims ` | Custom claim validations as JSON array (for CUSTOM_JWT) | +| `--client-id ` | OAuth client ID for agent bearer token | +| `--client-secret ` | OAuth client secret | +| `--request-header-allowlist ` | Comma-separated list of inbound header names to forward to the agent. `X-*` names (e.g. `X-Api-Key`, `X-Custom-Signature`) pass through unchanged; bare names without an `X-` prefix are auto-prefixed with the legacy `X-Amzn-Bedrock-AgentCore-Runtime-Custom-` prefix for backward compatibility. | +| `--session-storage-mount-path ` | Absolute mount path for session filesystem storage (e.g. `/mnt/session-storage`) | +| `--with-config-bundle` | [preview] Wire a config bundle into the generated agent template | +| `--idle-timeout ` | Idle session timeout in seconds | +| `--max-lifetime ` | Max instance lifetime in seconds | +| `--json` | JSON output | ### add memory