Skip to content

feat: Add A2A/MCP integration with OTEL tracing support - #187

Draft
yoavkatz wants to merge 83 commits into
mainfrom
feature/mcp-command
Draft

feat: Add A2A/MCP integration with OTEL tracing support#187
yoavkatz wants to merge 83 commits into
mainfrom
feature/mcp-command

Conversation

@yoavkatz

Copy link
Copy Markdown
Collaborator

Overview

This PR adds comprehensive Agent-to-Agent (A2A) and Model Context Protocol (MCP) integration to Exgentic, along with OpenTelemetry (OTEL) tracing support for distributed agent execution.

Key Features

1. A2A Protocol Integration

  • New CLI Command: exgentic a2a - Expose exgentic agents via the A2A protocol
  • A2A Executor (src/exgentic/adapters/agents/a2a_executor.py): Full agent-to-agent execution support with OTEL tracing
  • MCP Wrapper (src/exgentic/adapters/agents/mcp_wrapper.py): Wrapper for MCP server integration with A2A agents

2. MCP Server Enhancements

  • New CLI Command: exgentic mcp - Expose benchmark actions as MCP tools
  • Dynamic Session Management: Support for multiple sessions with different tasks
  • Evaluation Endpoint: Added evaluate_session endpoint to MCP server
  • Configurable Options: DNS rebinding protection toggle, benchmark-specific parameters via --set

3. OpenTelemetry Tracing

  • A2A Span Filtering: Filter SDK spans and prevent invalid parent span IDs
  • Root Span Metadata: Add OTEL root span metadata to A2A executor
  • Enhanced OTEL Utils (src/exgentic/utils/otel.py): Improved tracing utilities with 183+ lines of additions

4. Performance Testing & Monitoring

  • A2A Test Harness (misc/performance/test_a2a_agent.py): Comprehensive test harness with 934 lines for A2A agent evaluation
  • MCP Memory Test (misc/performance/test_mcp_memory_harness.py): Performance testing for MCP memory usage (545 lines)
  • Parallel Session Timing (misc/performance/time_parallel_gsm8k_create_session.py): Timing script for parallel MCP session creation (223 lines)

5. Bug Fixes & Improvements

  • Session Management: Fixed session closure issues in evaluate_session
  • Serialization: Fixed A2A serialization errors with custom Pydantic pickle handler
  • Process Cleanup: Improved process cleanup with kill() fallback
  • Lock Contention: Reduced MCP session lock contention
  • HTTP Client: Handle closed HTTP client in evaluate_session
  • Timeout Handling: Extended timeout for MCP client to prevent connection closure

Documentation

  • CLAUDE.md: Added comprehensive guide for Claude Code integration (116 lines)
  • Updated architecture documentation with A2A/MCP adapter details

Files Changed

  • 16 files changed: 4,041 insertions(+), 643 deletions(-)
  • New files: 7 major new files including CLI commands, adapters, and test harnesses
  • Modified files: Enhanced OTEL handlers, transport layer, and MCP server

Testing

  • Added comprehensive test harnesses for A2A and MCP performance
  • Memory monitoring and evaluation capabilities
  • Parallel session creation timing tests

Breaking Changes

None - all changes are additive and backward compatible.

Related Issues

Closes #[issue-number] (if applicable)

Checklist

  • Code follows project conventions (CLAUDE.md)
  • All new files have proper SPDX headers
  • Dependencies have version caps
  • Tests added for new functionality
  • Documentation updated
  • Commits signed off (DCO)

yoavkatz added 30 commits March 9, 2026 10:33
- Created new 'exgentic mcp' CLI command
- Accepts --benchmark, --task-id, --subset, --host, --port options
- Dynamically generates tool signatures from action argument schemas
- Includes timeout protection for tool execution (30s)
- Calls session.start() for proper initialization
- Registered command in CLI main.py under Tools category

Usage: exgentic mcp --benchmark <name> --task-id <id>
Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
…command

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Remove --task-id command line parameter
- Add list_tasks endpoint to return available benchmark tasks
- Add create_session(task_id) endpoint to create sessions on-demand
- Add delete_session(task_id) endpoint to close and delete sessions
- Store and propagate context to tool functions for thread safety
- Update call_mcp_tool.py example to demonstrate new workflow

Sessions are now managed dynamically via MCP endpoints instead of
being created at server startup. This allows clients to create and
destroy sessions as needed during runtime.

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Added --set option to allow passing benchmark-specific parameters
- Only benchmark.* parameters are allowed (e.g., benchmark.user_simulator_model)
- Validates parameters against benchmark's accepted kwargs
- Example: --set benchmark.user_simulator_model='openai/Azure/gpt-4o'

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Added *.svc.cluster.local:* to the allowed hosts and origins lists (lines 63 and 71). This will allow the MCP server to accept connections from Kubernetes services with hostnames like exgentic-mcp-tau2-mcp.team1.svc.cluster.local:8000.

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Add --disable-dns-rebinding-protection flag to allow MCP server to accept
connections from any host, including Kubernetes services. This is useful
for deployments in trusted environments like Kubernetes clusters where
the MCP library's wildcard hostname matching doesn't work.

When enabled (default), DNS rebinding protection validates Host and Origin
headers. When disabled, all hosts are allowed, fixing 'Invalid Host header'
warnings for Kubernetes service DNS names.

Changes:
- Add enable_dns_rebinding_protection parameter to MCPServer
- Add --disable-dns-rebinding-protection CLI flag to mcp command
- Remove specific Kubernetes hostname patterns (too specific)
- Keep Docker/container hostname patterns when protection is enabled

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Changed MCP server to use UUID-based session_id instead of task_id
- Updated action tools to accept session_id parameter
- Updated delete_session to use session_id parameter
- Modified sessions dictionary to be keyed by session_id (UUID)
- Updated example code to extract and use session_id
- Allows multiple concurrent sessions per task
- Tested and verified with call_mcp_tool.py

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Added evaluate_session tool that calls session.score()
- Returns success status and score from session evaluation
- Updated example to test evaluate_session functionality
- Tool accepts session_id parameter and returns evaluation results

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- SessionScore is a Pydantic model, not a dict
- Access fields directly (score_result.success, score_result.score)
- Automatically closes session if not done before evaluating
- Returns all SessionScore fields in response

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
… method

- Add kill() fallback after terminate() timeout in RemoteProcessExecuter
- Add kill() fallback in RemoteProcess.close() and _cleanup_resources()
- Enhance RemoteSession.close() with proper validation before calling remote close()
- Restore and document RemoteSession.shutdown() method for forceful termination
- Ensure processes are properly cleaned up even when unresponsive to terminate()

This prevents zombie processes and resource leaks when remote processes
become unresponsive during shutdown.

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Resolved conflicts:
- Merged CLI command groups: kept both 'Tools' (mcp) and 'Infrastructure' (serve)
- Added all commands from both branches: install_cmd, uninstall_cmd, mcp_cmd, serve_cmd
- Removed deprecated executor files (executer.py, remote_process_class.py) in favor of new runners architecture
- Updated __all__ exports to include both mcp_cmd and new commands from main

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Use benchmark.get_evaluator() to access list_tasks() method
- Use evaluator.get_session_kwargs() and benchmark.get_session() to create sessions
- Fixes 'GSM8kBenchmark' object has no attribute 'list_tasks' error
- Fixes 'GSM8kBenchmark' object has no attribute 'create_session' error

Tested successfully with gsm8k benchmark - server starts and loads 1319 tasks with 2 action types

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Add new 'exgentic a2a' CLI command with --agent, --mcp, --host, --port, --set parameters
- Create mcp_wrapper.py to connect to external MCP servers and extract tools
- Create a2a_executor.py to run exgentic agents via A2A protocol (JSON-RPC 2.0)
- Implement A2A server with agent card, task execution, and event streaming
- Add a2a-sdk>=0.2.16 as optional dependency in pyproject.toml
- Integrate a2a command into main CLI interface

The command allows exgentic agents to be exposed as A2A-compatible agents,
enabling agent-to-agent communication. It connects to external MCP servers
to extract tools, creates an exgentic agent instance with those tools, and
exposes it via the A2A protocol for other agents to interact with.

Tested and verified:
- Agent card discovery at /.well-known/agent-card.json
- JSON-RPC 2.0 request/response handling
- Task execution with proper context management
- Error reporting via A2A protocol with artifacts
- Event streaming for task status updates

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Add new 'exgentic a2a' CLI command to expose exgentic agents as A2A servers
- Implement MCP wrapper to extract tool metadata from external MCP servers
- Create A2A executor that converts MCP tools to ActionTypes and handles execution
- Use ThreadPoolExecutor to run synchronous agent.react() without blocking
- Handle both tool calls and message actions (agent's final responses)
- Add a2a-sdk>=0.2.16 as core dependency
- Remove dead code (_dynamic_actions.py) and unused imports

The command syntax: exgentic a2a --agent <name> --mcp <address> [--host <host>] [--port <port>] [--set <key=value>]

Tested successfully with LiteLLM Tool Calling agent connecting to external MCP server.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Created misc/performance/test_a2a_agent.py test harness
  - Connects to MCP server to fetch tasks and create sessions
  - Calls A2A agent server to solve tasks using proper A2A client API
  - Monitors memory consumption of A2A server process
  - Tracks task execution time and success/failure rates
  - Calls evaluate_session after each successful task completion
  - Pretty-prints evaluation results (success, score, metrics)
  - Provides summary statistics including evaluation metrics

- Enhanced A2A executor logging
  - Prints log location when each request arrives
  - Shows: outputs/a2a_<timestamp>/ directory path

- Improved debug output
  - Pretty-prints JSON responses with indentation
  - For status updates, shows only the text message (not full JSON)
  - For other events, shows full details
  - Configurable httpx client timeout
  - Proper resource cleanup (closes httpx client)

Usage:
  python misc/performance/test_a2a_agent.py \
    --mcp-url http://localhost:8000/mcp \
    --a2a-url http://localhost:9000 \
    --limit 5 --timeout 600
Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Fixed A2A executor to handle ParallelAction properly
  - Use to_action_list() for all action types (works uniformly)
  - Execute all actions in the list
  - Return SingleObservation for single actions
  - Return MultiObservation for multiple actions (follows core/actions.py pattern)
  - Each SingleObservation in MultiObservation has its own invoking action

- Enhanced task input with session_id instructions
  - Explicitly tells agent to use session_id in all tool calls
  - Reminds agent to call submit MCP tool when needed
  - Improves task completion rates

Fixes error: 'ParallelAction' object has no attribute 'name'

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Extract context from create_session response and append it to the
task input sent to the A2A agent. This provides the agent with
additional context information that may be needed to solve the task.

Format:
- Task description
- Context: (if available)
  - key: value
  - key: value
- Session ID instructions

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Created misc/performance/test_a2a_agent.py: comprehensive test harness for A2A agents
  - Connects to MCP server and A2A agent server
  - Creates sessions and calls A2A agent to solve tasks
  - Evaluates results and tracks success/failure metrics
  - Monitors memory consumption and execution time
  - Added --debug flag for detailed output including evaluation responses

- Fixed src/exgentic/adapters/agents/a2a_executor.py:
  - Mark message action with is_message=True flag for proper agent handling
  - Auto-inject session_id parameter when calling message tool
  - Extract session_id from task context instead of generating random UUID
  - Added error handling for missing session_id in task context

- Updated src/exgentic/adapters/agents/mcp_wrapper.py:
  - Filter out admin tools (create_session, delete_session, list_tasks, evaluate_session)
  - Agents now only see task-specific tools, preventing session management conflicts

These changes ensure proper session management and enable comprehensive testing
of A2A agents solving MCP server tasks.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
The MCP HTTP client was timing out during long-running A2A agent tasks,
causing 'Cannot send a request, as the client has been closed' errors
when trying to evaluate sessions after task completion.

Solution: Create MCP client with extended timeout (2x task timeout) to
ensure the connection stays alive throughout the entire task execution
and evaluation process.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Make memory monitoring optional when the A2A server PID cannot be detected,
which commonly occurs when the server runs in a container. The test now:

- Attempts to auto-detect the server PID by port
- Falls back to running without memory monitoring if PID not found
- Adds conditional checks before all monitor method calls
- Provides clear warnings when memory monitoring is disabled

This allows the test harness to work with both local and containerized
A2A servers, gracefully degrading functionality when process monitoring
is not available.

Changes:
- Initialize monitor to None by default
- Skip PID detection errors and continue without monitoring
- Wrap all monitor.measure() and monitor.print_*() calls with 'if monitor:' checks
- Update status messages to indicate when monitoring is disabled

Signed-off-by: Yoav Katz <katz@il.ibm.com>
…st harness

This commit fixes the 'Cannot send a request, as the client has been closed'
error that occurred when evaluating sessions after A2A agent execution.

Root cause:
- A2A executor creates its own MCP session with an HTTP client
- When A2A agent completes, it closes its MCP session and HTTP client
- Benchmark session stores reference to A2A's HTTP client
- evaluate_session tries to close the session, which attempts to use the
  already-closed HTTP client from the A2A executor

Changes:
1. MCP Server (src/exgentic/interfaces/cli/commands/mcp.py):
   - Wrap sess.close() in try-except to handle already-closed clients
   - Add full traceback to error messages for better debugging

2. Test Script (misc/performance/test_a2a_agent.py):
   - Move httpx client creation outside try block for proper lifecycle
   - Set timeout=None for read operations to prevent premature closure
   - Add finally block to ensure HTTP client cleanup
   - Rename 'session' to 'mcp_session' for clarity
   - Improve error handling and debug output for evaluation

3. A2A Command (src/exgentic/interfaces/cli/commands/a2a.py):
   - Move imports to top of file (code cleanup)

The evaluation can now complete successfully even when the A2A agent has
already closed its connection to the benchmark session.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
…client closure

Root cause analysis:
- Tau2 benchmark uses service runner which wraps sessions in HTTP transport
- When sess.close() is called, it closes the HTTPTransport's httpx.Client
- Calling sess.score() after close() fails with 'client has been closed' error
- The session's score() method needs the HTTP transport to be open to make RPC calls

Solution:
- Reorder operations in evaluate_session_tool to call score() BEFORE close()
- This ensures the HTTP transport is still available when score() needs it
- Add detailed comment explaining why this order is critical

This fixes the evaluation error for benchmarks using service runner (tau2, etc.)
while maintaining backward compatibility with local sessions (gsm8k, etc.).

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Refactored the A2A agent test harness for better maintainability:

Changes:
- Split large test_a2a_agent() function into smaller, focused functions:
  * fetch_tasks() - Fetch tasks from MCP server
  * create_mcp_session() - Create MCP session for a task
  * build_enhanced_task_input() - Build task input with context
  * evaluate_mcp_session() - Evaluate session results
  * delete_mcp_session() - Delete session
  * print_task_results_summary() - Print results summary

- Simplified call_a2a_agent():
  * Removed unnecessary timeout parameter from httpx client
  * Removed verbose debug logging
  * Cleaner error handling with finally block
  * Simplified response processing

- Removed unnecessary timeout settings added during debugging
- Improved code organization and readability
- Maintained all functionality while reducing complexity
- Better separation of concerns

The refactored code is easier to understand, test, and maintain.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
…agent.py

- Replace all 'if debug:' print statements with logger calls
- Remove debug parameters from all functions (call_a2a_agent, fetch_tasks, create_mcp_session, evaluate_mcp_session, delete_mcp_session, test_a2a_agent)
- Configure module-specific logger that only affects this file
- Use appropriate log levels: debug, info, warning, error, exception
- Restore comprehensive debug output for A2A agent communication
- Clean up function signatures by removing debug parameter pollution

Benefits:
- Standard Python logging best practices
- Module-specific logging (doesn't affect other loggers)
- Proper log levels and automatic exception tracebacks
- Cleaner, more maintainable code

Signed-off-by: Yoav Katz <katz@il.ibm.com>
yoavkatz and others added 7 commits May 2, 2026 23:06
- Change default timeout from 60s to 30s in check_model_accessible_sync()
- Disable retries by setting _HEALTH_MIN_RETRIES from 7 to 0
- This prevents long hangs when model endpoints are misconfigured or unreachable
- Particularly affects tau2 benchmark initialization which checks user simulator model

The tau2 benchmark was hanging at 'Initializing action types...' because the
health check for the user simulator model (openai/Azure/gpt-4.1) was timing
out after 60+ seconds with multiple retries. With these changes, failures
occur within 30 seconds, providing faster feedback to users.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Add _remove_session_id_from_action_types() method to strip session_id from action schemas
- Pass cleaned action types (without session_id) to agent initialization
- Inject session_id into all MCP tool calls at execution time
- This allows session_id to be managed separately from agent input while maintaining compatibility with MCP tools that require it

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Instead of always returning the generic "Session completed" string,
extract the actual action arguments when the completing action is a
message or finish action. Also bump max_iterations from 50 to 100.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@zeroasterisk

Copy link
Copy Markdown
Contributor

Hi @yoavkatz — we've been working on the consume-side A2A adapter (PR #232) while your PR covers the serve-side. @elronbandel suggested we coordinate, and we agree — together these make Exgentic a full A2A citizen (both producer and consumer).

A few areas where alignment would help:

  1. Shared A2A types/utilities — both PRs translate between Exgentic types and A2A types. Could share translation helpers (action→tool, observation→message, etc.) rather than duplicating.

  2. MCP handshake convention — we've opened issue Design: Task-scoped MCP provisioning convention for A2A agents #236 proposing a structured-metadata convention for passing MCP endpoints in A2A task messages. This affects how consume-side agents discover their tools. Would value your input since the serve-side would need to understand this convention too.

  3. Agent Card extensions — for declaring A2A+MCP capabilities (e.g., "I accept MCP tool provisioning"). Relevant to both directions.

Happy to align on shared structure however works best for you — issue discussion, shared module, or a quick call.

- Add agent_a2a_images: Docker wrapper for Exgentic agents using A2A protocol
  - Dockerfile with build-time agent installation
  - Build script with docker/podman auto-detection and GHCR push support
  - Entrypoint with runtime configuration via environment variables
  - Comprehensive README with usage examples

- Add benchmark_mcp_images: Docker wrapper for Exgentic benchmarks using MCP
  - Dockerfile with build-time benchmark installation
  - Build script with docker/podman auto-detection and GHCR push support
  - Entrypoint with runtime configuration via environment variables
  - Comprehensive README with usage examples

Both implementations include:
- Non-root user execution (UID 1001/1000)
- Flexible runtime configuration via EXGENTIC_SET_* environment variables
- Support for pushing to GitHub Container Registry
- Production-ready error handling and logging
- .dockerignore for optimized builds
- Example environment files

Signed-off-by: Yoav Katz <katz@il.ibm.com>
yoavkatz and others added 16 commits July 7, 2026 17:23
…P startup

Three bugs prevented the AppWorld benchmark from starting via `exgentic mcp`:

1. Pipe buffer deadlock (root cause): VenvRunner captured subprocess
   stdout/stderr via pipes but never drained them while polling the health
   endpoint. AppWorld imports emit ~65KB of StarletteDeprecationWarnings at
   module-load time, filling the 64KB OS pipe buffer and stalling the
   subprocess before uvicorn could start. Fix: add background drain threads
   (_start_drain_threads) that continuously read both pipes. The drained
   output is included in timeout/crash error messages for diagnostics.

2. Missing HOME forwarding: prepare_subprocess_env() stripped all env vars
   except LLM provider credentials. Python's Path.home() and EnvironmentManager
   both resolve benchmark data directories from HOME (~/.exgentic). Without it
   the subprocess couldn't locate installed benchmark data. Fix: always forward
   HOME in prepare_subprocess_env().

3. httpx2 missing from appworld venv: Starlette's test client (used by
   AppWorld's internal FastAPI apps) warns that httpx is deprecated and requires
   httpx2. Fix: add httpx2 to appworld/requirements.txt.

Additional improvements to VenvRunner:
- Use start_new_session=True + os.killpg for clean process group teardown
- Replace _wait_for_health with _wait_for_health_with_process_check that
  raises _ProcessExitedError immediately when the subprocess crashes, rather
  than waiting the full timeout before reporting the error

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
fix(venv): drain subprocess pipes and forward HOME to fix appworld MCP startup
VenvRunner passes is_alive to HTTPTransport to detect dead subprocesses
fast, but HTTPTransport.__init__ did not accept the parameter.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Color helpers, detect_runtime, login_to_ghcr, and the core build/push
logic are factored into misc/build_lib.sh and sourced by both
benchmark_mcp_images/build.sh and agent_a2a_images/build.sh.
The agent script gains --multiplatform support and login-before-build
for free from the shared library.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
- Add agent.action_timeout --set parameter to mcp command (default 30s)
- Pass timeout value through make_action_tool into the step thread join
- Return the raw timeout error string instead of a hardcoded message
- Replace silent "Session completed (timeout)" with the actual error
  message so the cause is visible in traces and logs

Fixes: rossoctl/workload-harness#31
Signed-off-by: Yoav Katz <katz@il.ibm.com>
…able

fix(mcp): make action timeout configurable and surface it as an error
…idation

The entrypoint translates EXGENTIC_SET_BENCHMARK_ACTION_TIMEOUT to
--set benchmark.action_timeout, so it arrives as group=benchmark.
Intercept it before _validate_set_keys_for_benchmark to avoid
'Unknown benchmark override' error.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
Add check_git_clean to build_lib.sh that fails if there are uncommitted
changes, if the current branch doesn't match the branch in the Dockerfile,
or if there are unpushed commits. Call it in both build.sh scripts before
the build starts.

Also carry forward misc/benchmark_mcp_images/Dockerfile and build_lib.sh
fixes (--local flag, --network=host for multiplatform push).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
The appworld setup.sh ran `uv pip install "."` unconditionally. In the
benchmark Docker image (and `exgentic install --local` into a system
Python) there is no active virtualenv, and build_subprocess_env() strips
UV_* vars, so UV_SYSTEM_PYTHON=1 from the Dockerfile does not reach the
script. uv then aborts with 'No virtual environment found'.

Wrap the installs in a helper that passes --system when VIRTUAL_ENV is
unset, so it works both under the venv runner and in the container.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Add a prune_build_space helper to the shared build lib that prunes the
buildx cache and dangling images, then reports freed and remaining disk
usage. Multi-platform builds export the image to a tarball and re-import
it, needing a large slab of temporary disk; a full VM is the most common
cause of those builds failing. Call it before building in both the MCP
benchmark and A2A agent build scripts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
A tool call (e.g. submit rejecting unparsable LLM output) can raise
mid-loop. The agent keeps looping to recover, but if such an error is
the last thing to happen before the loop ends, the run was previously
reported as completed — delivering the error text as a normal artifact
that downstream (A2A runner, MLflow) recorded as success.

Track the terminating exception in last_tool_error, clearing it once a
later step succeeds. On finalization, if it is set, record the exception
on the span, call on_session_error, and end the A2A task in the failed
state instead of the success path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
…ipts in clean check

Multiplatform+push now builds to an OCI archive and pushes with skopeo
copy --all, avoiding the docker buildx registry push path. Errors out
early if skopeo is missing, and cleans up the temp tarball on both the
push-failure and build-failure paths.

The git-clean guard also stops flagging build.sh/build_lib.sh/Dockerfile
so local edits to the build tooling don't block a build.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
… annotations

Bake the source revision into built images so they trace back to the
commit they were built from. build_lib.sh resolves the local HEAD (which
check_git_clean guarantees equals the pushed branch tip the Dockerfile
clones) and passes it as GIT_COMMIT / GIT_BRANCH build args.

The Dockerfiles set standard OCI config labels (revision, version,
source, title, description, url), overwriting the labels inherited from
the uv base image. Config labels are per-platform, though, and GHCR's
package page and multi-arch consumers read the manifest index instead —
so build_lib.sh also passes matching buildx --annotation index:... flags,
which skopeo copy --all preserves onto the pushed index.

Inspect with: docker inspect --format '{{ json .Config.Labels }}' <image>
or: skopeo inspect --raw docker://<image> | jq .annotations

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
((SUCCESS_COUNT++)) with SUCCESS_COUNT starting at 0 evaluates to the
pre-increment value (0), which bash treats as a failed command under
`set -e` -- so the script exits immediately right after a successful
build, before ever printing the build summary, and reports exit code 1
despite the build having actually succeeded. Same latent bug applies to
FAIL_COUNT on the first real failure. Switched both to plain arithmetic
assignment ($((x + 1))), which has no such trap.

Found while building exgentic-mcp-tau2 for a downstream integration --
the image built fine (confirmed via `podman images`) but the script
reported failure with no error message, right after printing
"Successfully built ...".

Signed-off-by: Vitaly Zabershinsky <VITALYZ@il.ibm.com>
fix(build): build.sh reports false failure on successful builds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants