Skip to content

feat(analyze-run): support OpenShift MLflow and time-window trace fetch - #5

Open
yoavkatz wants to merge 49 commits into
mainfrom
feat/analyze-run-openshift-window
Open

feat(analyze-run): support OpenShift MLflow and time-window trace fetch#5
yoavkatz wants to merge 49 commits into
mainfrom
feat/analyze-run-openshift-window

Conversation

@yoavkatz

@yoavkatz yoavkatz commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

Rework analyze-run.sh to reach MLflow via kubectl port-forward for both --kind and --openshift, mirroring how evaluate-benchmark.sh reaches the OTEL collector. MLflow location, TLS, workspace, auth mode, and experiment id now default per cluster mode, with env-var and CLI-flag overrides.

Changes

  • Cluster modes — add --kind (default) and --openshift DOMAIN, each with per-mode defaults (namespace, service, remote port, TLS, workspace, auth mode, experiment id).
  • Auth — add --auth-mode (secret | oc-token); oc-token uses oc whoami -t for RHOAI's oauth-proxy.
  • Time window — replace the --limit trace count with --window (e.g. 3h, 90m, 2d). The downloader pages newest-first and stops at the first out-of-window trace.
  • Short-trace filter — skip trivially short traces (MIN_DURATION_S, default 0.1s) before fetching their spans.
  • RHOAI support — send the x-mlflow-workspace header and allow insecure TLS for port-forwarded reencrypt HTTPS endpoints.
  • Context check — verify the kubectl context matches the cluster mode via the libsh/check-kubectl-context.sh helper.
  • Robustness — harden the secret-mode token flow against set -e command-substitution aborts.

Test plan

  • bash -n analyze-run.sh — passes
  • python3 -m py_compile download_mlflow_traces.py — passes
  • Manual --kind --window ... run against a kind cluster
  • Manual --openshift DOMAIN run against RHOAI

Review notes (non-blocking)

  • INGRESS_DOMAIN is required by --openshift but not yet consumed (port-forward + oc whoami -t path) — appears to be a forward-looking placeholder.
  • The --openshift empty-domain check is duplicated (arg-parse and mode dispatch).
  • The Python TLS opt-out uses ssl._create_unverified_context() (private API) — justified and documented for the localhost reencrypt cert.

🤖 Generated with Claude Code

kellyaa and others added 30 commits May 13, 2026 11:54
@kellyaa
First IBAC integraiton
5ede588 
Signed-off-by: Kelly Abuelsaad <kna@us.ibm.com>
@kellyaa
Externalize IBAC prompt
7d57067 
Signed-off-by: Kelly Abuelsaad <kna@us.ibm.com>
@kellyaa
Migrate IBAC to authbridge plugin pipeline
d0b95de 
Drop the standalone IBAC sidecar (forked authbridge image + Envoy +
iptables init container) in favor of patching the operator-injected
authbridge sidecar's plugin pipeline. The new flow appends a2a-parser
(inbound) and inference-parser/mcp-parser/ibac (outbound) into the
operator-managed authbridge-config-<agent> ConfigMap, and waits for
the sidecar's filesystem-watch hot-reload to swap pipelines.

Also wires --ibac to authBridgeEnabled=true in the kagenti API call
so the operator injects the sidecar (and creates the ConfigMap) in
the first place.

Removes envoy-config.yaml and patch-deployment.yaml (legacy stack);
adds ibac-patch.yaml (envsubst template), ibac-merge.py (idempotent
ConfigMap merge with --prompt-file), and wait-for-reload.sh.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kna@us.ibm.com>
@kellyaa
Merge remote-tracking branch 'origin/main' into feat/experimental_ibac
70da607 
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
Update --ibac help text to plugin-pipeline language
8ea1703 
Stale leftover from before the migration commit; the flag no longer
injects an Envoy overlay sidecar.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kna@us.ibm.com>
@kellyaa
fix(otel): point agent OTLP endpoint at otel-collector port 8335
6b45488 
The kagenti-deps otel-collector binds OTLP/HTTP on 8335 (and gRPC on
4317); nothing is listening on 4318, so requests to 4318 return 503
and crash the agent's strict OTEL startup probe.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kna@us.ibm.com>
@yoavkatz
feat: add standalone --authbridge flag and rename session_id variable
b682480 
Allow authbridge sidecar to be enabled independently of IBAC via
--authbridge/--no-authbridge flags and AUTHBRIDGE_ENABLED env var.
Previously authbridge was only activated implicitly by --ibac. This
decouples the two so auth/routing can be used without the IBAC plugin.

Also rename the local `session_id` variable in a2a_client.py to
`x_session_id` to avoid shadowing and clarify its purpose as the
x-session-id header value.

Signed-off-by: Yoav Katz <yoavkatz@gmail.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
Merge pull request #1 from yoavkatz/feat/ibac
157e887 
feat: add standalone --authbridge flag
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
feat: replace --ibac/--authbridge with composable plugin pipeline flags
b8d5f67 
Re-architect the deploy scripts to drive AuthBridge as a generic plugin
pipeline. The old --ibac and --authbridge flags treated the sidecar as
two opaque toggles; AuthBridge has since moved to independently
composable plugins (jwt-validation, token-exchange, token-broker,
a2a-parser, mcp-parser, inference-parser, ibac) with per-plugin
on_error policies (enforce/observe/off).

Spec: exgentic_a2a_runner/AUTHBRIDGE_PIPELINE_SPEC.md.

CLI surface (deploy-agent.sh and deploy-and-evaluate.sh):
  --plugin-preset {auth-only|ibac-only|full}
  --plugin <name>[:enforce|observe|off]   (repeatable)
  --no-plugin <name>                      (shorthand for :off)
  --plugin-config-file <path>             (flat-map per-plugin overrides)

Removed: --authbridge, --no-authbridge, --ibac, --no-ibac flags and
the IBAC_ENABLED / AUTHBRIDGE_ENABLED env-var aliases. No back-compat
shim — old call sites fail loudly with "unknown option".

Implementation:
  - Rename ibac/ -> authbridge/. Split ibac-patch.yaml into per-plugin
    fragments under plugins/ and three preset YAMLs.
  - Replace ibac-merge.py with pipeline-merge.py: emits every
    supported plugin (active ones with their resolved policy, others
    with on_error: off so the framework skips dispatch — required
    because the operator base config enables every plugin by default).
  - Replace apply-ibac.sh with apply-pipeline.sh: renders fragments
    via envsubst, runs the merge, validates the token-exchange /
    token-broker mutex, applies, waits for reload.
  - The resolver in deploy-agent.sh delegates to embedded Python so
    it works under macOS bash 3.2 (no associative arrays).

Sidecar injection (authBridgeEnabled in the operator API call) now
fires when ANY plugin selector is supplied; omit them all for a
sidecar-free deployment.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kna@us.ibm.com>
@kellyaa
feat(ibac): wire judge bearer, auto-load .env, and harden agent startup
a121aad 
- apply-pipeline.sh / ibac.yaml: thread JUDGE_BEARER (falling back to
  OPENAI_API_KEY) into the IBAC plugin config so the judge endpoint
  authenticates correctly.
- deploy-agent.sh: auto-source the script-local .env so IBAC_* values
  reach apply-pipeline.sh without the caller pre-sourcing; existing
  shell exports still win.
- deploy-agent.sh: set LITELLM_LOCAL_MODEL_COST_MAP=True to stop
  litellm's startup fetch of the remote pricing JSON, which IBAC was
  rejecting with no_session/no_intent before any request landed.
- intent_prompt.txt: always allow raw.githubusercontent.com so legitimate
  static-config fetches during init aren't blocked.
- deploy-agent.sh: forward JUDGE_BEARER and OPENAI_API_KEY into the
  authbridge pipeline invocation.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kna@us.ibm.com>
@kellyaa
Merge pull request rossoctl#18 from kellyaa/feat/ibac
b07f729 
feat: IBAC integration via authbridge plugin pipeline
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
docs(exgentic): document AuthBridge plugin pipeline and IBAC
f75287e 
Add a Getting Started section near the top consolidating the cluster
setup, image build, runner install, and deploy-and-evaluate commands
(plain, auth-only, IBAC enforcing, IBAC observe canary).

Promote the AuthBridge plugin pipeline material out of Installation
into its own top-level section with architectural framing: sidecar is
opt-in per deploy, plugins mediate every inbound/outbound request,
plugins share a Session object with hard runtime dependencies (e.g.
IBAC needs a2a-parser), and selector resolution is last-write-wins
over a base config that enables every plugin by default.

Add a Running with IBAC subsection covering the judge prerequisites
(OpenAI-compatible endpoint, sidecar image >= v0.6.0-alpha.4), the
.env knobs (IBAC_JUDGE_ENDPOINT, IBAC_JUDGE_MODEL, IBAC_TIMEOUT_MS,
IBAC_AGENT_LLM_HOST), and three concrete deploy invocations: enforce,
observe canary, and ibac-only (for environments where a fronting
gateway already terminates auth).

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kaymar@gmail.com>
@kellyaa
docs(exgentic): add Benchmarks section and model-name guidance
0428700 
Add a top-level Benchmarks section between Architecture and
Installation summarizing the three benchmarks the runner currently
drives (gsm8k, tau2, appworld) - what each tests, the tool surface,
and per-benchmark deploy notes (gsm8k's direct runner, tau2's
user-simulator model passthrough, appworld's tool-surface limitation
that excludes OpenAI models).

Document LiteLLM's provider/model routing convention with concrete
guidance for OpenAI-compatible backends (vLLM, Ollama, custom proxies):
prefix with openai/ to force the OpenAI-compatible route, e.g.
"openai/Azure/gpt-4o-mini" passed via --model, and point
OPENAI_API_BASE at the endpoint.

Replace the standalone appworld-needs-gemini sentence with a pointer
to the new section.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kaymar@gmail.com>
@kellyaa
docs(readme): preload images in setup, document alpha.7 prerequisite
dd9059b 
- Apply review suggestion to add --preload-images to
  setup-kagenti.sh invocations in both Getting Started and
  Install from source.
- Add a caveat to the IBAC prerequisites: the additive
  plugin-pipeline merge and IBAC plugin require AuthBridge
  v0.6.0-alpha.7, which is not yet in a stable Kagenti release.
  Install Kagenti from main and verify the deployed sidecar tag
  via kagenti-platform-config until a release containing alpha.7
  is cut.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Kelly Abuelsaad <kaymar@gmail.com>
@yoavkatz
fix: use correct API version for MCPServerRegistration
46899c3 
Changed API version from mcp.kagenti.com/v1alpha1 to mcp.kuadrant.io/v1alpha1
to match the actual CRD installed in the cluster. This fixes the error:
'no matches for kind MCPServerRegistration in version mcp.kagenti.com/v1alpha1'

The MCPServerRegistration now successfully registers MCP servers with the
gateway and discovers tools correctly.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
fix(deploy-agent): fail deployment when agent card endpoint is not ac…
9164978 
…cessible

- Enable HTTP route creation for all deployment types (source and image)
- Change agent card endpoint test to use HTTP routes instead of port-forward
- Make deployment fail with exit 1 when agent card returns 404 or is not accessible after 60s
- Fix MCP_URL to use full Kubernetes service DNS for cluster communication
- Properly respect AuthBridge configuration for image deployments

This ensures deployments fail fast when HTTP routes are not properly configured,
preventing silent failures where agents appear deployed but are not accessible.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
refactor: migrate from port-forward to HTTP routes for API access
c4123cc 
- Replace kubectl port-forward with HTTP routes using localtest.me domains
- Update Kagenti API endpoint to http://kagenti-api.localtest.me:8080
- Update Keycloak endpoint to http://keycloak.localtest.me:8080
- Simplify connectivity checks by removing port-forward setup logic
- Add AGENTS.md with git commit guidelines
- Clear CLAUDE.md content

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
feat: pull container images from Exgentic registry by default
233321d 
- Add support for pulling images from ghcr.io/exgentic registry
- Images are pulled, tagged as localhost for kind compatibility
- Add --local-image flag to force use of locally built images
- Update deploy-agent.sh, deploy-benchmark.sh, and deploy-and-evaluate.sh
- Registry can be overridden with EXGENTIC_REGISTRY environment variable
- Fallback to local images if registry pull fails

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
feat(deploy-and-evaluate): add --dry flag for dry-run mode
bc19b90 
- Add --dry flag to print commands without executing them
- Remove quotes from parameter names in dry-run output for better readability
- Separate dry-run display logic from execution logic
- Update README.md with dry-run mode documentation and examples
- Useful for verifying commands, debugging, and documentation

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
docs: make agent guidelines apply to all agents, not just Bob
cfba565 
- Generalize title and add note that guidelines apply to Bob, Claude, etc.
- Replace Bob-specific Signed-off-by example with a generic placeholder

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
fix(deploy): fail fast on unreachable services and fix dry-run quoting
6eb6725 
- Retry Keycloak/Kagenti accessibility checks for 10s, then exit 1
  instead of warning and continuing with an unreachable service
- Quote arguments via printf %q in deploy-and-evaluate dry-run output
  so the displayed command is copy-paste safe and handles empty flags

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
chore: remove CLAUDE.md
713e2ca 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
docs: add CLAUDE.md pointer to AGENTS.md
8a2c6df 
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@rubambiza
fix: Correct AuthBridge path casing in docs links
6b88a7c 
The authbridge directory in kagenti-extensions is lowercase, not
PascalCase. Fixes broken link in README.md and inline path reference
in AUTHBRIDGE_PIPELINE_SPEC.md.

Fixes rossoctl#20

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
@kellyaa
Merge pull request rossoctl#22 from rubambiza/fix/broken-authbridge-link
3b08a5f 
fix: Correct AuthBridge path casing in docs links
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
Merge pull request rossoctl#21 from yoavkatz/feat/use_routes_instead_…
71af820 
…of_port_forward

refactor: migrate from port-forward to HTTP routes for API access
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
Merge remote-tracking branch 'origin/main' into docs/authbridge-pipel…
77619ca 
…ine-readme

Signed-off-by: Kelly Abuelsaad <kaymar@gmail.com>

# Conflicts:
#	exgentic_a2a_runner/README.md
@kellyaa
Merge pull request rossoctl#19 from kagenti/docs/authbridge-pipeline-…
482d4fb 
…readme

docs(exgentic): document AuthBridge plugin pipeline and IBAC

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@rubambiza
docs: Fix broken internal link in workload-harness
afc258b 
Update AuthBridge/docs/ -> authbridge/docs/ (case-corrected path) in
exgentic_a2a_runner/README.md.

Automated fix by OpenClaw Link Health Fixer (2026-06-30).

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
@yoavkatz
feat(exgentic-runner): in-cluster execution support
f664f89 
Add full in-cluster execution for the exgentic A2A runner as a Kubernetes
Job, e2e test script, shared URL helpers, and a set of correctness fixes.

Key changes:
- feat: add in-cluster Kubernetes Job execution (k8s/job.yaml, Dockerfile)
- feat: add e2e-test.sh — runs all benchmarks and emits a results table
- feat: add libsh/urls.sh — URL helpers that switch between localtest.me
  and cluster-internal DNS based on KUBERNETES_SERVICE_HOST
- feat: add sync-image-to-cluster.sh and update-secrets.sh helpers
- refactor: deploy-agent/benchmark use GHCR registry images by default;
  local image sync only triggered by --local-image
- fix: replace kubectl port-forward health checks with HTTP wait_for_url
- fix: skip kubectl context checks inside cluster (no kubeconfig)
- fix: MLflow tracing enabled by default; --mlflow renamed --disable-mlflow
- fix: OTEL exporter uses HTTP/protobuf in-cluster (port 8335)
- fix: preserve benchmark exit status in parallel e2e runs
- fix: respect CLI/env overrides over .env file values
- fix: unique port-forward ports per benchmark in parallel mode
- docs: add In-Cluster Execution and E2E Test Script sections to README
- docs: add AGENTS.md branching and DCO sign-off policy

Signed-off-by: Yoav Katz <katz@il.ibm.com>
kellyaa and others added 19 commits July 8, 2026 10:20
@kellyaa
Merge pull request rossoctl#26 from clawgenti/fix/broken-links-worklo…
2c04a19 
…ad-harness-2026-06-30

docs: Fix broken internal link in workload-harness
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
feat(exgentic-runner): OpenShift cluster support via INGRESS_DOMAIN
36abc58 
Add INGRESS_DOMAIN env var to url helpers so scripts can target an
OpenShift cluster from a laptop without port-forwarding. Harden
Keycloak admin credential resolution (auto-fetch from
keycloak-initial-admin secret, KEYCLOAK_ADMIN_USERNAME/PASSWORD
overrides) and make deploy/evaluate fail fast on auth errors instead
of silently continuing with wrong credentials.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
refactor(exgentic-runner): replace env-var heuristics with explicit -…
b04c2d1 
…-cluster-mode flags

Replace implicit KUBERNETES_SERVICE_HOST / INGRESS_DOMAIN detection with an
explicit --kind / --openshift DOMAIN / --in-cluster flag accepted by all four
entry-point scripts.  CLUSTER_MODE is exported before sourcing urls.sh so URL
helpers dispatch via case rather than a chain of env-var ifs.

Extract kubectl context validation into libsh/check-kubectl-context.sh so each
deploy/evaluate script calls a single check_kubectl_context function instead of
duplicating the inline block.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
fix(exgentic-runner): correct OpenShift route targetPort and tool URL…
00d8cd9 
… pattern

Patch the route's targetPort from numeric 8080 to the named port "http"
after deploy — OpenShift's router resolves by name when the service port
is named, so the numeric value fails with 503.

Also drop the spurious "-mcp" infix from the OpenShift tool URL template
in urls.sh so route hostnames match what Kagenti actually creates.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
feat(exgentic-runner): add --max-tasks and --max-parallel-sessions flags
1804ed7 
Add --max-tasks N and --max-parallel-sessions N to all three evaluation
scripts. Both default to 1 (respecting env vars set before invocation).

- evaluate-benchmark.sh: parses both flags, exports MAX_TASKS and
  MAX_PARALLEL_SESSIONS for the Python runner
- deploy-and-evaluate.sh: parses both flags, forwards them to
  evaluate-benchmark.sh in both live and dry-run paths
- e2e-test.sh: replaces --tasks / --parallel-sessions with --max-tasks /
  --max-parallel-sessions; run_local passes them as CLI flags instead of
  env-var prefixing; run_k8s injects them as Job env vars as before

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
docs: fix Mermaid parse errors in workflow-diagrams.md sections 2.1 a…
53a24e0 
…nd 3.1

Replace angle-bracket placeholders (<tool>, <agent>) with curly-brace
form ({tool}, {agent}), replace => with → in note text, and replace the
semicolon statement separator with a comma to avoid Mermaid tokeniser
conflicts.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
refactor(deploy-agent): remove source/generic_agent build path
981bc6b 
All agents are now deployed from prebuilt images. Drop the
DEPLOYMENT_TYPE=source branch (generic_agent → BuildRun), the
MCP_URLS env-var injection, and the per-type AGENT_JSON construction.
Update help text, README, and workflow diagram to match.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
fix(exgentic-runner): simplify unreachable cluster hint to oc login
cf8ee25 
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
Merge pull request rossoctl#27 from yoavkatz/feat/exgentic-runner-in-…
ebc41dc 
…cluster

feat(exgentic-runner): OpenShift and in-cluster execution support

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@kellyaa
chore: remove appworld_a2a_runner and promote exgentic README to top …
3c8918e 
…level

- Delete appworld_a2a_runner/ package
- Move exgentic_a2a_runner/README.md to top-level README.md, updating
  intro to preserve workload-harness framing and fixing sibling links
  now one directory deep (ibac-benchmarking.md, AUTHBRIDGE_PIPELINE_SPEC.md,
  analyze-run.sh, download_mlflow_traces.py, analyze_traces.py)
- Drop "Key Differences from AppWorld Runner" section from README
- Scrub appworld references from exgentic IMPLEMENTATION_CHECKLIST.md

The three top-level planning docs (IMPLEMENTATION_SUMMARY.md,
EXGENTIC_A2A_RUNNER_PLAN.md, EXGENTIC_ARCHITECTURE.md) are historical
planning artifacts and are left as-is per user request.

Signed-off-by: Kelly Abuelsaad <kaymar@gmail.com>
@yoavkatz
Merge pull request rossoctl#29 from kagenti/chore/remove-appworld-a2a…
a4af49b 
…-runner

chore: remove appworld_a2a_runner and promote exgentic README to top level
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
fix: fail fast on deployment errors and add pyyaml dependency
fef2443 
Replace dead post-command `if [ $? -ne 0 ]` checks (killed by set -e before
they run) with `|| fail "..."` pattern, and convert warnings on unexpected
API responses to hard failures to avoid proceeding with a broken backend.
Add pyyaml as an explicit dependency for authbridge/pipeline-merge.py.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
feat(deploy-benchmark): add --action-timeout flag and extend MCP wait…
f0ce136 
… time

Adds a --action-timeout CLI argument that sets EXGENTIC_SET_BENCHMARK_ACTION_TIMEOUT
env var on the deployed pod, and extends MCP readiness wait from 180s to 300s.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
Merge pull request rossoctl#32 from yoavkatz/fix/fail-fast-deployment…
6901ffb 
…-errors

fix: fail fast on deployment errors and add action timeout flag
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
feat: add --experiment flag and delete-all-deployments script
6a52068 
Add --experiment NAME flag to deploy-agent.sh, deploy-benchmark.sh, deploy-and-evaluate.sh,
and evaluate-benchmark.sh so parallel experiments get distinct pod/service names (e.g.
exgentic-a2a-tool-calling-gsm8k-exp1 vs …-exp2).

Reorder steps in deploy-agent.sh and deploy-benchmark.sh so resource limits and rollout
stabilization happen before the health-check wait, avoiding a race where the pod restarts
mid-check.

Add delete-all-deployments.sh to bulk-delete all agent and benchmark deployments in a
namespace via the Kagenti API.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
Merge pull request #3 from yoavkatz/feat/experiment-name-support
57396d8 
feat: add --experiment flag and delete-all-deployments script
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
fix: harden MCP session cleanup and extend deploy timeout
c9a36bd 
- runner: catch delete_session errors inside the MCP.DeleteSession child
  span so a residual cleanup failure (e.g. sidecar reaped after an agent
  timeout) never flips a completed session's span to ERROR
- mcp_client: match benign "already gone" markers case-insensitively and
  recognize "no longer alive" as a clean no-op on session delete
- deploy-benchmark: extend rollout stabilization timeout 120s -> 360s
- analyze_traces: aggregate experiment comparison into a single table
  with dynamic column widths

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
Merge pull request #4 from yoavkatz/fix/mcp-session-cleanup-timeout
496cb83 
fix: harden MCP session cleanup and extend deploy timeout
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz @claude
feat(analyze-run): support OpenShift MLflow and time-window trace fetch
2afc116 
Rework analyze-run.sh to reach MLflow via kubectl port-forward for both
--kind and --openshift, mirroring evaluate-benchmark.sh's OTEL collector
approach. MLflow location, TLS, workspace, auth mode, and experiment id
now default per cluster mode, with env/flag overrides.

- Add --openshift DOMAIN / --kind cluster modes with per-mode defaults
  (namespace, service, port, TLS, workspace, auth mode, experiment id)
- Add --auth-mode (secret | oc-token); oc-token uses `oc whoami -t`
- Replace --limit trace count with --window (e.g. 3h, 90m, 2d); the
  downloader pages newest-first and stops at the first out-of-window trace
- Skip trivially short traces (MIN_DURATION_S) before fetching their spans
- Send x-mlflow-workspace header and allow insecure TLS for port-forwarded
  reencrypt HTTPS endpoints (RHOAI)
- Verify kubectl context matches the cluster mode via libsh helper
- Harden secret-mode token flow against set -e command-substitution aborts

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yoavkatz @kellyaa @rubambiza