Skip to content

[ci] Improve Blueprint QA readiness checks - #123

Merged
antoniomtz merged 3 commits into
mainfrom
codex/ci-agent-readiness
Aug 3, 2026
Merged

[ci] Improve Blueprint QA readiness checks#123
antoniomtz merged 3 commits into
mainfrom
codex/ci-agent-readiness

Conversation

@antoniomtz

Copy link
Copy Markdown
Collaborator

Summary

  • wait for Milvus catalog seeding before running UI automation
  • require all NAT agents to pass internal health checks
  • capture Docker Compose status and relevant service logs when CI fails

Why

The Blueprint QA job could start its UI tests after the UI health endpoint passed while the search agent and its Milvus-backed catalog were still unavailable. That race can produce intermittent Apps SDK failures, and the previous cleanup path removed the service evidence needed to confirm the cause.

Impact

The UI automation suite now starts only after its runtime dependencies are ready. Failed runs retain service diagnostics in the existing report artifact. Tests are still run once; this change does not hide failures with retries.

Test plan

  • parsed the workflow YAML successfully
  • syntax-checked all five embedded Bash blocks
  • passed Prettier for the workflow file
  • passed git diff --check
  • validated the merged Docker Compose configuration with docker compose ... config --quiet

The live GitHub Actions run will provide the final runtime validation after this PR is opened.

@antoniomtz
antoniomtz marked this pull request as ready for review July 24, 2026 18:11
@thanhpt1110

Copy link
Copy Markdown
Contributor

Thanks for implementing this, @antoniomtz! I just have a couple of requested changes:

  • The UI readiness check isn't actually checking the UI

    /api/health is served by the merchant service via nginx (location /api/ -> proxy_pass http://merchant/), not the Next.js app, and merchant's handler is a static response. Since the ui service has no HEALTHCHECK and nginx only waits on service_started for it, browser tests can still start before the frontend is ready.

    Could we wait for both the API and the UI root page instead?

    echo "Waiting for the merchant API and the UI to answer (up to 10 min)..."
    if ! timeout 600 bash -c 'until curl -fsS "${UI_URL}/api/health" >/dev/null 2>&1 && curl -fsS -o /dev/null "${UI_URL}/"; do sleep 10; done'; then
  • Include more diagnostics

    Could we also collect logs for psp, milvus-standalone, milvus-etcd, milvus-minio, and phoenix? I'd also suggest increasing retention-days since the report artifact from the last failed run expired before it could be investigated.

  • Before merging

    Could you rerun the workflow on this branch a few times once these changes are in? I'd like to confirm the fix is stable.

Question

Does nat serve's /health exercise the retriever/LLM path, or does it only confirm the server is listening? If it's the latter, a failure mode like empty search results or a 429 from the NIM endpoint would still pass this gate. A single real query might be a good follow-up.

@antoniomtz antoniomtz self-assigned this Aug 3, 2026
@antoniomtz
antoniomtz merged commit 074edfe into main Aug 3, 2026
10 of 11 checks passed
@antoniomtz
antoniomtz deleted the codex/ci-agent-readiness branch August 3, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants