From f5a17b69e0751fa978644ede4379f282253289fe Mon Sep 17 00:00:00 2001 From: cafitac Date: Fri, 1 May 2026 21:13:09 +0900 Subject: [PATCH] docs: record Hermes round-trip QA --- .../hermes-executor-readiness/README.md | 29 +++++++++---------- ...pr-02-hermes-mcp-registration-roundtrip.md | 18 ++++++++++++ 2 files changed, 32 insertions(+), 15 deletions(-) diff --git a/.dev/refactor/hermes-executor-readiness/README.md b/.dev/refactor/hermes-executor-readiness/README.md index 6877bd1..efbfcdf 100644 --- a/.dev/refactor/hermes-executor-readiness/README.md +++ b/.dev/refactor/hermes-executor-readiness/README.md @@ -6,26 +6,25 @@ Repository: `/Users/reddit/Project/hermit-agent` -Current base at time of writing: +Current base at latest QA update: - branch: `main` -- HEAD observed: `6bf092f Merge pull request #66 from cafitac/release-sync/v0.3.65` -- working tree was clean before writing this planning set -- latest release observed: `v0.3.65` +- HEAD observed before this docs-only QA note: `62cb930 chore: release v0.3.67 [skip ci] [skip release] (#70)` +- working tree was clean before starting the follow-up QA branch +- latest release observed: `v0.3.67` Verification already run: - `.venv/bin/python -m pytest tests/test_orchestrator_contracts.py tests/test_orchestrator_prompt_mapping.py tests/test_hermes_orchestrator_adapter.py -q` - - expected/current: `10 passed` + - earlier expected/current: `10 passed` - `.venv/bin/python -m pytest tests/ -q` - - expected/current: `1063 passed` + - latest preserved result after PR-02 fixes: `1066 passed` -Important local Hermes observation: -- `command -v hermes` returned `/Users/reddit/.local/bin/hermes` -- `hermes mcp list` returned no configured MCP servers -- `hermit doctor` reported `WARN Hermes MCP: hermit-channel missing` -- `.venv/bin/python -m hermit_agent install --test-hermes-mcp` initially incorrectly reported `passed` because Hermes CLI printed `Server 'hermit-channel' not found in config.` while returning exit code 0; PR-01 is intended to make that output fail truthfully. -- direct `hermes mcp test hermit-channel` printed `✗ Server 'hermit-channel' not found in config.` while returning exit code 0 - -This means the current `run_hermes_mcp_connection_test()` is not trustworthy: it treats return code 0 as success even when Hermes reports that the server is missing. +Important local Hermes observation history: +- Before PR-01, `hermes mcp test hermit-channel` printed `✗ Server 'hermit-channel' not found in config.` while returning exit code 0, and `hermit install --test-hermes-mcp` incorrectly reported `passed`. +- PR-01 fixed the live-smoke false positive by inspecting failure-looking Hermes output, not just exit code. +- During PR-02 QA, `hermit install --fix-hermes-mcp` also had a false-positive path: Hermes could cancel the tool-enable prompt while returning exit code 0. PR-02 fixed this by sending explicit approval input and verifying `hermes mcp list` after registration. +- Current local Hermes state after v0.3.67 reinstall: `hermes mcp list` shows `hermit-channel hermit mcp-server all ✓ enabled`, `hermit doctor` reports `✅ Hermes MCP`, and `hermit install --test-hermes-mcp` passes. +- Follow-up direct MCP client smoke through `hermit mcp-server` successfully called `run_task`, registered the returned task id, polled `check_task`, and received `HERMIT_E2E_OK` from `glm-5.1` through the Gateway/AgentLoop/provider path. +- The direct Python MCP SDK client printed validation warnings for the custom `notifications/claude/channel` notification method, but polling still returned `status: done`; treat this as generic-client compatibility noise unless Hermes itself shows the same issue. ## Product goal @@ -56,7 +55,7 @@ Do not claim "Claude Code / Codex / Hermes executor readiness is complete" until - `hermit install --fix-hermes-mcp` is idempotent and registers `hermit-channel -> hermit mcp-server` through the Hermes CLI. - `hermit doctor` agrees with `hermes mcp list` and does not report a false PASS. - A live Hermes MCP probe succeeds only after the server is registered. -- At least one tiny Hermit task can be invoked through the same MCP server surface used by Hermes, or the limitation is explicitly documented as not yet ready. +- At least one tiny Hermit task can be invoked through the same MCP server surface used by Hermes. Latest direct stdio MCP smoke did this with `run_task` → `register_task` → `check_task`, returning `HERMIT_E2E_OK` through Gateway/AgentLoop/provider. - Claude Code and Codex regression tests remain green after Hermes changes. - Public docs distinguish stable behavior from experimental / adapter-in-progress behavior. diff --git a/.dev/refactor/hermes-executor-readiness/pr-02-hermes-mcp-registration-roundtrip.md b/.dev/refactor/hermes-executor-readiness/pr-02-hermes-mcp-registration-roundtrip.md index 91667e9..566bfca 100644 --- a/.dev/refactor/hermes-executor-readiness/pr-02-hermes-mcp-registration-roundtrip.md +++ b/.dev/refactor/hermes-executor-readiness/pr-02-hermes-mcp-registration-roundtrip.md @@ -153,6 +153,24 @@ Do not fake this in tests. Expected: - full suite passes +## Verified follow-up status + +Implemented/verified by PR #67, PR #69, and the v0.3.67 release-sync: +- registration is explicit and idempotent +- `--test-hermes-mcp` no longer reports pass for a missing server output with exit code 0 +- `--fix-hermes-mcp` handles Hermes' tool-enable prompt and treats `Cancelled`/failure-looking output as failure +- post-registration verification checks `hermes mcp list` +- local `hermes mcp list`, `hermit doctor`, and `hermit install --test-hermes-mcp` agree that `hermit-channel -> hermit mcp-server` is enabled + +Additional follow-up smoke after MCP reload: +- Gateway was started with `./bin/gateway.sh --daemon` and `/health` reported `glm-5.1` as the active default model. +- A direct stdio MCP client connected to `hermit mcp-server`, listed `register_task`, `run_task`, `reply_task`, `check_task`, and `cancel_task`, then called: + - `run_task` with a no-edit QA prompt in a temporary git repo + - `register_task` with the returned task id + - `check_task` with `full=true` +- Result: `status: done`, `result: "HERMIT_E2E_OK"`, proving the MCP server surface can drive the Gateway/AgentLoop/provider task path. +- Limitation: this direct Python MCP SDK smoke printed validation warnings for the custom `notifications/claude/channel` notification method. Polling still succeeded. Hermes Agent itself needs continued observation for whether it accepts that extension cleanly; generic SDK warnings alone are not a round-trip failure. + ## Acceptance criteria - Hermes registration command is idempotent.