Skip to content

fix(install): surface Higress gateway startup diagnostics - #984

Open
RerankerGuo wants to merge 2 commits into
agentscope-ai:mainfrom
RerankerGuo:fix/issue-888-higress-gateway-diagnostics
Open

fix(install): surface Higress gateway startup diagnostics#984
RerankerGuo wants to merge 2 commits into
agentscope-ai:mainfrom
RerankerGuo:fix/issue-888-higress-gateway-diagnostics

Conversation

@RerankerGuo

@RerankerGuo RerankerGuo commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • print architecture, relevant Linux kernel limits, and the Higress Gateway log tail when the embedded installer times out waiting for /status
  • distinguish a low fs.inotify.max_user_instances limit from the ARM64 Envoy/V8 crash that can persist after sysctls are raised
  • explain that disabling Wasm is useful diagnostic evidence, but is not a supported workaround because AgentTeams relies on Higress Wasm for gateway authentication and AI routing
  • document the same troubleshooting guidance in the English and Chinese FAQ

Refs #888.

Verification

  • bash -n install/hiclaw-install.sh
  • git diff --check
  • extracted-function simulation verifies ARM64, all three kernel values, the
    fatal log tail, and the Wasm compatibility warning
  • confirmed the branch is based on current main (6f0c7da)
  • PowerShell parser not run locally because pwsh is not installed in this environment

CI note

Both refreshed runs passed every build and 9/10 integration matrix jobs. The
only failures were two different pre-existing test-06-multi-worker setup
races in copaw/hermes SHARD_A:

  • run 29233390641: Bob was fully provisioned, but the collaborative-task message denied a pending Tool Guard approval instead of starting a new Manager turn
  • run 29236623909: the test aborted before Bob setup because the Manager DM did not become quiet within 180 seconds

This PR only changes installer diagnostics and FAQ text. #985 now removes both
Manager-conversation dependencies from Bob's deterministic test setup.

Temporary dependency

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=hermes)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=copaw)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

📊 CI Metrics Report

Summary

Metric Current Baseline Change
LLM Calls 93 81 +12 ↑ +14.8%
Input Tokens 3131722 2803871 +327851 ↑ +11.7%
Output Tokens 21622 16791 +4831 ↑ +28.8%
Total Tokens 3153344 2820662 +332682 ↑ +11.8%

By Role

Role Metric Current Baseline Change
🧠 Manager LLM Calls 84 68 +16 ↑ +23.5%
Input Tokens 2918133 2502214 +415919 ↑ +16.6%
Output Tokens 18003 13725 +4278 ↑ +31.2%
Total Tokens 2936136 2515939 +420197 ↑ +16.7%
🔧 Workers LLM Calls 9 13 -4 ↓ -30.8%
Input Tokens 213589 301657 -88068 ↓ -29.2%
Output Tokens 3619 3066 +553 ↑ +18.0%
Total Tokens 217208 304723 -87515 ↓ -28.7%

Per-Test Breakdown

Test Mgr Calls Wkr Calls Δ Calls Mgr In Wkr In Mgr Out Wkr Out Δ Tokens Trend
02-create-worker 8 0 -4 ↓ -33.3% 239610 0 1785 0 -117227 ↓ -32.7% ✅ improved
03-assign-task 9 4 -2 ↓ -13.3% 256810 92187 1190 633 -122836 ↓ -25.9% ✅ improved
04-human-intervene 12 0 -1 ↓ -7.7% 305750 0 1319 0 -125929 ↓ -29.1% ✅ improved
05-heartbeat 4 0 -3 ↓ -42.9% 143483 0 1411 0 -130358 ↓ -47.4% ✅ improved
06-multi-worker 51 5 +22 ↑ +64.7% 1972480 121402 12298 2986 +829032 ↑ +64.8% ⚠️ regressed

Trends

4 test(s) improved (fewer LLM calls)
⚠️ 1 test(s) regressed (more LLM calls)


Generated by HiClaw CI on 2026-07-13 15:50:03 UTC


📦 Download debug logs & test artifacts

@RerankerGuo

Copy link
Copy Markdown
Contributor Author

@shiyiyue1102 Hi! Quick note on the SHARD_A copaw/copaw and copaw/hermes failures: I downloaded the test artifacts from run 28634278368. In the copaw/hermes case, the hiclaw-worker-bob session directory does not exist at all and Manager never invoked hiclaw create worker --name bob — Manager only created alice. In the copaw/copaw case, bob did get created and reached 14 LLM calls, but a later assertion still failed.

So the failure mode is the Manager not creating bob (a test-prompt interpretation flake), not anything in the install diagnostics this PR adds. The change itself is shell diagnostics + a sysctl note in the FAQ. Could you either re-run CI or merge if you agree the flake is unrelated? Happy to dig further if you can point me at what assertion actually failed.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction-2 / mgr=copaw / wk=hermes)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@RerankerGuo

Copy link
Copy Markdown
Contributor Author

@shiyiyue1102 Hi! Re-rebased onto current main (HEAD 0b562ff) and re-ran CI. SHARD_A is now passing on this PR — but a different integration test failed:

integration-tests (llm-interaction-2, SHARD_B_TESTS, copaw, hermes) — that's tests/test-14-git-collab.sh (the 4-phase non-linear multi-worker git collaboration workflow). Took 57 minutes before failing.

This is a different failure pattern from the earlier SHARD_A copaw/hermes flake. Test 14 has its own LLM-flake surface (3 workers, 4 phases, complex branch-name / file-path / report-name invariants), and there's no equivalent deterministic-bypass like #985 yet for it. The change in this PR is shell diagnostics for Higress Gateway startup + an English/Chinese FAQ note on sysctl checks — it doesn't touch Manager agent runtime, worker git ops, or test-14 setup. Could you re-run SHARD_B copaw/hermes, or merge this PR if you agree the test 14 failure is unrelated? Happy to dig into the test-14 artifacts (run 28926397804, job 85816940745) if it would help.

Also: PR #985 (SHARD_A bob-setup deterministic fix) just turned green and was pinged this morning as priority.

RerankerGuo added a commit to RerankerGuo/AgentTeams that referenced this pull request Jul 9, 2026
v1 over-prescribed by pre-creating the project room and instructing Manager
not to call create-project.sh. That broke Manager's natural workflow —
it lost its entry point for the rest of the coordination and ended up
writing ad-hoc Matrix HTTP scripts (run 28948704338). copaw/hermes made
Manager do 136 LLM calls + alice 54 calls + bob/charlie 0 (phases 2-4
never started).

v2 only pre-creates the 3 workers (the actual flake source on SHARD_A).
Manager still walks through create-project.sh + YOLO Step 1d (meta.json
active transition) + 4-phase coordination as the workflow intends — this
test measures that flow, not LLM timing on worker creation.

Refs: agentscope-ai#984
@RerankerGuo
RerankerGuo force-pushed the fix/issue-888-higress-gateway-diagnostics branch 3 times, most recently from 7756e4d to b0d4f81 Compare July 13, 2026 14:44
@RerankerGuo
RerankerGuo force-pushed the fix/issue-888-higress-gateway-diagnostics branch from b0d4f81 to ae9dfa8 Compare July 14, 2026 11:07
RerankerGuo added a commit to RerankerGuo/AgentTeams that referenced this pull request Jul 14, 2026
Pre-create the 3 workers (alice, bob, charlie) via hiclaw apply worker and
the shared project via create-project.sh + immediate meta.json activation
+ MinIO sync, before sending the 4-phase task to Manager. Manager now
focuses on phase coordination only — the worker + project setup steps that
previously stalled on Manager's tool-guard approval prompt (jq | mv | cat
on meta.json) are now in the test runner and never enter the LLM control
loop.

Mirrors the fix in agentscope-ai#985 for test-06-multi-worker.sh (bob creation flake).
Both fixes isolate LLM-driven coordination from infrastructure setup so
the test measures coordination quality, not LLM timing.

Fixes the SHARD_B copaw/hermes test-14 failure seen on agentscope-ai#984 (run 28926397804,
job 85816940745), where Manager hung for 30 minutes on a tool-guard approval
prompt while trying to update meta.json status.

Refs: agentscope-ai#984
RerankerGuo added a commit to RerankerGuo/AgentTeams that referenced this pull request Jul 14, 2026
v1 over-prescribed by pre-creating the project room and instructing Manager
not to call create-project.sh. That broke Manager's natural workflow —
it lost its entry point for the rest of the coordination and ended up
writing ad-hoc Matrix HTTP scripts (run 28948704338). copaw/hermes made
Manager do 136 LLM calls + alice 54 calls + bob/charlie 0 (phases 2-4
never started).

v2 only pre-creates the 3 workers (the actual flake source on SHARD_A).
Manager still walks through create-project.sh + YOLO Step 1d (meta.json
active transition) + 4-phase coordination as the workflow intends — this
test measures that flow, not LLM timing on worker creation.

Refs: agentscope-ai#984
@github-actions

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction-2 / mgr=openclaw / wk=openclaw)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=openclaw / wk=openclaw)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction-2 / mgr=copaw / wk=copaw)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr-2 / mgr=copaw / wk=copaw)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=copaw / wk=hermes)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=openclaw / wk=openclaw)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=copaw / wk=copaw)

Commit: ae9dfa8
Workflow run: #1649

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant