feat(manager): migrate Manager runtime from copaw to qwenpaw 2.0 - #1095
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
0676a69 to
9e293b2
Compare
After PR agentscope-ai#1095 the Manager runs QwenPaw 2.0, not copaw. The script name should reflect reality. start-manager-agent.sh now accepts both 'copaw' (legacy runtime identifier used by controller/installer/CI) and 'qwenpaw' (forward- looking name). Both map to the same start-qwenpaw-manager.sh. Controller, installer, CI, Makefile, and Docker image name still use 'copaw' as the runtime identifier — changing those is a separate PR (the controller already defines RuntimeQwenPaw='qwenpaw' for Workers; adding a qwenpaw Manager runtime requires new config fields, image selection logic, and CI matrix entries).
Change-Id: I9ad741de07f4b7bfbbe3721030df8749546a0def
Change-Id: I7d7930c2391217d2c1887eb4d7e20449b4b16ce8
Change-Id: I3073b27f43f0ecde4e6fa2164fd49d2659d50dcf
Change-Id: I31687da14f2d40bfb50dade36a8d7467e9aa8ddd
Change-Id: Ieeed7e69bf3b23e3f03aaa5168e4fe807040c1fe
Change-Id: I9b2094e39e7dfc12437aedf347199ace79de1f5b
Change-Id: I04a62daf94efad6d0445dce93a56cf897530c25e
Change-Id: I7af2aece15ab2c733754beee59e31d377ca1d810
Change-Id: I55ac99d6d22041bc7e0d229449fd0333f6ca8a29
Change-Id: I961ed24fb022cbe8ac01fab81dfe018a00c14a5d
Change-Id: Ic437642f20426d03f585442fb06867ede35e6151
Change-Id: I352dc6fa16715f0ce03ce00931e61a0abadc4efc
Change-Id: I03e49cd512911c176c9a8cb3d407877fdf8b2363
Change-Id: I50577926a078ed50232e3eb6af6583d1162be7df
Change-Id: I88060c8e6c49862205ec602de2ae3e72a4f2c879
shiyiyue1102 requested end-to-end validation of the legacy CoPaw →
QwenPaw migration (secrets, sessions, plugins, workspace) so existing
installs of HiClaw/AgentTeams with CoPaw runtime upgrade automatically
without data loss.
1. Extract §10b migration logic from start-qwenpaw-manager.sh into a
standalone manager/scripts/init/migrate-copaw-state.sh so CI can run
the REAL production migration path directly. Script is:
- copy-then-verify: writes .copaw-migrated only after every critical
artifact (incl. .copaw.secret master_key/providers.json/envs.json)
is verified in the target
- retry-safe: partial copy returns exit 1, marker NOT written,
retried on next startup
- idempotent: marker present → skip
- environment-overridable (HOME/QWENPAW_WORKING_DIR/QWENPAW_SECRET_DIR/
WORKSPACE_DIR) so tests run in an isolated throwaway HOME
- fixes standalone-run bug: mkdir target root before copying
2. New tests/test-28-migration-e2e.sh runs against the real script in a
throwaway HOME (/tmp/mig-e2e) inside the Manager container (never
touches live state). Covers: full secret migration with content
equality (14 artifacts), marker only after full verification,
idempotency (no clobber), partial-copy failure → no marker → retry
succeeds with data intact. Explicitly overrides QWENPAW_WORKING_DIR/
QWENPAW_SECRET_DIR/WORKSPACE_DIR because the Manager image exports
QWENPAW_WORKING_DIR.
3. Register test-28 in SHARD_C_TESTS + NON_GITHUB_TESTS.
Locally verified (mock env): success path, idempotency, failure retry
all pass. Live container E2E runs in CI.
…low, sync) Merge upstream/main (2ea0274, ba161a8, 00a5d20) into manager-qwenpaw-2.0. Three conflicts resolved: 1. copaw/src/copaw_worker/hooks/tools/taskflow.py - Ours (PR agentscope-ai#1095): delegate_task auto-notifies the Worker via _notify_task_assignment with stable txn_id, prepared->assigned state machine, room ID normalization. - Theirs (upstream agentscope-ai#1120): manual handoff - delegate returns notificationRequired + nextAction for the Leader to call message. - Resolution: keep ours (auto-notify). shiyiyue1102 explicitly requires automatic send; manual handoff would produce duplicate assignments. Updated test_taskflow_tool.py assertion accordingly. 2. copaw/src/copaw_worker/sync.py - Ours: .copaw -> .qwenpaw path migration + QWENPAW_WORKING_DIR. - Theirs: runtime/runtime.yaml sync (new). - Resolution: merge both - keep .qwenpaw paths AND add runtime/runtime.yaml to push_files + _EXCLUDE_PATHS. 3. changelog/current.md - Ours: QwenPaw 2.0 runtime unification entry. - Theirs: custom model capability overrides entry + bug fixes. - Resolution: keep both What's New entries; bug fixes auto-merged. Auto-merged cleanly: bridge.py (our agents.profiles nesting + upstream multimodal env override), matrix_channel.py, matrix/channel.py, message_filter.py, cli.py, docs, installer. Tests: test_taskflow_tool 36 passed, test_notify_task_assignment passed, qwenpaw plugin + trace passed. Full pytest failures are pre-existing (agt CLI env deps) or test-order pollution (each passes standalone); CI runs bash integration tests (SHARD_C now includes test-28).
|
Thanks for the update. I rechecked the exact current head (
The current PR head also conflicts with the latest 感谢更新。我基于当前精确 head（
此外，当前 head 与最新 |
…hangelog
Deep audit of the shiyiyue1102 four-blocker fix + migration E2E found
and fixed 3 issues:
1. Matrix txn_id spec compliance: copaw_worker and MCP delegate_task used
'delegate:{task_id}' but Matrix txnId only allows [0-9a-zA-Z.=_-].
Unified to 'delegate-{task_id}' in both runtimes. nio does not
validate, but servers may reject non-spec characters.
2. team-leader-agent/skills/task-management/SKILL.md still taught the
OLD manual handoff ('call message to @mention the assigned Worker')
in its After-delegation step list, contradicting the auto-notification
contract stated in the same file (duplicate assignment risk). Replaced
with 'do NOT call message' guidance.
3. changelog/current.md agentscope-ai#1120 entry still described the manual
'Return the required Team Room message action' behavior; updated to
describe the final auto-notification behavior (atomic state, stable
txn_id) with agentscope-ai#1095 attribution.
Audit verification:
- txn_id fix: test_taskflow_tool + test_notify_task_assignment 38 passed
- qwenpaw plugin + trace 50 passed
- test_channel_mention 7 failures = pre-existing upstream issue (identical
on upstream/main 2ea0274, zero new regressions from this merge)
- test_sync 3 failures = pre-existing agt CLI env dependency (identical
before and after merge)
…prompt residuals, E2E errexit Three blocking issues from maintainer recheck: 1. Migration marker could be written when legacy data was NOT migrated. migrate-copaw-state.sh used cp -an (no-clobber) and only verified target existence, so a pre-existing target file (e.g. master_key) silently skipped the legacy value while .copaw-migrated was still written - permanently skipping legacy credentials on later boots. Fix: explicit conflict policy - legacy is authoritative on upgrade, so use cp -a (overwrite) for state/secret/workspace artifacts and content-verify every copied file with cmp before writing the marker. 2. Leader instructions still contradicted the auto-notification contract. communication/SKILL.md still said the first Team Room assignment is 'sent with message target=room:<Team Room ID>' and that the cross-room message call is mandatory, with a manual assignment JSON example. Fix: replaced with delegate_task auto-notification wording; cross-room message example is now requester-status only. Same Room example no longer shows a manual New task assignment. 3. test-28 failure/retry E2E never reached its assertions. The inner shell enabled set -e then ran the intentionally-failing migration; the shell exited there before RC1=$?, the marker assertion, or the retry. Fix: capture the expected failure with if/else so no-marker and successful-retry checks actually execute. Also added a pre-existing-target conflict E2E section (legacy must overwrite). Verified locally: conflict overwrite (legacy-key wins), idempotency (no clobber on re-run), failure retry (no marker on partial copy, retry succeeds with data intact).
|
@shiyiyue1102 Thanks for the detailed recheck. The PR head is now 感谢详细复核。PR head 已更新到 1. Migration marker written without legacy data migrated — fixed with an explicit conflict policy + content verification The root cause was exactly as you described: Fix (
1. 迁移 marker 在旧数据未真正迁移时仍被写入 — 已通过明确的冲突策略 + 内容校验修复 根因与您描述完全一致： 修复（
2. Leader instructions still contradicting auto-notification — all remaining manual assignment-send instructions removed You were right that the previous pass missed residuals. Fixed:
I also scanned the full team-leader-agent prompt set for any remaining 2. Leader 指令仍与自动通知契约冲突 — 已清除全部残留的手工派单指令/示例 您说得对，上一轮漏掉了残留。已修复：
同时全量扫描了 team-leader-agent 全部 prompts，Leader 路径已无任何 3. test-28 failure/retry E2E never reached its assertions — fixed Exactly as you found: the inner shell enabled Fix ( 3. test-28 失败/重试 E2E 实际没有执行到断言 — 已修复 与您发现的一致：内层 shell 开启了 修复（ 4. Merge conflicts + missing checks — resolved The head you reviewed ( 4. Merge 冲突 + 缺少 checks — 已解决 您复核的 head（ |
|
Thanks for the update. I rechecked the exact current head (
I reproduced this following the production startup order: So the standalone migration E2E passes, but an upgraded Manager may still start with a stale Matrix token and fail to reply or collaborate with the Team. Please either run legacy user-data migration before bridge/profile/plugin initialization and then let the current startup flow re-overlay Controller-owned values, or migrate/merge only explicitly user-owned artifacts. Please also add a production-startup-order upgrade E2E that verifies all of the following together:
There is also one current CI blocker: 感谢更新。我重新复核了当前精确 head（
按生产启动顺序复现结果如下： 因此，虽然独立迁移 E2E 已通过，但升级后的 Manager 仍可能使用旧 Matrix token 启动，导致不能回复或无法完成 Team 协作。建议把 legacy 用户数据迁移放到 bridge/profile/plugin 初始化之前，再由当前启动流程覆盖 Controller 管理字段；或者只迁移/合并明确属于用户的数据，避免整体覆盖当前配置。 还请补充一个覆盖真实生产启动顺序的升级 E2E，同时验证：
此外当前还有一个 CI 阻塞： |
shiyiyue1102 third review on head 22b8e1e: - Migration order (upgrade contract): migrate-copaw-state.sh ran AFTER the bridge/startup overlay steps, so legacy .copaw files (old Matrix token, old QA state) clobbered the Controller values just written from openclaw.json — upgraded Manager could boot with a stale token and fail to reply/collaborate. Move the migration to §1b (before bridge); the bridge (§2), prompt sync (§3-§4), DM patch (§6), QA disable (§7) and plugin install (§10) then re-overlay Controller- owned values while legacy user data (master_key, envs, sessions, memory, digest, models, custom_channels) is preserved. - helm-lint rename-contract: drop PR-introduced 'HiClaw' from migrate-copaw-state.sh and test-28 comments; also clean inherited 'hiclaw-install.sh' wording in README/README.zh-CN (upstream v1.2.0 news) so the tree-wide brand check passes. - test-28: add Production Startup Order E2E simulating mkdir -> migrate(OLD token) -> bridge(NEW token) -> QA disable -> plugin re-install, asserting NEW token preserved, QA disabled with default profile, legacy user data migrated, current plugins win, marker written.
This reverts commit 51f5541.
…als"" This reverts commit 2ee2b55.
…LLM CI - install/agentteams-install.sh: only migrate copaw→qwenpaw default Worker runtime on upgrades (AGENTTEAMS_UPGRADE=1). Fresh installs that explicitly request AGENTTEAMS_DEFAULT_WORKER_RUNTIME=copaw keep it, so the copaw-copaw CI shard exercises the real copaw worker. Previously the unconditional redirect turned every no-runtime Worker into a qwenpaw Worker, breaking test-17/18/19/20/22/24 (MinIO-root AGENTS.md/SOUL.md/skills are only produced by DeployWorkerConfig, which the qwenpaw runtime path skips). - team-leader communication SKILL.md: restore 'only for explicit cross-room' phrase asserted by test-21 (removed during prompt cleanup). - tests/lib/test-helpers.sh: require_llm_key() skips when AGENTTEAMS_CI_NO_LLM=1 — the ci-no-llm-placeholder key is non-empty but cannot reach the model, so test-21's LLM E2E previously ran against it and failed with 'Unauthorized access to model: qwen3.6-plus'.
…json The OpenClaw runtime strictly validates agents.defaults and does not recognize supports_multimodal / supports_image (added by upstream agentscope-ai#1103). When the Manager or a Worker runs openclaw, the generated openclaw.json caused 'Config invalid ... Unrecognized keys' and the openclaw Manager entered a restart loop (openclaw-openclaw CI shard failed right after test-100). Emit the keys only for non-openclaw runtimes. - WorkerConfigRequest gains Runtime - GenerateOpenClawConfig skips multimodal keys when Runtime == openclaw - worker and manager deploy call sites pass spec.runtime
The installer unconditionally redirects AGENTTEAMS_DEFAULT_WORKER_RUNTIME=copaw to qwenpaw (copaw is the legacy name; fresh installs must default to QwenPaw 2.0). This made the copaw-copaw CI shard's implicit-runtime Workers actually run qwenpaw, while the tests verified copaw artifacts (MinIO-root AGENTS.md/SOUL.md/skills are only produced by DeployWorkerConfig, which the qwenpaw runtime path skips) — test-17/18/19/20/22/24 failed. Pin spec.runtime / manifest runtime to TEST_WORKER_RUNTIME in those tests so each shard exercises the runtime it declares, independent of the Controller default. test-15/21 already did this; the redirect stays unconditional so the migration (copaw -> qwenpaw as the default worker runtime) is preserved.
|
刚才推了一版（a1a8659b），先说清楚改了什么，避免误读： openclaw shard 的失败——根因是我们之前 #1103 把 supports_multimodal/supports_image 写进了 openclaw.json 的 agents.defaults，OpenClaw runtime 严格校验不认，Manager 直接重启循环。修复：GenerateOpenClawConfig 加 Runtime 参数，openclaw 不再写这俩键。这是生产 bug 修复。 |
…pin test-22 worker runtime - test-22: TEST_WORKER_RUNTIME was referenced but never defined, so 'agt create worker --runtime ""' fell back to the Controller default (qwenpaw) and the delete-cleanup assertions looked for copaw artifacts (MinIO-root SOUL.md) that qwenpaw workers never produce. Define it like every other test (AGENTTEAMS_DEFAULT_WORKER_RUNTIME fallback). - update.py: _apply_matrix_channel_access_flags could take the whole worker down with QwenPawApiError when access_control_dm/group did not round-trip on readback (channel plugin re-initializes state concurrently on some runtimes). The allow-list itself is applied separately via the ACL endpoint in _write_matrix_access_control(); log and continue instead of crashing so qwenpaw stays up and TeamHarness/TEAMS.md render normally.
…ger readiness pgrep
- _send_delegate_notification(): remove unused txn_id parameter — the
Matrix transaction id is derived from task_id inside the function
(f"delegate-{task_id}"), identical to what the caller passed. A
dead parameter is a latent contract break: callers who customize it
would silently see no effect.
- wait_for_manager_agent_ready(): accept both 'copaw app' (run_copaw_app.py
via runpy, argv keeps copaw_worker.run_copaw_app) and 'qwenpaw app'
(direct python3 -m qwenpaw app) process patterns — mirrors
test-01-manager-boot.sh. Defensive only; current startup path still
matches the copaw pattern.
|
Thanks for the update. I rechecked the exact current head (
Please make the QwenPaw TeamHarness flow validate room membership and use 感谢更新。我重新复核了当前精确 head（
请将 QwenPaw TeamHarness 派单流程调整为：校验房间成员关系， |
…rker .copaw migration on qwenpaw_worker Address shiyiyue1102 review: the QwenPaw TeamHarness delegate_task path did not preserve the notification/state atomicity contract. server.py (TeamHarness MCP): - delegate_task now validates that the assignee is a joined member of the target Matrix room before writing any assignment state (strict when Matrix env + mxid are available, skipped otherwise). - Delegation is atomic: prepare (status=prepared, files published) -> send notification with stable txn id -> commit status=assigned + event_id only after the send succeeded. A send failure returns a retryable error and leaves the task prepared (never assigned without the Worker being notified). - Re-delegating an already assigned task is idempotent (returns the recorded event_id, no duplicate notification). qwenpaw_worker (Worker runtime): - Migrate a legacy .copaw working dir to .qwenpaw at Worker startup so an explicit runtime switch to QwenPaw carries the state over. The migration belongs to the target (qwenpaw_worker) startup path, not the legacy copaw_worker path. Runs in __init__ before RuntimeUpdater creates .qwenpaw; idempotent when .qwenpaw already exists. Tests: - test-taskflow.rb: mock /members endpoint, fix m.file event assertion (filter by msgtype), add failure-injection (Matrix 500 -> retryable + prepared persisted) and idempotent re-delegation cases. - test_trace_integration.py: simulate Matrix env + patch the automatic notification so the delegation commits assigned. - test_worker_lifecycle.py: 3 migration cases (rename, skip, noop).
test-roomflow.rb's create_task_room call omitted workspaceDir, so the server fell back to _default_workspace_dir() (the container's shared default workspace, derived from QWENPAW_WORKING_DIR). The first run wrote the room binding there and succeeded; every subsequent run found the stale binding, took the existing-room branch and POSTed an invite the mock does not serve -> HTTP 404. Pin workspaceDir to the test's tmpdir (matching every other roomflow call) so the test is idempotent and has no side effects on the shared default workspace.
|
Clarification on the trusted real-LLM/TeamHarness E2E item in my previous comment: this is not a pre-merge blocker for this fork PR. GitHub correctly withholds repository secrets from untrusted fork workflows. The PR should not attempt to bypass that boundary, expose the real LLM key, or replace the validation with a mock LLM. Please focus the pre-merge work on the source/test issues that can be fixed inside the PR. After merge, the 补充澄清我上一条评论中的可信环境真实 LLM/TeamHarness E2E：这一项不作为当前 fork PR 的合并前阻塞项。 GitHub 对不可信 fork workflow 隐藏仓库 Secret 是正确的安全机制。本 PR 不需要绕过该边界、暴露真实 LLM Key，也不需要用 mock LLM 替代这项验证。合并前请集中处理能够在 PR 内修复的代码和测试问题。 合并后， |
|
Thanks for the clarification — agreed, the trusted real-LLM/TeamHarness E2E is a post-merge validation on main's push workflow (which can use secrets.AGENTTEAMS_LLM_API_KEY); this fork PR will not bypass the fork-secret boundary or substitute a mock LLM. The three source/test issues are fixed on head 0e8189a:
收到澄清——真实 LLM/TeamHarness E2E 合入后在 main 的可信环境跑（可用 AGENTTEAMS_LLM_API_KEY），fork PR 不碰 secret 边界、不用 mock LLM 替代。 三个问题已修在 0e8189a：
|
|
Thanks for the update. I rechecked the exact current head (
Local checks on this head:
The trusted real-LLM/TeamHarness E2E remains a post-merge 感谢更新。我重新复核了当前精确 head（
当前 head 的本地检查结果：
可信环境的真实 LLM/TeamHarness E2E 继续作为合并后 |
…bership; keep copaw worker on .copaw Address shiyiyue1102's third-round review blockers: 1. delegate_task now gates the Matrix notification on the initial file publish: a failed _sync_task keeps the task prepared and never sends the notification (no Worker receives a reference to unpublished files). The assigned/eventId commit gates success too: if the post-commit sync fails, the tool returns a retryable failure and the idempotent retry (assigned + eventId) finishes the sync instead of reporting success with stale shared storage. 2. _matrix_room_member_user_ids accepts an optional memberships set (default join+invite preserved for roomflow helpers); delegation validation requires strictly 'join', so an invited-but-not-joined Worker fails with a retryable error and no notification. Regression covered by test-taskflow.rb (mock /members now includes an invite-only member). 3. copaw_worker keeps .copaw: the legacy CoPaw Worker no longer renames its working dir to .qwenpaw on startup. The .copaw -> .qwenpaw migration is owned exclusively by qwenpaw_worker, so an explicitly configured runtime: copaw Worker restart never migrates before a switch. Regression test proves a CoPaw start keeps .copaw and does not create .qwenpaw (qwenpaw_worker migration tests already cover the explicit-switch path). changelog updated.
shiyiyue1102
left a comment
shiyiyue1102
left a comment
There was a problem hiding this comment.
LGTM. Rechecked exact head 855e5c4: the public installer remains gated on CoPaw, integration CI explicitly exercises the PR-built QwenPaw Manager, all required checks pass, and the CoPaw/QwenPaw/Hermes lifecycle and migration paths are covered. The trusted real-LLM Team E2E remains a post-merge main-branch validation as previously agreed.
3 participants
Summary
Migrate the Manager container from copaw 1.0.2 to QwenPaw 2.0.1, following the same plugin-based architecture that PR #1077 established for Workers.
What changed
Manager runtime — separate venv for QwenPaw 2.0.1 to avoid agentscope version conflicts with copaw 1.0.2 (used only by bridge.py for config conversion). Manager tools (projectflow, taskflow, message, filesync) registered via a QwenPaw plugin instead of monkey-patching
CoPawAgent._create_toolkit.Matrix channel — physical overlay replaced with the
agentteams-matrix-channelplugin (same as Workers). CMS observability packages installed in the QwenPaw venv (aligned with Worker).Configuration —
QWENPAW_WORKING_DIRset alongside legacyCOPAW_WORKING_DIR. Session-file privacy policy injected into prompts.approval_level=AUTOat top level of agent template. YOLO mode bridged toapproval_level=OFF. Built-in QA Agent disabled. DeadPYTHONPATHentries removed.Tool compatibility —
messageandtaskflowtools read Matrix credentials directly fromagent.jsoninstead of importingcopaw.config.config(not available in qwenpaw venv). Working directory resolution handles bothQWENPAW_WORKING_DIRandqwenpaw.constantfallback.CI — pgrep health check remains
copaw(_worker\.run_copaw_app)? app(Manager process cmdline is unchanged).Plugins at runtime — stored image-local and copied to working dir on startup, since the workspace is a host-mounted volume.
Stack
#1077 merged → retargeted to main ✅
Follow-up
Controller/install/Makefile runtime identifier migration (
copaw→qwenpawfor Manager) is tracked for a separate PR — it touches Go controller code that #1077 also modifies, and should follow the same pattern (addqwenpawalongsidecopaw, then deprecate).