Skip to content

Commit 9602209

Browse files
committed
feat(e2e): integrate e2e package and update test configurations
- Added e2e package as a workspace dependency in pnpm-lock.yaml. - Updated globalSetup path in vite.config.ts to point to the new e2e package location. - Enhanced documentation for CLI E2E testing, including architecture layers and triggering conditions. - Refactored test paths and commands in the documentation to align with the new structure. - Removed outdated dataset and auth E2E tests to streamline the test suite. - Updated command references and validation checks in the documentation to reflect recent changes.
1 parent 49095c3 commit 9602209

70 files changed

Lines changed: 1743 additions & 1026 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/agents/cli-e2e-tests.md

Lines changed: 64 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,44 @@
11
# CLI E2E 测试规范
22

3+
## 架构分层
4+
5+
| 层级 | 路径 | 测什么 |
6+
| --------------- | ----------------------------------------------------- | --------------------------------------------------------------- |
7+
| **共享基建** | `packages/e2e` | gating、子进程 runner、output、globalSetup(`private`,不发布) |
8+
| **命令 E2E** | `packages/commands/tests/e2e` | help、缺参、dry-run、live(gated);harness `binName: "bl"` |
9+
| **bl smoke** | `packages/cli/tests/e2e/registry.smoke.e2e.test.ts` | 产品 map 全部 path `--help`、分组 help、根 help |
10+
| **kscli smoke** | `packages/kscli/tests/e2e/registry.smoke.e2e.test.ts` | 6 条 flat path `--help``search --help` 与 harness 一致 |
11+
| **runtime** | `packages/runtime/tests` | `proxy.e2e`、console 跨域 flag 拒绝 |
12+
13+
**依赖边界**`e2e``core``commands/tests``e2e` + `commands/src`;产品 tests → `e2e` + 各自 `src`**禁止**产品 import `commands/tests/**`(子进程 spawn harness 路径除外)。
14+
315
## 触发条件
416

517
- 新增/修改 `packages/commands/src/commands` 下的 command 实现
618
- 新增/修改 `packages/cli/src/commands.ts``bl` 命令路径 map
7-
- 新建或扩展 `packages/cli/tests/e2e/*.e2e.test.ts` 用例
8-
- 为命令补 help / 缺参 / dry-run / 真实集成测试
19+
- 新建或扩展 `packages/commands/tests/e2e/<topic>.e2e.test.ts`
20+
- 新增 bl 产品 path → 同步 `registry.smoke`(自动覆盖 leaf path)与 `harness/e2e-command-map.ts`
921

10-
以上情况必须同步维护 `packages/cli/tests/e2e/<topic>.e2e.test.ts`跑测与环境变量见 `.cursor/skills/bailian-cli-e2e/SKILL.md`
22+
跑测与环境变量见 `.cursor/skills/bailian-cli-e2e/SKILL.md`
1123

1224
## 文件与工具
1325

14-
- 路径:`packages/cli/tests/e2e/<kebab-topic>.e2e.test.ts`
15-
- 框架:`vite-plus/test`;子进程跑 CLI:`runCli` from `./helpers.ts`
26+
### commands E2E
27+
28+
- 路径:`packages/commands/tests/e2e/<kebab-topic>.e2e.test.ts`
29+
- 子进程:`runCommandE2e` from `./helpers.ts`(spawn `harness/main.ts`
30+
- fixtures:`packages/commands/tests/e2e/fixtures/`
31+
- map:`harness/e2e-command-map.ts`(自维护,初始从 `cli/commands.ts` 复制)
32+
33+
### 产品 smoke
34+
35+
- bl:`runCli` from `packages/cli/tests/e2e/helpers.ts`
36+
- kscli:`runKscli` from `packages/kscli/tests/e2e/helpers.ts`
37+
38+
### 共享
39+
40+
- gating / output / runner:`e2e/gating``e2e/output``e2e/runner`
41+
- globalSetup:根 `vite.config.ts``packages/e2e/src/global-setup.ts`
1642
- 解析 JSON stdout:`parseStdoutJson`;输出目录:`makeE2eOutputDir(e2eLabelFromMetaUrl(import.meta.url))`
1743
- 长任务:`cliTimeoutPrefix()`;视频用例加 `test(..., 3_600_000)` 等显式超时
1844

@@ -33,24 +59,31 @@ describe.skipIf(<ready>)("e2e: <topic>(DashScope …)", () => {
3359
});
3460
```
3561

36-
## skip 条件(helpers.ts
62+
## skip 条件(`e2e/gating`,commands helpers re-export
3763

38-
| 场景 | 条件 |
39-
| ------------------- | ----------------------------------------------------- |
40-
| 文本/搜索/记忆/配置 | `isDashScopeE2EReady()` |
41-
| 图像/语音 | `isBailianE2EMediaEnabled() && isDashScopeE2EReady()` |
42-
| 视频 | `isBailianE2EVideoEnabled() && isDashScopeE2EReady()` |
43-
| 知识库 | `isKnowledgeE2EReady()` |
44-
| 视频 download/task | 另需 `BAILIAN_E2E_VIDEO_TASK_ID` |
64+
| 场景 | 条件 |
65+
| ----------------------- | ---------------------------------------------------------------------------------------------------------- |
66+
| 文本/搜索/记忆/配置 | `isDashScopeE2EReady()` |
67+
| 图像/语音 | `isBailianE2EMediaEnabled() && isDashScopeE2EReady()` |
68+
| 视频 | `isBailianE2EVideoEnabled() && isDashScopeE2EReady()` |
69+
| 视频 download/task | 另需 `BAILIAN_E2E_VIDEO_TASK_ID` |
70+
| 知识库 chat/search live | `isChatE2EReady()` / `isSearchE2EReady()``knowledge chat/search`,需 `BAILIAN_WORKSPACE_ID` + agent ID) |
4571

4672
## 用例类型
4773

48-
1. **分组 help**`runCli(["image"])``exitCode === 0`,stdout+stderr 含子命令名
49-
2. **--help**`runCli([..., "--help"])` → stderr 含主要 flags
74+
1. **分组 help**`runCommandE2e(["image"])``exitCode === 0`,stdout+stderr 含子命令名
75+
2. **--help**`runCommandE2e([..., "--help"])` → stderr 含主要 flags
5076
3. **缺参**:带一个无害全局 flag(如 `--quiet`)且不传 required flag → `exitCode === 2`,stderr 匹配 `--flag|Missing required argument`
5177
4. **--dry-run**:仅当实现在联网/上传/写盘**之前**返回;断言 stdout JSON/文本,不入网
5278
5. **真实集成**:保留既有用例名称与断言;放在 skip 块**末尾**
5379

80+
## 契约与防漂移
81+
82+
- `harness/e2e-command-map.contract.test.ts`:path 唯一、合法 export、白名单、测试 argv 前缀在 map 中
83+
- Advisory(不阻塞 CI):`node tools/compare-e2e-command-map.ts` 对比 `cli/commands.ts` 与 harness map
84+
85+
增删命令须同步三处:**实现 export** + **`cli/commands.ts`** + **`e2e-command-map.ts`**(见 [command-add-remove.md](command-add-remove.md))。
86+
5487
## 安全与例外
5588

5689
- **禁止真实破坏性操作**`auth logout` 只用 `--dry-run``config set` 只用 `--dry-run`
@@ -60,24 +93,35 @@ describe.skipIf(<ready>)("e2e: <topic>(DashScope …)", () => {
6093

6194
## 新增 command 检查清单
6295

63-
- [ ] `packages/commands/src/index.ts` 导出 + `packages/cli/src/commands.ts` 暴露路径 + `tests/e2e/<topic>.e2e.test.ts`(新建或扩展)
96+
- [ ] `packages/commands/src/index.ts` 导出 + `packages/cli/src/commands.ts` 暴露路径 + `harness/e2e-command-map.ts` 登记
97+
- [ ] `packages/commands/tests/e2e/<topic>.e2e.test.ts`(新建或扩展)
6498
- [ ] 若改了 `usageArgs` / `flags` / `exampleArgs`,跑 `pnpm --filter bailian-cli run generate:reference` 更新 `skills/bailian-cli/reference/` 并提交
6599
- [ ] 顶层:分组 help + 子命令 `--help`(多子命令则各一条 help)
66100
- [ ] skip 块:每个 required flag 缺参;可 dry-run 则加一条
67101
- [ ] 至少一条真实集成(或说明为何仅 smoke);不破坏已有集成用例顺序
68-
- [ ] `pnpm test packages/cli/tests/e2e/<file>` 通过
102+
- [ ] `vp test packages/commands/tests/e2e/<file>` 通过
103+
104+
## 调试命令
105+
106+
```sh
107+
pnpm --filter bailian-cli-commands exec vp test packages/commands/tests/e2e/text-chat.e2e.test.ts
108+
pnpm --filter bailian-cli exec vp test packages/cli/tests/e2e/registry.smoke.e2e.test.ts
109+
pnpm --filter knowledge-studio-cli exec vp test packages/kscli/tests/e2e/registry.smoke.e2e.test.ts
110+
pnpm --filter bailian-cli-runtime exec vp test packages/runtime/tests/proxy.e2e.test.ts
111+
node tools/compare-e2e-command-map.ts
112+
```
69113

70114
## 示例片段
71115

72116
```ts
73117
test("foo bar 缺少 --prompt 时退出为用法错误 (2)", async () => {
74-
const { stderr, exitCode } = await runCli(["foo", "bar", "--quiet"]);
118+
const { stderr, exitCode } = await runCommandE2e(["foo", "bar", "--quiet"]);
75119
expect(exitCode).toBe(2);
76120
expect(stderr).toMatch(/--prompt|Missing required argument/i);
77121
});
78122

79123
test("foo bar --dry-run 仅输出计划", async () => {
80-
const { stdout, stderr, exitCode } = await runCli([
124+
const { stdout, stderr, exitCode } = await runCommandE2e([
81125
"foo",
82126
"bar",
83127
"--dry-run",
@@ -97,4 +141,4 @@ test("foo bar --dry-run 仅输出计划", async () => {
97141
- **E2E**:单条/少量调用、断言固定、可进 `vp test`(见上文 skip 条件)
98142
- **批量压测**`packages/cli/tests/stress/run.mjs` + `targets/*.mjs`,并发 + 报告,**仅手动** `pnpm run test:stress -- <target>`
99143

100-
勿把压测并入 E2E 或默认 CI。详见 [stress-batch-tests.md](stress-batch-tests.md)
144+
勿把压测并入 E2E 或默认 CI。详见 [stress-batch-tests.md](stress-batch-tests.md).

docs/agents/command-add-remove.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -92,9 +92,12 @@ packages/commands/src/index.ts
9292

9393
### D. 测试层
9494

95-
- [ ][cli-e2e-tests.md](cli-e2e-tests.md) 新建或更新 `packages/cli/tests/e2e/<topic>.e2e.test.ts`
96-
- [ ] 删除命令时一并删对应 e2e / README 示例 / reference 生成结果
97-
- [ ] 如果 shared command 在不同入口路径下复用,至少确保 `bl` 入口 e2e 覆盖;`rag` 入口改动需补对应入口测试或手工 smoke
95+
- [ ][cli-e2e-tests.md](cli-e2e-tests.md) 新建或更新 `packages/commands/tests/e2e/<topic>.e2e.test.ts`
96+
- [ ] 同步 `packages/commands/tests/e2e/harness/e2e-command-map.ts``E2E_COMMAND_EXPORTS`
97+
- [ ] `packages/cli/src/commands.ts` 增删 path 后跑 `node tools/compare-e2e-command-map.ts`(advisory)确认与 harness 一致
98+
- [ ] bl 产品 path 变更由 `packages/cli/tests/e2e/registry.smoke.e2e.test.ts` 覆盖;kscli 变更同步 `packages/kscli/tests/e2e/registry.smoke.e2e.test.ts`
99+
- [ ] 删除命令时一并删对应 commands e2e / README 示例 / reference 生成结果 / harness map 条目
100+
- [ ] 如果 shared command 在不同入口路径下复用,至少确保 commands e2e 覆盖 `bl` path;`rag` / `kscli` 入口改动需补对应 smoke 或说明不测 flat path live
98101

99102
### E. 重命名特殊处理
100103

@@ -111,7 +114,8 @@ packages/commands/src/index.ts
111114
pnpm run sync:skill-assets
112115
node packages/cli/src/main.ts <new-command> --help
113116
node packages/cli/src/main.ts
114-
vp test packages/cli/tests/e2e/<topic>.e2e.test.ts
117+
vp test packages/commands/tests/e2e/<topic>.e2e.test.ts
118+
node tools/compare-e2e-command-map.ts
115119
```
116120

117121
如改了 `rag` 入口:

packages/cli/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@
5959
"ajv": "catalog:",
6060
"boxen": "catalog:",
6161
"chalk": "catalog:",
62+
"e2e": "workspace:*",
6263
"typescript": "^6.0.2",
6364
"undici": "catalog:",
6465
"vite-plus": "0.1.22",

0 commit comments

Comments
 (0)