Skip to content

Commit b80e931

Browse files
DenysKuchmaDenysKuchma
committed
add aiTrace plugin for AI-assisted test debugging
1 parent bb6410e commit b80e931

File tree

5 files changed

+983
-0
lines changed

5 files changed

+983
-0
lines changed

‎docs/aitrace.md‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
---
2+
permalink: /aitrace
3+
title: AI Trace Plugin
4+
---
5+
6+
# AI Trace Plugin
7+
8+
AI Trace Plugin generates AI-friendly trace files for debugging with AI agents like Claude Code.
9+
10+
When a test fails, you need to understand what went wrong: what the page looked like, what elements were present, what errors occurred, and what led to the failure. This plugin automatically captures all that information and organizes it in a format optimized for AI analysis.
11+
12+
## Quick Start
13+
14+
Enable the plugin in your `codecept.conf.js`:
15+
16+
```javascript
17+
export const config = {
18+
tests: './*_test.js',
19+
output: './output',
20+
helpers: {
21+
Playwright: {
22+
url: 'https://example.com',
23+
// Optional: Enable HAR/trace for HTTP capture
24+
recordHar: {
25+
mode: 'minimal',
26+
content: 'embed',
27+
},
28+
trace: 'on',
29+
keepTraceForPassedTests: true,
30+
},
31+
},
32+
plugins: {
33+
aiTrace: {
34+
enabled: true,
35+
},
36+
},
37+
}
38+
```
39+
40+
Run tests:
41+
42+
```bash
43+
npx codeceptjs run
44+
```
45+
46+
After test execution, trace files are created in `output/trace_*/trace.md`.
47+
48+
## Artifacts Created
49+
50+
For each test, a `trace_<sha256>` directory is created with the following files:
51+
52+
**trace.md** - AI-friendly markdown file with test execution history
53+
54+
**0000_screenshot.png** - Screenshot for each step
55+
56+
**0000_page.html** - Full HTML of the page at each step
57+
58+
**0000_aria.txt** - ARIA accessibility snapshot (AI-readable structure without HTML noise)
59+
60+
**0000_console.json** - Browser console logs
61+
62+
When HAR or trace recording is enabled in your helper config, links to those files are also included.
63+
64+
## Trace File Format
65+
66+
The `trace.md` file contains a structured execution log with links to all artifacts:
67+
68+
```markdown
69+
file: /path/to/test.js
70+
name: My test scenario
71+
time: 3.45s
72+
---
73+
74+
I am on page "https://example.com"
75+
> navigated to https://example.com/
76+
> [HTML](./0000_page.html)
77+
> [ARIA Snapshot](./0000_aria.txt)
78+
> [Screenshot](./0000_screenshot.png)
79+
> [Browser Logs](0000_console.json) (7 entries)
80+
> HTTP: see [HAR file](../har/...) for network requests
81+
82+
I see "Welcome"
83+
> navigated to https://example.com/
84+
> [HTML](./0001_page.html)
85+
> [ARIA Snapshot](./0001_aria.txt)
86+
> [Screenshot](./0001_screenshot.png)
87+
> [Browser Logs](0001_console.json) (0 entries)
88+
```
89+
90+
## Configuration
91+
92+
### Basic Configuration
93+
94+
```javascript
95+
plugins: {
96+
aiTrace: {
97+
enabled: true,
98+
}
99+
}
100+
```
101+
102+
### Advanced Configuration
103+
104+
```javascript
105+
plugins: {
106+
aiTrace: {
107+
enabled: true,
108+
109+
// Artifact capture options
110+
captureHTML: true, // Save HTML for each step
111+
captureARIA: true, // Save ARIA snapshots
112+
captureBrowserLogs: true, // Save console logs
113+
captureHTTP: true, // Links to HAR/trace files
114+
captureDebugOutput: true, // CodeceptJS debug messages
115+
116+
// Screenshot options
117+
fullPageScreenshots: false, // Full page or viewport only
118+
119+
// Output options
120+
output: './output', // Where to save traces
121+
deleteSuccessful: false, // Delete traces for passed tests
122+
123+
// Step filtering
124+
ignoreSteps: [
125+
/^grab/, // Ignore all grab* steps
126+
/^wait/, // Ignore all wait* steps
127+
],
128+
}
129+
}
130+
```
131+
132+
### Configuration Options
133+
134+
| Option | Type | Default | Description |
135+
|--------|------|---------|-------------|
136+
| `enabled` | boolean | `false` | Enable/disable the plugin |
137+
| `captureHTML` | boolean | `true` | Capture HTML for each step |
138+
| `captureARIA` | boolean | `true` | Capture ARIA snapshots |
139+
| `captureBrowserLogs` | boolean | `true` | Capture browser console logs |
140+
| `captureHTTP` | boolean | `true` | Capture HTTP requests (requires HAR/trace) |
141+
| `captureDebugOutput` | boolean | `true` | Capture CodeceptJS debug output |
142+
| `fullPageScreenshots` | boolean | `false` | Use full page screenshots |
143+
| `output` | string | `'./output'` | Directory for trace files |
144+
| `deleteSuccessful` | boolean | `false` | Delete traces for passed tests |
145+
| `ignoreSteps` | array | `[]` | Steps to ignore (regex patterns) |
146+
147+
## Best Practices
148+
149+
### Optimize for Failing Tests
150+
151+
Save disk space by only keeping traces for failed tests:
152+
153+
```javascript
154+
plugins: {
155+
aiTrace: {
156+
enabled: true,
157+
deleteSuccessful: true, // Only keep failing tests
158+
}
159+
}
160+
```
161+
162+
### Ignore Noise
163+
164+
Don't capture logs for `grab` and `wait` steps:
165+
166+
```javascript
167+
plugins: {
168+
aiTrace: {
169+
enabled: true,
170+
ignoreSteps: [/^grab/, /^wait/],
171+
}
172+
}
173+
```
174+
175+
### Selective Artifact Capture
176+
177+
Capture only what you need to reduce file sizes:
178+
179+
```javascript
180+
plugins: {
181+
aiTrace: {
182+
enabled: true,
183+
captureHTML: false, // Skip HTML (saves ~500KB per step)
184+
captureARIA: true, // Keep ARIA (only ~16KB)
185+
captureBrowserLogs: false, // Skip console logs
186+
}
187+
}
188+
```
189+
190+
### Enable HTTP Capture
191+
192+
For network debugging, enable HAR/trace in your helper:
193+
194+
```javascript
195+
helpers: {
196+
Playwright: {
197+
recordHar: {
198+
mode: 'minimal',
199+
content: 'embed',
200+
},
201+
trace: 'on',
202+
keepTraceForPassedTests: true,
203+
},
204+
plugins: {
205+
aiTrace: {
206+
enabled: true,
207+
captureHTTP: true, // Links to HAR/trace files
208+
},
209+
},
210+
}
211+
```
212+
213+
## Using with AI Agents
214+
215+
The trace format is optimized for AI agents like Claude Code. When debugging a failing test:
216+
217+
1. Open the generated `trace.md` file
218+
2. Copy its contents along with relevant artifact files (ARIA snapshots, console logs, etc.)
219+
3. Provide to the AI agent with context about the failure
220+
221+
Example prompt:
222+
```
223+
I have a failing test. Here's the AI trace:
224+
225+
[paste trace.md contents]
226+
227+
[paste relevant ARIA snapshots]
228+
229+
[paste console logs]
230+
231+
Analyze this and explain why the test failed and how to fix it.
232+
```
233+
234+
The AI agent can analyze all artifacts together - screenshots, HTML structure, console errors, and network requests - to provide comprehensive debugging insights.
235+
236+
## Troubleshooting
237+
238+
### No trace files created
239+
240+
**Possible causes:**
241+
1. Plugin not enabled
242+
2. No steps executed
243+
3. Tests skipped
244+
245+
**Solution:**
246+
```bash
247+
# Check if plugin is enabled
248+
grep -A 5 "aiTrace" codecept.conf.js
249+
250+
# Run with verbose output
251+
npx codeceptjs run --verbose
252+
```
253+
254+
### ARIA snapshots missing
255+
256+
**Possible cause:** Helper doesn't support `grabAriaSnapshot`
257+
258+
**Solution:** Use Playwright or update to latest CodeceptJS
259+
260+
### HAR files missing
261+
262+
**Possible cause:** HAR/trace not enabled in helper config
263+
264+
**Solution:**
265+
```javascript
266+
helpers: {
267+
Playwright: {
268+
recordHar: { mode: 'minimal' },
269+
trace: 'on',
270+
},
271+
}
272+
```
273+
274+
## Related
275+
276+
- [AI Features](/ai) - AI-powered testing features
277+
- [Plugins](/plugins) - All available plugins
278+
- [Configuration](/configuration) - General configuration
279+
- [Playwright Helper](/playwright) - Playwright-specific configuration

‎docs/plugins.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,114 @@ exports.config = {
6363

6464
Returns **void**&#x20;
6565

66+
## aiTrace
67+
68+
Generates AI-friendly trace files for debugging with AI agents like Claude Code.
69+
70+
When a test fails, you need to understand what went wrong. This plugin automatically captures comprehensive information about test execution - screenshots, HTML, ARIA snapshots, console logs, and HTTP requests - and organizes it in a format optimized for AI analysis.
71+
72+
The generated trace files are structured markdown documents that AI agents can easily parse to understand test context and provide debugging insights.
73+
74+
#### Usage
75+
76+
Enable this plugin in your config:
77+
78+
```js
79+
// in codecept.conf.js
80+
exports.config = {
81+
plugins: {
82+
aiTrace: {
83+
enabled: true
84+
}
85+
}
86+
}
87+
```
88+
89+
#### Configuration
90+
91+
* `deleteSuccessful` (boolean) - delete traces for successfully executed tests. Default: false.
92+
* `fullPageScreenshots` (boolean) - should full page screenshots be used. Default: false.
93+
* `output` (string) - a directory where traces should be stored. Default: `output`.
94+
* `captureHTML` (boolean) - capture HTML for each step. Default: true.
95+
* `captureARIA` (boolean) - capture ARIA snapshot for each step. Default: true.
96+
* `captureBrowserLogs` (boolean) - capture browser console logs. Default: true.
97+
* `captureHTTP` (boolean) - capture HTTP requests (requires `trace` or `recordHar` enabled in helper config). Default: true.
98+
* `captureDebugOutput` (boolean) - capture CodeceptJS debug output. Default: true.
99+
* `ignoreSteps` (array) - steps to ignore in trace. Array of RegExps is expected. Default: [].
100+
101+
#### Artifacts Created
102+
103+
For each test, a `trace_<sha256>` directory is created with:
104+
105+
* **trace.md** - AI-friendly markdown file with test execution history
106+
* **0000_screenshot.png** - screenshot for each step
107+
* **0000_page.html** - full HTML of the page at each step
108+
* **0000_aria.txt** - ARIA accessibility snapshot (AI-readable structure)
109+
* **0000_console.json** - browser console logs
110+
111+
When HAR or trace recording is enabled in your helper config, links to those files are also included.
112+
113+
#### Example Output
114+
115+
```markdown
116+
file: /path/to/test.js
117+
name: My test scenario
118+
time: 3.45s
119+
---
120+
121+
I am on page "https://example.com"
122+
> navigated to https://example.com/
123+
> [HTML](./0000_page.html)
124+
> [ARIA Snapshot](./0000_aria.txt)
125+
> [Screenshot](./0000_screenshot.png)
126+
> [Browser Logs](0000_console.json) (7 entries)
127+
> HTTP: see [HAR file](../har/...) for network requests
128+
129+
I see "Welcome"
130+
> navigated to https://example.com/
131+
> [HTML](./0001_page.html)
132+
> [ARIA Snapshot](./0001_aria.txt)
133+
> [Screenshot](./0001_screenshot.png)
134+
> [Browser Logs](0001_console.json) (0 entries)
135+
```
136+
137+
#### Best Practices
138+
139+
**Save disk space** - Only keep traces for failed tests:
140+
141+
```js
142+
aiTrace: {
143+
enabled: true,
144+
deleteSuccessful: true
145+
}
146+
```
147+
148+
**Ignore noise** - Don't capture logs for `grab` and `wait` steps:
149+
150+
```js
151+
aiTrace: {
152+
enabled: true,
153+
ignoreSteps: [/^grab/, /^wait/]
154+
}
155+
```
156+
157+
**Reduce file sizes** - Capture only what you need:
158+
159+
```js
160+
aiTrace: {
161+
enabled: true,
162+
captureHTML: false, // Skip HTML (saves ~500KB per step)
163+
captureARIA: true, // Keep ARIA (only ~16KB)
164+
captureBrowserLogs: false // Skip console logs
165+
}
166+
```
167+
168+
### Parameters
169+
170+
* `config` **[Object][1]** Plugin configuration (optional, default `{}`)
171+
172+
Returns **void**&#x20;
173+
66174
## auth
67175

68176
Logs user in for the first test and reuses session for next tests.

0 commit comments

Comments
 (0)