diff --git a/.claude/skills/bug-hunt/SKILL.md b/.claude/skills/bug-hunt/SKILL.md new file mode 100644 index 0000000..397256e --- /dev/null +++ b/.claude/skills/bug-hunt/SKILL.md @@ -0,0 +1,454 @@ +--- +name: bug-hunt +description: Hunt for instrumentation bugs by analyzing code gaps and running e2e tests through DISABLED/RECORD/REPLAY cycle +disable-model-invocation: true +--- + +# Instrumentation Bug Hunting + +## Arguments + +$ARGUMENTS - The library name, optionally followed by focus context. + +**Format**: ` [focus on ]` + +**Examples**: + +- `/bug-hunt redis` — broad bug hunting across all redis functionality +- `/bug-hunt redis focus on pub sub interactions` — prioritize pub/sub patterns +- `/bug-hunt mysql2 focus on prepared statements and transactions` — prioritize those areas + +**Parsing**: The first word of `$ARGUMENTS` is always the library name. Everything after it is the optional focus context. All references to `` below mean this parsed first word — NOT the raw `$ARGUMENTS` string. + +## Library-to-GitHub-Repo Mapping + +Use this mapping to clone the package source code for analysis: + +| Library | GitHub Repo | Notes | +| ---------------- | ---------------------------------------------- | --------------------------------------- | +| mysql2 | https://github.com/sidorares/node-mysql2 | | +| redis | https://github.com/redis/node-redis | Monorepo — focus on `packages/client/` | +| pg | https://github.com/brianc/node-postgres | Monorepo — focus on `packages/pg/` | +| mongodb | https://github.com/mongodb/node-mongodb-native | | +| ioredis | https://github.com/redis/ioredis | | +| postgres | https://github.com/porsager/postgres | (postgres.js) | +| prisma | https://github.com/prisma/prisma | Monorepo — focus on `packages/client/` | +| firestore | https://github.com/googleapis/nodejs-firestore | | +| grpc | https://github.com/grpc/grpc-node | Monorepo — focus on `packages/grpc-js/` | +| fetch | N/A | Built-in Node.js API — no repo to clone | +| http | N/A | Built-in Node.js API — no repo to clone | +| mysql | https://github.com/mysqljs/mysql | | +| nextjs | https://github.com/vercel/next.js | Monorepo — focus on `packages/next/` | +| upstash-redis-js | https://github.com/upstash/redis-js | | + +## E2E Test Variants + +Each library has ESM and CJS variants. Use the **CJS variant** as the primary target for bug hunting: + +| Library | CJS variant path | +| ---------------- | -------------------------------------------------------------------------------- | +| mysql2 | `src/instrumentation/libraries/mysql2/e2e-tests/cjs-mysql2/` | +| redis | `src/instrumentation/libraries/redis/e2e-tests/cjs-redis/` | +| pg | `src/instrumentation/libraries/pg/e2e-tests/cjs-pg/` | +| mongodb | `src/instrumentation/libraries/mongodb/e2e-tests/cjs-mongodb/` | +| ioredis | `src/instrumentation/libraries/ioredis/e2e-tests/cjs-ioredis/` | +| postgres | `src/instrumentation/libraries/postgres/e2e-tests/cjs-postgres/` | +| prisma | `src/instrumentation/libraries/prisma/e2e-tests/cjs-prisma/` | +| firestore | `src/instrumentation/libraries/firestore/e2e-tests/cjs-firestore/` | +| grpc | `src/instrumentation/libraries/grpc/e2e-tests/cjs-grpc/` | +| fetch | `src/instrumentation/libraries/fetch/e2e-tests/cjs-fetch/` | +| http | `src/instrumentation/libraries/http/e2e-tests/cjs-http/` | +| mysql | `src/instrumentation/libraries/mysql/e2e-tests/cjs-mysql/` | +| nextjs | `src/instrumentation/libraries/nextjs/e2e-tests/cjs-nextjs/` | +| upstash-redis-js | `src/instrumentation/libraries/upstash-redis-js/e2e-tests/cjs-upstash-redis-js/` | + +--- + +## Phase 0: Environment Setup + +### 0.1 Parse and validate the arguments + +Extract the library name (first word) and optional focus context (remaining words) from the arguments. + +The library must be one of: fetch, firestore, grpc, http, ioredis, mongodb, mysql, mysql2, nextjs, pg, postgres, prisma, redis, upstash-redis-js. + +If the library is invalid, list the valid options and stop. + +If focus context is provided, it will guide Phases 1 and 2 to prioritize that area of the library's functionality. + +### 0.2 Docker Setup (Claude Code Web only) + +Check if Docker is running. If not, start it: + +```bash +dockerd --storage-driver=vfs &>/tmp/dockerd.log & +# Wait for Docker to be ready +for i in $(seq 1 30); do + docker info &>/dev/null 2>&1 && break + sleep 1 +done +docker info &>/dev/null 2>&1 || { echo "Docker failed to start. Check /tmp/dockerd.log"; exit 1; } +``` + +If Docker is already running, skip this step. + +### 0.3 Clone the package source code (for analysis only) + +If the library has a GitHub repo (see mapping above), clone it for reference: + +```bash +git clone --depth 1 /tmp/-source +``` + +This is read-only reference material — you will NOT modify this repo. + +### 0.4 Create a working branch + +Skip this step if you are already on a dedicated branch (e.g., in Claude Code Web where each session has its own branch). + +```bash +git checkout -b bug-hunt/-$(date +%Y-%m-%d) +``` + +--- + +## Phase 1: Develop Understanding + +**If focus context was provided**, prioritize your analysis around that area. For example, if the focus is "pub sub interactions", concentrate on pub/sub-related code paths in the instrumentation, tests, and package source. + +### 1.1 Analyze the Instrumentation Code + +Read the instrumentation code at: + +``` +src/instrumentation/libraries//Instrumentation.ts +``` + +Identify: + +- Which functions from the package are patched/instrumented +- The patching strategy (what gets wrapped, when, and how) +- Any helper files in the same directory +- **If focus context provided**: Which patches relate to the focus area, and what's missing? + +### 1.2 Analyze Existing E2E Tests + +Review the CJS variant's test files: + +- `src/instrumentation/libraries//e2e-tests/cjs-/src/index.ts` — all test endpoints +- `src/instrumentation/libraries//e2e-tests/cjs-/src/test_requests.mjs` — which endpoints are called + +Understand what functionality is already tested and identify coverage gaps. + +- **If focus context provided**: What tests already exist for the focus area? What's missing? + +### 1.3 Analyze the Package Source Code + +If you cloned the package source, read it to understand: + +- The package's entry points and full API surface +- Functions that are currently patched vs functions that exist but aren't patched +- Alternative call patterns, overloads, and edge cases +- **If focus context provided**: Deep-dive into the focus area's API surface and usage patterns + +--- + +## Phase 2: Identify Potential Gaps + +**If focus context was provided**, prioritize bugs related to that area. You may still note other potential issues, but test the focus area first. + +Reason about potential issues. Consider: + +- **Untested parameters**: Parameter combinations not covered by existing tests +- **Alternative call patterns**: Can patched functions be invoked differently (callbacks vs promises, different overloads)? +- **Missing patches**: Functions that should be instrumented but aren't +- **Edge cases**: Null/undefined values, empty results, large payloads, streaming, connection errors +- **ORM/wrapper usage**: Libraries like Sequelize, Knex, Prisma that wrap the base driver — are those call paths instrumented? +- **Real-world usage patterns**: How is the package typically used in production apps? + +Produce a prioritized list of potential bugs to investigate. + +--- + +## Phase 3: Initialize Bug Tracking Document + +Create `BUG_TRACKING.md` in the CJS e2e test directory: + +```bash +# Path: src/instrumentation/libraries//e2e-tests/cjs-/BUG_TRACKING.md +``` + +```markdown +# Instrumentation Bug Tracking + +Generated: + +## Summary + +- Total tests attempted: 0 +- Confirmed bugs: 0 +- No bugs found: 0 +- Skipped tests: 0 + +--- + +## Test Results + +(Tests will be documented below as they are completed) +``` + +--- + +## Phase 4: Write Tests and Verify Issues + +For each potential bug, follow this workflow: + +### 4.1 Initial Setup (Once) + +Navigate to the CJS e2e test directory: + +```bash +cd src/instrumentation/libraries//e2e-tests/cjs-/ +``` + +Start Docker containers: + +```bash +docker compose up -d --build --wait +``` + +Install dependencies: + +```bash +docker compose exec -T app npm install +``` + +### 4.2 Test Each Potential Bug (Repeat for each) + +#### A. Clean Previous Test Data + +```bash +rm -rf .tusk/traces/* .tusk/logs/* +``` + +#### B. Write New Test Endpoint + +Add a new endpoint to `src/index.ts` that exercises the potential bug. Also add the corresponding request to `src/test_requests.mjs`. + +Example: + +```typescript +app.get("/test/my-new-test", async (req, res) => { + // Your test code here + res.json({ success: true }); +}); +``` + +#### C. Test in DISABLED Mode (No Instrumentation) + +Start server without instrumentation: + +```bash +docker compose exec -d -e TUSK_DRIFT_MODE=DISABLED app sh -c "npm run build && npm run dev" +``` + +Wait for server to start: + +```bash +sleep 5 +``` + +Hit the endpoint: + +```bash +docker compose exec app curl -s http://localhost:3000/test/my-new-test +``` + +**Verify**: Response is correct and endpoint works. + +Stop the server: + +```bash +docker compose exec app pkill -f "node" || true +sleep 2 +``` + +**If the endpoint fails in DISABLED mode**: + +- Update `BUG_TRACKING.md` with status: "Skipped - Failed in DISABLED mode" +- Fix the test code or move on to next potential bug + +#### D. Test in RECORD Mode (With Instrumentation) + +Clean traces and logs: + +```bash +rm -rf .tusk/traces/* .tusk/logs/* +``` + +Start server in RECORD mode: + +```bash +docker compose exec -d -e TUSK_DRIFT_MODE=RECORD app sh -c "npm run build && npm run dev" +``` + +Wait for server to start: + +```bash +sleep 5 +``` + +Hit the endpoint: + +```bash +docker compose exec app curl -s http://localhost:3000/test/my-new-test +``` + +Wait for spans to export: + +```bash +sleep 3 +``` + +Stop the server: + +```bash +docker compose exec app pkill -f "node" || true +sleep 2 +``` + +**Check for issues:** + +1. **Endpoint returns error or wrong response vs DISABLED mode**: + - BUG FOUND: Instrumentation breaks functionality + - Update `BUG_TRACKING.md`: Status "Confirmed Bug - RECORD mode failure", Failure Point "RECORD" + - Keep the endpoint, move to next + +2. **No traces created** (`ls .tusk/traces/`): + - BUG FOUND: Instrumentation failed to capture traffic + - Update `BUG_TRACKING.md`: Status "Confirmed Bug - No traces captured", Failure Point "RECORD" + - Keep the endpoint, move to next + +#### E. Test in REPLAY Mode + +Run the Tusk CLI to replay: + +```bash +docker compose exec -T -e TUSK_ANALYTICS_DISABLED=1 app tusk drift run --print --output-format "json" --enable-service-logs --disable-sandbox +``` + +**Check for issues:** + +1. **Test fails** (`"passed": false` in JSON output): + - BUG FOUND: Replay doesn't match recording + - Update `BUG_TRACKING.md`: Status "Confirmed Bug - REPLAY mismatch", Failure Point "REPLAY" + +2. **No logs created** (`ls .tusk/logs/`): + - BUG FOUND: Replay failed to produce logs + - Update `BUG_TRACKING.md`: Status "Confirmed Bug - No replay logs", Failure Point "REPLAY" + +3. **Logs contain TCP warnings**: + + ```bash + docker compose exec app cat .tusk/logs/*.log | grep -i "TCP called from inbound request context" + ``` + + - BUG FOUND: Unpatched dependency detected + - Update `BUG_TRACKING.md`: Status "Confirmed Bug - Unpatched dependency", Failure Point "REPLAY" + +#### F. No Bug Found + +If all modes pass with no issues: + +- Update `BUG_TRACKING.md`: Status "No Bug - Test passed all modes" +- **Remove the test endpoint** from `src/index.ts` and `src/test_requests.mjs` +- Move to next potential bug + +--- + +## Phase 5: Bug Tracking Documentation Format + +After each test, append to `BUG_TRACKING.md`: + +```markdown +### Test N: [Brief description] + +**Status**: [Confirmed Bug | No Bug | Skipped] + +**Endpoint**: `/test/endpoint-name` + +**Failure Point**: [DISABLED | RECORD | REPLAY | N/A] + +**Description**: +[What this test was trying to uncover] + +**Expected Behavior**: +[What should happen] + +**Actual Behavior**: +[What actually happened] + +**Error Logs**: +``` + +[Relevant error messages, stack traces, or warnings] + +``` + +**Additional Notes**: +[Observations, potential root causes, context] + +--- +``` + +**Important**: Update `BUG_TRACKING.md` immediately after each test — do not batch updates. + +--- + +## Phase 6: Cleanup and Commit + +After testing all potential bugs: + +```bash +docker compose down +``` + +Clean up cloned package source: + +```bash +rm -rf /tmp/*-source +``` + +**Final state of the e2e test files:** + +- `src/index.ts` should contain ONLY the original endpoints + new endpoints that expose confirmed bugs +- `src/test_requests.mjs` should be updated to include requests to bug-exposing endpoints +- `BUG_TRACKING.md` should have accurate summary counts and all test results + +Commit the changes: + +```bash +git add src/instrumentation/libraries//e2e-tests/cjs-/ +git commit -m "bug-hunt(): add e2e tests exposing instrumentation bugs + +Found N confirmed bugs in instrumentation. +See BUG_TRACKING.md for details." +``` + +Push the branch (skip if in Claude Code Web where the session handles this): + +```bash +git push origin bug-hunt/-$(date +%Y-%m-%d) +``` + +--- + +## Success Criteria + +1. Created `BUG_TRACKING.md` before starting any tests +2. Tested all identified potential bugs +3. Updated `BUG_TRACKING.md` after each individual test +4. Only bug-exposing endpoints remain in test files +5. Removed test endpoints that didn't expose bugs +6. Accurate summary counts in `BUG_TRACKING.md` +7. Changes committed and pushed to a `bug-hunt/` branch diff --git a/.gitignore b/.gitignore index e04ee09..5e64844 100644 --- a/.gitignore +++ b/.gitignore @@ -43,8 +43,6 @@ coverage/ ehthumbs.db Thumbs.db -**/BUG_TRACKING.md - # E2E test generated files **/e2e-tests/**/.tusk/logs/ **/e2e-tests/**/.tusk/traces/ diff --git a/E2E_TESTING_GUIDE.md b/E2E_TESTING_GUIDE.md index c141433..2c094cb 100644 --- a/E2E_TESTING_GUIDE.md +++ b/E2E_TESTING_GUIDE.md @@ -72,7 +72,7 @@ docker compose exec -T app npm install Start the application server in RECORD mode to capture network traffic: ```bash -docker compose exec -e TUSK_DRIFT_MODE=RECORD app sh -c "npm run build && npm run dev" +docker compose exec -d -e TUSK_DRIFT_MODE=RECORD app sh -c "npm run build && npm run dev" ``` Wait a few seconds for the server to fully start (5-10 seconds recommended): @@ -119,7 +119,7 @@ sleep 2 Run the Tusk CLI to replay the recorded traces: ```bash -docker compose exec -T -e TUSK_ANALYTICS_DISABLED=1 app tusk run --print --output-format "json" --enable-service-logs +docker compose exec -T -e TUSK_ANALYTICS_DISABLED=1 app tusk drift run --print --output-format "json" --enable-service-logs ``` **Flags explained:** @@ -131,7 +131,7 @@ docker compose exec -T -e TUSK_ANALYTICS_DISABLED=1 app tusk run --print --outpu To see all available flags, run: ```bash -tusk run --help +tusk drift run --help ``` **Interpreting Results:** @@ -262,7 +262,7 @@ docker compose exec -d -e TUSK_DRIFT_MODE=RECORD app sh -c "npm run build && npm docker compose exec app pkill -f "node" || true # Run tests -docker compose exec -T -e TUSK_ANALYTICS_DISABLED=1 app tusk run --print --output-format "json" --enable-service-logs +docker compose exec -T -e TUSK_ANALYTICS_DISABLED=1 app tusk drift run --print --output-format "json" --enable-service-logs # View logs docker compose exec app ls .tusk/logs diff --git a/docs/environment-variables.md b/docs/environment-variables.md index fd6f48f..9584f51 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -8,12 +8,12 @@ The `TUSK_DRIFT_MODE` environment variable controls how the SDK operates in your ### Available Modes -| Mode | Description | When to Use | -| ---------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------- | -| `RECORD` | Records traces for all instrumented operations | Set this in environments where you want to capture API traces (e.g., staging, production) | -| `REPLAY` | Replays previously recorded traces | Automatically set by the Tusk CLI when running `tusk run` - you should NOT set this manually | -| `DISABLED` | Disables all instrumentation and recording | Use when you want to completely disable Tusk with no performance impact | -| Unset | Same as `DISABLED` - no instrumentation or recording | Default state when the variable is not set | +| Mode | Description | When to Use | +| ---------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| `RECORD` | Records traces for all instrumented operations | Set this in environments where you want to capture API traces (e.g., staging, production) | +| `REPLAY` | Replays previously recorded traces | Automatically set by the Tusk CLI when running `tusk drift run` - you should NOT set this manually | +| `DISABLED` | Disables all instrumentation and recording | Use when you want to completely disable Tusk with no performance impact | +| Unset | Same as `DISABLED` - no instrumentation or recording | Default state when the variable is not set | ### Important Notes @@ -25,7 +25,7 @@ The `TUSK_DRIFT_MODE` environment variable controls how the SDK operates in your **Replaying Traces:** -- `TUSK_DRIFT_MODE` is automatically set to `REPLAY` by the Tusk CLI when you run `tusk run` +- `TUSK_DRIFT_MODE` is automatically set to `REPLAY` by the Tusk CLI when you run `tusk drift run` - **Do NOT** manually set `TUSK_DRIFT_MODE=REPLAY` in your application startup commands - The start command specified in your `.tusk/config.yaml` should NOT cause `TUSK_DRIFT_MODE` to be set to anything - the CLI handles this automatically @@ -61,7 +61,7 @@ start_command: "npm run dev" # Do NOT include TUSK_DRIFT_MODE here ```bash # The CLI automatically sets TUSK_DRIFT_MODE=REPLAY -tusk run +tusk drift run ``` **Disabling Tusk:** @@ -99,7 +99,7 @@ Your Tusk Drift API key, required when using Tusk Cloud for storing and managing - Can be set as an environment variable: ```bash - TUSK_API_KEY=your-api-key-here tusk run + TUSK_API_KEY=your-api-key-here tusk drift run ``` - Or use the Tusk CLI login command (recommended): diff --git a/docs/quickstart.md b/docs/quickstart.md index 4f51cc7..ac03a34 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -56,7 +56,7 @@ Need to install the Tusk CLI? See [CLI installation guide](https://github.com/Us Replay the recorded test: ```bash -tusk run +tusk drift run ``` You should see an output similar to: diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index ce906dd..4feb97c 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -21,7 +21,7 @@ Ensure that `TuskDrift.initialize()` is called before any other telemetry provid 1. **Enable service and CLI logs**: ```bash - tusk run --debug + tusk drift run --debug ``` Logs will be written to `.tusk/logs/` diff --git a/src/instrumentation/libraries/e2e-common/base-entrypoint.sh b/src/instrumentation/libraries/e2e-common/base-entrypoint.sh index 5fffc87..d4e0605 100644 --- a/src/instrumentation/libraries/e2e-common/base-entrypoint.sh +++ b/src/instrumentation/libraries/e2e-common/base-entrypoint.sh @@ -230,7 +230,7 @@ log "Phase 3: Running Tusk Tests" "$BLUE" log "================================================" "$BLUE" set +e -TEST_OUTPUT=$(TUSK_ANALYTICS_DISABLED=1 TUSK_REQUIRE_INBOUND_REPLAY_SPAN=1 tusk run --print --output-format json --enable-service-logs 2>&1) +TEST_OUTPUT=$(TUSK_ANALYTICS_DISABLED=1 TUSK_REQUIRE_INBOUND_REPLAY_SPAN=1 tusk drift run --print --output-format json --enable-service-logs 2>&1) TUSK_EXIT=$? set -e