[nightshift] test-flakiness: Time-dependent and env-mutation flaky patterns

[nightshift-micr]

Test Flakiness Analysis Report — micr-dev/tokens

Executive Summary

Metric	Value
Total test files	12
Test framework	Node.js built-in node:test
Overall flakiness risk	🟡 Medium
Files with flaky patterns	5 of 12
Critical flakiness (P0-P1)	3 findings

---

Flakiness Findings

🔴 P1 — High Risk

1. Time-dependent date generation in cli.test.ts (lines 23-46)

Pattern: recentIso() and recentDate() generate dates relative to new Date() at test runtime.

function recentIso(daysAgo = 0) {
  const date = new Date();
  date.setUTCDate(date.getUTCDate() - daysAgo);
  return date.toISOString();
}

Risk: Tests crossing midnight UTC (or local midnight for recentDate) may generate unexpected date values. recentDate uses local time while the CLI may use UTC — this mismatch causes date-boundary failures.

Fix: Use fixed ISO date strings instead of runtime-relative dates.

---

2. Environment variable mutation without test isolation (droid.test.ts, gemini.test.ts, helios.test.ts, hermes.test.ts, pi.test.ts)

Pattern: Multiple test files mutate process.env and restore in t.after():

const originalHome = process.env.HOME;
t.after(() => { process.env.HOME = originalHome; });
process.env.HOME = workspace;

Risk: If two tests run concurrently and both mutate process.env.HOME, one test's env mutation leaks into the other. The cleanup may not execute before the next test starts.

Severity: P1 — Works now with sequential execution, but any future parallelism will break.

Fix: Use unique env var names per test, or enforce --test-concurrency=1.

---

🟡 P2 — Medium Risk

3. cli.test.ts spawns child processes — non-deterministic timing (line 16)

Risk: CLI tests spawn actual child processes. Under high system load, tests asserting on CLI output may timeout or receive partial output.

Fix: Increase timeouts for CLI integration tests.

---

4. SQLite-backed tests skip on Bun (helios.test.ts line 9, hermes.test.ts line 9)

const skipSqliteBackedTest = Boolean(process.versions.bun);
const sqliteBackedTest = skipSqliteBackedTest ? test.skip : test;

Risk: Tests silently skip on Bun instead of failing. Masks real regressions.

Fix: Log a warning when SQLite tests are skipped.

---

5. droid.test.ts — Potential env leak between HOME and provider-specific env vars

Risk: If test runner ever runs tests concurrently, env mutations break.

Fix: Use provider-specific env vars explicitly.

---

🟢 P3 — Low Risk

6. File system temp directory collisions — stale slopmeter-* dirs from crashed runs

7. cli.test.ts fixture dates use recentIso() — not deterministic across days

8. t3-chat.test.ts — Provider merge order assertion is brittle but intentional

---

Files with No Flaky Patterns

File	Status
packages/cli/test/export.test.ts	✅ Clean
packages/cli/test/graph.test.ts	✅ Clean
packages/web/test/analytics.test.ts	✅ Clean
packages/web/test/usage.test.ts	✅ Clean
packages/web/test/publish-usage.test.ts	✅ Clean

---

Recommended Fixes (Prioritized)

1. P1-1: Replace recentIso()/recentDate() with fixed date constants
2. P1-2: Document or enforce serial test execution for env-mutating tests
3. P2-3: Add generous timeouts (30s+) for CLI spawn tests
4. P2-4: Log warning when SQLite tests are skipped on Bun
5. P3-6: Add periodic cleanup of stale temp dirs