Skip to content

Add backend E2E story tests: dashboard user stories over a fake agent - #1527

Merged
brillout merged 1 commit into
mainfrom
claude/backend-e2e-mock-tests-t7ci4f
Aug 10, 2026
Merged

Add backend E2E story tests: dashboard user stories over a fake agent#1527
brillout merged 1 commit into
mainfrom
claude/backend-e2e-mock-tests-t7ci4f

Conversation

@brillout

Copy link
Copy Markdown
Member

What

A backend-only E2E suite (packages/the-framework/src/e2e/) that walks every major dashboard user story through the real business logic — no browser, no real AI agent, and fast: the 14 stories add ~2s to the suite's wall clock (files parallelize; full suite stays ~45s).

Each story is end-to-end in the sense that matters:

  • Entry points are the dashboard's own RPCs (sendStart, sendChoice, onRuns, onQueue, …), with the Telefunc request context wired exactly as runDaemon wires it (runtime closures, registry-backed preferences/discord stores; stub quota/auto-PM reporters where the daemon holds live pollers).
  • Runs are real detached processes spawned by createProjectRuntime, exercising the production lifecycle — worktree allocation, run store, events.jsonl, control watcher, gates, teardown, retention. Only the agent turn is scripted: the harness binPath points at fake-agent-bin.ts, which forwards to the real CLI with --fake appended (FRAMEWORK_FAKE_AWAIT parks a run on a scripted gate).
  • State is real: temp git repos registered through sendAddProject, an isolated $XDG_CONFIG_HOME per test process, and (for the publish story) a real bare origin the push actually lands on.

Stories covered

File Stories
story-projects-and-settings Add/register a project + sidebar reads (#396); unknown-project degradation (#427); dashboard-written preferences reaching a resumed run's argv (#858/#1467); usage panel + auto-PM sweep (#533/#1210)
story-session-lifecycle Start → live feed → archived row; concurrent sessions in own worktrees (#736); publish a finished session's branch to origin (#799)
story-steering-and-gates Answer a parked question from the hub (#304/#1455); live chat becomes the next agent turn + committed conversation (#714/#908); rearm handoff mid-run (#1102); stop → retained worktree → remove → delete (#737/#1032)
story-tickets-and-queue Tickets list/detail/cross-project (#697/#1144); queue a ticket, drain run claims it, boards show "implementing" (#1164/#1117); unrelated prompts claim nothing (#1117)

The spawned argv is recorded per run (opt-in via $FRAMEWORK_E2E_ARGV_FILE), which is the only observable place the launcher's toggle→flag contract (startOptionFlags) can be asserted — the spawn is detached.

Product findings (documented in src/e2e/spec.md)

Writing the stories surfaced two real races in the finished-session seam. The stories model the same healing the dashboard relies on, but both are candidates for tightening:

  1. Acting on a session the moment its meta flips done races teardown. sendPushBranch/resume inside that window runs commitPendingWork against the checkout teardown is committing in; the loser reports "could not commit the work this session left uncommitted", and teardown then retains a worktree it would have removed. A user clicking Push right as a run finishes can hit this; a second click succeeds.
  2. A live event tail whose file is retired goes permanently silent. Teardown moves events.jsonl into the archive; when tailEvents' fs.watch misses the final appends (watcher pressure), the 1s poll finds the file already gone and the tail never delivers the run's end. The dashboard heals because the view swaps to the archived replay once the row settles — the harness's tailRun mirrors that swap.

Not in scope

The Telefunc transport hop (mount, CSRF/rebinding, Channel/SSE) — covered by dashboard/server.test.ts and dashboard-rpc/stream-channel.test.ts; telefunc Channels only pump over a real wire, so stream assertions tail the same events.jsonl the channel wraps.

Verification

  • Full suite: 1706 tests, 0 failures (twice).
  • Stress runs: lifecycle file 15×, other story files 8× each — all green after fixing two test-side races the findings above explain.
  • src/e2e is excluded from the publish build (tsconfig.build.json); nothing ships in dist/. Note the baseline requires the dashboard bundle built (pnpm build at root before pnpm test), same as CI's order.

Generated by Claude Code

Every major dashboard flow now has an end-to-end test on the backend
alone: the stories call the same telefunctions the browser invokes,
against the real daemon runtime (createProjectRuntime), real registry
state, and real spawned run processes — with the deterministic --fake
driver in the agent seat, so the whole suite runs offline and adds ~2s
of wall clock. No browser involved.

Covered stories: add/register a project and the sidebar reads; start a
session and follow its live feed to the archived row; two concurrent
sessions in their own worktrees (#736); publish a finished session's
branch to a real bare origin (#799); answer a parked question from the
questions hub (#304/#1455); live chat becoming the next agent turn and
its committed conversation (#714/#908); rearming the handoff mid-run
(#1102); stop -> retained worktree -> remove -> delete (#737/#1032);
tickets list/detail and cross-project pages (#697/#1144); queueing a
ticket and a drain run claiming it (#1164/#1117); dashboard-written
preferences reaching a resumed run's argv (#858/#1467); and the usage
panel + auto-PM sweep surface (#533/#1210).

The harness (src/e2e/harness.ts) wires the Telefunc request context the
way runDaemon does, isolates $XDG_CONFIG_HOME per test process, spawns
runs through fake-agent-bin.ts (the real CLI with --fake appended), and
records each spawn's argv — the only observable place the launcher's
toggle-to-flag contract can be asserted, since the spawn is detached.

Writing the stories surfaced two real races in the finished-session
seam (documented in src/e2e/spec.md): acting on a session the moment
its meta flips done collides with teardown's own commits, and a live
event tail whose file is retired misses the final events when the
fs.watch signal is lost. The stories model the same healing the
dashboard uses (wait for retirement; swap to the archived replay).

src/e2e is excluded from the publish build; it compiles only into
dist-test with the rest of the suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XKihCTfcazhDgt4dhUmfM9
@brillout
brillout merged commit 47b93e9 into main Aug 10, 2026
1 check passed
@brillout
brillout deleted the claude/backend-e2e-mock-tests-t7ci4f branch August 10, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants