test: close the test-vs-shipped-artifact gap - #1
Merged
Conversation
Every bug that actually shipped this project - a broken CLI entry point, an unreachable port-retry branch, dead change detection, a 404'd lazy chunk, a zero-height canvas, a store with exactly one issue collapsing to an object - passed a green suite and was only found by running the real, built, published artifact by hand. The suite tested imported modules; none of it touched what a user actually gets. This closes that gap along five axes, and none of it adds a runtime dependency. Asset-manifest consistency (test/assets.test.mjs): the build output and the server's static manifest must name exactly the same files. Needs no bd and no browser, so it always runs. Verified against the actual regression: renaming vite.config.ts's chunk output back to app2.js while leaving the manifest at elk.js fails this test immediately. Packaged-artifact smoke (test/package.test.mjs): npm pack the real tarball, install it into a project that has never seen this source tree, run the installed binary, hit every route. This is the manual check that found two of the bugs above, now automated. Also asserts the tarball ships zero runtime dependencies and no src/ or test/. Live-refresh delivery (added to security.test.mjs): a write must actually produce an `event: store-changed` SSE frame, and the frame's id must equal the store's new version. Verified this catches the dead-detection regression by reintroducing the wrong noms path and confirming the test fails. Boundary counts (test/boundary.test.mjs): 0, 1, and 2 issues, plus an epic with exactly one child, each in its own freshly created store rather than a shared one where ordering could hide the count that matters. Verified this catches the single-issue collapse by reverting listAll()'s fix and confirming the count=1 test fails. Render smoke (test/smoke.test.mjs, new playwright devDependency): the real built page, against a real seeded store, in headless Chromium. Asserts the canvas has real height and width (the exact zero-height regression), the node count matches the seeded issues, and there are zero console errors and zero failed requests. Screenshot saved as a CI artifact so a human can glance at every release rather than trusting only assertions about pixels. CI previously ran none of the integration or CLI suites at all - they existed, gated on `hasBd`, and silently skipped everywhere in CI because bd was never installed there. Fixed via `npm install -g @beads/bd` (the npm package, not the curl|sh script, to avoid running an arbitrary remote shell script in CI) on the newer runtime leg only, to avoid doubling browser/binary downloads across the Node 22/24 matrix for no extra coverage. The full battery also now runs directly inside release.yml, not only ci.yml: GitHub Actions has no cross-workflow `needs`, so without this a tag could publish even if the parallel ci.yml run failed on the very same tag. Verified: 45 tests pass (29 .mjs + 16 .ts), typecheck and lint clean, the render smoke test's screenshot confirms the collapse-toggle fix from the previous session still holds in a real browser.
…n test SKIPped The first CI run on this branch went green in 44s - far too fast to have installed Chromium and run the full battery. It had: newer npm blocks a dependency's lifecycle scripts by default and reports it as a warning, not a failure, so `npm install -g @beads/bd` exited 0 while never running the postinstall that fetches the actual bd binary. hasBd then evaluated false, and all nineteen integration and smoke tests reported SKIP rather than FAIL - a fully green job that had tested nothing beyond the pure-logic suite, which is precisely the failure mode this whole PR exists to close. Fixed with --allow-scripts=@beads/bd, the flag npm's own install --help documents for exactly this case on a global install (a project-scoped allowScripts entry in package.json is not available to `npm install -g`, which has no project to write one into). Verified against npm 12 in isolation: without the flag, `bd version` fails after install; with it, the real binary is present and functional. Also added an explicit `bd version` step immediately after install, so a future regression in this exact shape fails loudly on its own line instead of manifesting fifty lines later as a wall of quiet SKIPs that still exits 0.
The --allow-scripts fix let the postinstall run, but it still silently produced no
binary: read scripts/postinstall.js from the actual published tarball rather than
guessing further, and line 310 is `if (!process.env.CI) { ...download... } else {
console.log('Skipping binary download in CI environment'); }`. GitHub Actions sets
CI=true on every runner by default, so the script deliberately no-ops - which is
@beads/bd being conservative by design, not a defect in it.
Confirmed directly: `CI=true npm install -g @beads/bd` installs only bd.js in 212ms;
`env -u CI npm install -g @beads/bd` installs the real binary in ~6s. Fixed with a
step-level `env: { CI: "" }` on the Install bd step only - every later step keeps CI=true
as GitHub sets it, which is correct, since only the download itself needed to see it
unset.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every bug that shipped in this project's history — broken CLI entry point, unreachable port-retry, dead change detection, a 404'd lazy chunk, a zero-height canvas, a single-issue collapse — passed a green suite and was only found by running the real artifact by hand. This closes that gap.
Additions
test/assets.test.mjs— the build output and the server's static manifest must name exactly the same files. Nobd, no browser, always runs.test/package.test.mjs—npm packthe real tarball, install it fresh, run the installed binary, hit every route. Also asserts zero runtime dependencies and nosrc/testin the tarball.security.test.mjs) — a write must actually produce astore-changedSSE frame with the correct version id.test/boundary.test.mjs— 0, 1, 2 issues, and an epic with one child, each in its own fresh store.test/smoke.test.mjs(newplaywrightdevDependency) — the real built page in headless Chromium: non-zero canvas size, correct node count, zero console errors, zero failed requests. Screenshot uploaded as a CI artifact.bd(npm install -g @beads/bd) and Chromium, on one leg of the Node 22/24 matrix, so the integration and CLI suites — which existed but silently skipped everywhere in CI — actually run.release.ymlruns the full battery too, not onlyci.yml: GitHub Actions has no cross-workflowneeds, so without this a tag could publish even if the parallelci.ymlrun failed on the same tag.Evidence, including that each test catches what it claims to
.mjs+ 16.ts), typecheck and lint clean.app2.jswhile leaving the manifest atelk.jsfails it immediately.watch.mjsto the original wrongnomspath fails it.listAll()'s array-normalization fails the count=1 case specifically.−/+Ncontrol renders beside the priority without clipping the title.Version
Bumped to 0.3.0 in this branch; tag and publish happen after merge.