chore: add dev-stories — platform/infra acceptance tests (open-PR previews reachable) - #81
Open
samo-agent wants to merge 11 commits into
Open
chore: add dev-stories — platform/infra acceptance tests (open-PR previews reachable)#81samo-agent wants to merge 11 commits into
samo-agent wants to merge 11 commits into
Conversation
… checks open-PR previews are live Adds dev-stories/, a lightweight test harness that verifies the running surface (not a status log) satisfies platform acceptance criteria. The first story asserts every open-PR preview returns HTTP 200, env=preview, and the correct branch from /api/version. Includes the bash runner, systemd unit references (installed on the control plane, firing every 10 min), and the motivation behind the concept (trigger mis-reporting "failed" while previews were healthy, 2026-06-22). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…temd timers Makes all five read-only dev stories self-running on the samohost control plane — no human or interactive session triggers them. Previously only previews-reachable was automated; prod-app-up, deploy-freshness, preview-comment-current and demo-envs-reachable ran on demand. For each of the 4: - ships the markdown spec + the runner (byte-identical to the stable ~/bin/dev-story-<name>.sh the timer executes; automation never depends on a repo checkout) - ships reference systemd units under automation/: a Type=oneshot .service (User=testuser, Environment=HOME=/home/testuser) + a .timer (OnBootSec=3min, OnUnitActiveSec=15min, AccuracySec=30s) A failing story leaves its .service in `failed` state (surfaced by `systemctl --failed`) with detail in the journal — the passive surfacing signal. Runners are read-only (curl / gh / jq / grep, mutate nothing) and hold no secrets: the gh CLI resolves its own auth at runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the PR-preview lifecycle fixture (dev-stories/functional/fixture-lifecycle.sh) that drives the throwaway samo-agent/samohost-fixture app through provision -> host-prep -> create -> scale -> redeploy -> teardown on a real Hetzner VM, plus FINDINGS.md cataloguing every fixture-side fix and every platform gap (P1-P8) the run surfaced. Harness fixes captured here: - F1: teardown now reclaims BY Hetzner provider id (independent of the state record), unique per-attempt names, tracks every provisioned id, and self-tests reclaiming a live VM whose state record is already 'destroyed'. - F2: redeploy step now actually clones the PR branch into a throwaway dir, edits only the BG constant, commits + pushes, then polls the preview for the new color and short-SHA. Secrets (HCLOUD_TOKEN, CLOUDFLARE_SAMOCAT) are referenced as env vars / sourced at runtime from an operator secrets file; no token literals committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Latest fixture-lifecycle run is 4 pass / 4 fail / 0 expected-fail. Provision came up on attempt 1/3 (no provisioning flake); the single root-cause failure is a product-side ROOT host-prep step (Caddy/443 not serving on the fresh fixture VM) -> previews CF 522, which cascades to the create/scale/redeploy/teardown assertions. This is the P7 522 host-prep trap surfacing live, NOT a harness bug and NOT provisioning reliability. Documents why the nightly is deliberately NOT installed yet (cleanEnoughToSchedule = FALSE) and that the orphan watchdog timer remains the budget backstop until host-prep is fixed and the lifecycle goes green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…so the private-repo clone actually happens The fixture-lifecycle host-prep step ran `samohost app bootstrap` over ssh with NO GitHub token, so bootstrap §0 found none, SKIPPED the private-repo clone (§12), left /opt/samohost-fixture/app empty, and §13's clone self-check exited 1 -> host-prep FAIL -> no app -> previews 521/522. The samo-agent `gh auth token` CAN read the PRIVATE samo-agent/samohost-fixture repo, so the token must reach bootstrap. The task's `bash -s 3< <(gh auth token)` FD-3 example cannot work on this delivery path: `samohost ssh` spawns the inner ssh with Node stdio:"inherit" (only FD 0/1/2 inherited — FD 3 is NOT), and ssh forwards only stdin/stdout/stderr over its channel (no arbitrary-FD forwarding); stdin is already consumed by the piped `bash -s` script body. Verified empirically. So use bootstrap §0's PRE-PLACED-FILE channel (the documented fallback): before the bootstrap apply, stream `gh auth token` over ssh STDIN into a pre-created 600 /opt/samohost-fixture/.gh-token via `dd status=none` (token never in argv, never on local disk, never committed/versioned, never logged). Remote commands use only single-word args because parseSsh joins post-`--` words with spaces (a quoted multi-word `bash -c` arg would lose its quoting). §0 then finds the file present, §12 clones the private repo and persists a runtime credential helper reading the same file for later env-create/redeploy fetches — so the file is left in place for the rest of the lifecycle and only disappears when the EXIT teardown trap destroys the throwaway VM wholesale. Add a host-prep `host-prep-clone` sub-assertion (test -d /opt/<app>/app/.git over ssh) so a future token regression FAILs loudly with the real cause instead of an opaque downstream preview 521. Dry-run PLAN-echoes the whole mechanism. bash -n clean; DRYRUN exit 0; no token literal anywhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… flag field-record cutover OPEN Annotate FINDINGS with the merge SHAs for the lifecycle fixes this MVP run landed: - dishonest-state stamp (failed create no longer stamps lastDeployedSha) — FIXED, #86 4648d26 - DB-UNREACHABLE self-heal gap (probeClones now reclassifies dead clones; heal flag ungated from --pr-previews) — FIXED, #86 4648d26 (follows DBLab self-heal #79 b308f87) - no-rebuild-command — ADDRESSED, preview rebuild subcommand merged #85 9aa93fb - no-idle-teardown — ADDRESSED warn-only but PENDING MERGE in #87 (open); no SHA yet Add a clearly-flagged OPEN held decision: field-record VM runs an UNVERSIONED snowflake preview cron (preview-reconcile.sh/preview-teardown.sh, not in this repo) missing self-heal. The real fix is a CUTOVER onto the samohost-managed lifecycle; NOT done here (touches a client VM, needs platform-team / samjr-bootstrap coordination). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bug-chain + ci-none OPEN - RESOLVED-merged with SHAs: preview rebuild (#85), self-heal/dishonest-state + DB-unreachable (#86), atomic idle-autodestroy warn-only (#87, now merged), selectable preview DB backend + register-drops-dbBackend fix (#88). - Add P9: previewDbBackend forced dblab for no-DB apps + register --from-toml silently dropped dbBackend/previewDbBackend (both fixed by #88). - Add the full fresh-VM from-scratch fixture bug-chain (~10 sequential bugs) as a single high-value finding. - Add OPEN: CI-NONE SKIPS PREVIEW DEPLOY (trigger.ts ~437-443 / ~467) with both resolution options + the code-map contradiction to investigate; NOT fixed. - Keep OPEN: wake-on-demand, DB-backed faithful clone-death self-heal test (no-DB fixture can't exercise it live), field-record cutover (now with the three observed snowflake failure modes: no self-heal, port-race on rebuild, unversioned). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…a real 200 + dbBackend=none Two self-bugs in the fixture-lifecycle harness: 1. teardown() is the EXIT/INT/TERM trap but returned without exiting. A bash signal handler that merely returns RESUMES the script where the signal interrupted it — so on SIGTERM teardown destroyed the VM and then the lifecycle resumed and fired a stray `trigger run --pr-previews` against the just-destroyed VM. Add `exit "$rc"` at the end of teardown so a signal ends the run cleanly. It preserves the exit-code-so-far on the normal EXIT path and cannot recurse (the handler clears its own traps first). 2. Make "create-serve-200" mean the no-DB surface REALLY served. The no-forced-dblab / db-preflight grep is necessary but NOT sufficient (absence of a forced clone never proves the preview answered), so it is now only a loud precondition. The per-PR create assertion polls the live URL to a real HTTP 200 (env=preview + branch match) AND requires the created env record to resolve dbBackend='none' where checkable before passing as serve-200. bash -n clean; dry-run green (2 pass / 0 fail / 2 xfail / 8 skip). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t false-orphan The fixture harness piped `samohost provision ... --json 2>&1` into jq. On the degraded/timeout path samohost writes a plain-text error to STDERR *and* the record JSON to STDOUT; `2>&1` merged them, so jq choked on the leading non-JSON and blanked .providerId. That empty id tripped the "no provider id reported" branch and emitted a FALSE "ORPHAN VM — manual cleanup needed" line even though the provider id was present (and the VM was being tracked for by-id teardown). Capture provision STDOUT only for the jq parse; redirect STDERR to its own temp file (surfaced via a diagnostic note, then removed). .providerId now parses on both the ok and degraded paths, and the provision exit code is preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… failure The fixture harness captured the create trigger output into $trig_out and sent both the redeploy trigger and the in-poll re-trigger to /dev/null, so when a per-PR preview action failed (e.g. the 2nd preview never creating an env, or a redeploy that never re-stamps lastDeployedSha) the underlying trigger error was invisible in the run log. Add dump_trigger_output() and emit the captured trigger stdout+stderr ONLY on failure for the create pass and the redeploy (initial trigger + last poll re-trigger), guarded so each blob appears at most once and never on a passing action. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l the multi-preview-on-same-host bug; arm nightly The single-preview lifecycle now passes for real on a fresh VM (run-6: 10 pass / 3 fail / 1 xfail). Bring the PR-branch harness up to the proven run-6 assertions and gate the one known remaining bug as a tracked expected-fail so the layer can run self-driving nightly. Harness (dev-stories/functional/fixture-lifecycle.sh): - idle-death is now a HARD PASS gate (real idle reap: env + systemd unit + Caddy include torn down together; samohost #87), replacing the stale "no reaper" XFAIL; add the rebuild HARD PASS gate (`samohost preview rebuild` restores a no-DB preview to 200) and the self-heal N/A note (no dblab clone for a no-DB app). These are the run-6 (real-run) assertions, previously only in the untracked working copy. - Mark the 3 multi-preview-on-same-host assertions XFAIL via new xfail_bug()/ xpass() helpers + a per-PR cr_ok/cr_bad dispatcher: (a) create of the 2nd preview (green-bg), (b) scale (env count == open-PR count), (c) redeploy-reflects-new-sha. The first open PR (newest = blue-bg) is the single-preview PRIMARY and stays a HARD PASS gate; the 2nd+ (green-bg) is the multi-preview case. - A run is GREEN when only these 3 fail; it goes NON-GREEN (exit 1 -> the nightly systemd service enters 'failed' = alert) if any of them starts PASSING (XPASS: bug fixed -> un-xfail it) OR if any single-preview HARD gate regresses (FAIL). - trig_out is surfaced on every per-PR action != ok (dump_trigger_output) so the next real run captures the exact 2nd-create throw. - SUMMARY + exit accounting carry the XPASS counter; bash -n clean; dry-run smoke green. FINDINGS.md: single-preview lifecycle GREEN end-to-end + nightly ARMED (original goal achieved); the ~17 merged fixes #83-#101 = the cold-start bug-chain RESOLVED (incl. #94 ci-none); the ONE open tracked bug = multi-preview-on-same-host (2nd env-create throws before the env upsert at src/commands/env.ts ~665-674/757 so no env persists; redeploy never re-stamps lastDeployedSha at src/preview/pr.ts:213) — marked xfail, fix pending; wake-on-demand + field-record cutover stay OPEN. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dev-stories/— a new lightweight test harness concept: acceptance tests for the platform/infra layer, verified against the live running surface, not internal state or logs.env=preview, and the correct branch from/api/version.check-previews.sh), its spec (previews-reachable.md), and reference copies of the systemd units that run it every 10 minutes on the control plane.Motivation
On 2026-06-22 the samohost trigger logged
action: "failed"for every open-PR preview. That log was initially treated as evidence that previews were down — they were not (each URL served HTTP 200). This harness makes the distinction a runnable test: the ground truth is the URL response, not the trigger log.Files added
dev-stories/README.mddev-stories/previews-reachable.mddev-stories/check-previews.shdev-stories/automation/dev-story-previews.servicedev-stories/automation/dev-story-previews.timerdev-stories/automation/README.mdWhat this does NOT touch
src/,scripts/,test/, or any existing file.Test plan
bash -n dev-stories/check-previews.shexits 0 (syntax clean — verified locally before commit)dev-stories/previews-reachable.mdlink indev-stories/README.mdresolves correctly in GitHub UIautomation/dev-story-previews.timerlink indev-stories/README.mdresolves correctly in GitHub UIgh auth tokenat runtime🤖 Generated with Claude Code