test: boot & OTA integration test (dummy server + QEMU harness) + release gate#38
Merged
Conversation
Cross-build A/B OTA integration test in QEMU that would have caught both shipped OTA-safety bugs (#36 fs-label, #37 ESP by-UUID): a dummy OTA server speaking the agent's real protocol + a serial-first pexpect harness. Success requires BOTH the /etc/issue banner+login over serial AND an agent check-in/ commit at the expected version. T2 boots the last release as slot A and OTAs the new build into slot B (genuinely cross-build → the only condition under which either bug appears). Plus L0 build-time guard failing on any fstab device-UUID mount. Triggers: L0 every PR, full T1/T2 on boot-critical PRs + as a release publish-gate. No KVM → TCG, condition-based waits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- release via workflow_dispatch + self-computed next version (mirrors FW), gate publish on boot-ota-test; retire tag-push trigger. Reusable build.yml + boot-ota-test.yml + compute-version composite action. - L0 split: L0a static UUID-pattern lint every PR; L0b authoritative ROOTFS_POSTPROCESS assertion during the build. - Dummy stays standalone (agent verifies only sha256 on the bz2 rootfs + signs requests ed25519 → no station-manager code reuse needed); drift guarded by a station-manager contract test. - Runner is a parameter; GH-hosted vs Hetzner decided by a feasibility spike (first impl step). No KVM either way → TCG; Hetzner = faster TCG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Promote the harness "transport" to a full Target interface (flash/seed_config/power/console/enter_flash_mode/dut_server_url/ boot_markers) so CM4 is a drop-in, not a rewrite. QemuTarget now, Cm4Target future. - Document the CM4 bench: rpiboot flash (carrier exposes USB-OTG + nRPIBOOT, confirmed), UART console (header confirmed), power control left OPEN. Closes the RPi release-gate gap and covers u-boot + real eMMC + real power-cycle rollback. - Add release-rework caveats: dry_run the dispatch/auto-version first; releases get slower under TCG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Prod uses eMMC, dev bench uses cheaper CM4 Lite (SD). Small medium gap (PARTLABEL-based path → both enumerate mmcblk0); documented, not covered. Flash backend cost-tiered: mux-less reset-over-serial first (€0; the OTA writes slot B over the network so the flasher is out of that loop), then self-fabbed SDWire (~€10-20 BOM, open hw) before commercial ~€90-120. flash() abstraction lets an eMMC/rpiboot target plug in later unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…green) Standalone stdlib dummy server mirroring the agent's real protocol (heartbeat/check/download/status/commit), rootfs-extraction + config-seed helpers, and a tuned-timer config renderer. 10 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…green) - Target ABC + QemuTarget (qemu/OVMF/pty/seed/grubenv-reset) + Cm4Target stub - L0a static lint: flags wks --use-uuid mounts without --no-fstab-update - compute-release-version.sh: next YYYY.MM.DD-HH[a-z], suffix bump on collision 22 unit tests pass; shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…low-up) - L0b: ROOTFS_POSTPROCESS assert_no_uuid_fstab fails the build on any device UUID mount in /etc/fstab (authoritative #37 guard). - T1 boot smoke + T2 cross-build OTA cycle (qemu-marked, run in CI). T3 rollback shipped skipped (spec follow-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ci.yml: L0a UUID lint + ota-integration unit tests every PR; shellcheck scripts/ - boot-ota-test.yml: reusable, runner-parametrized (GH-hosted default, TCG) - boot-ota-pr.yml: path-filtered boot-critical PRs -> build + T1 - boot-ota-spike.yml: TCG boot feasibility spike (GH vs Hetzner) - release.yml: workflow_dispatch + compute-version + boot/OTA publish gate; action-gh-release creates the tag; dry_run skips sign/tag/publish - compute-version composite action + script; release.sh -> gh dispatcher Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a QEMU-based boot + cross-build OTA integration test harness (dummy OTA server + Target abstraction) and wires it into CI/release so x86 releases are gated on actually booting and completing an OTA cycle, plus introduces fast/static (L0a) and build-time (L0b) guards against UUID-based /etc/fstab mounts (the #37 class of OTA failures).
Changes:
- Introduce
tests/ota-integration/harness: dummy OTA server, image extraction/seeding helpers, QEMU target, and T1/T2 (with T3 scaffolded + skipped). - Add L0a UUID-mount lint script + unit tests, and L0b Yocto ROOTFS postprocess assertion to fail builds with UUID mounts in
/etc/fstab. - Rework release process to
workflow_dispatchwith computed version + boot/OTA gate; add reusable workflows for boot/OTA testing and PR path-filtered smoke runs.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/ota-integration/test_target.py | Unit tests for Target shapes/markers and CM4 stub behavior. |
| tests/ota-integration/test_rollback.py | Skipped (documented) T3 rollback follow-up scaffold. |
| tests/ota-integration/test_ota_cycle.py | T2 cross-build OTA cycle test (old release → new build, assert commit). |
| tests/ota-integration/test_l0a_lint.py | Unit tests for the L0a static UUID-mount lint script. |
| tests/ota-integration/test_image_ops.py | Unit tests for checksum/compress/decompress and seed config rendering/keygen. |
| tests/ota-integration/test_dummy_server.py | Unit tests validating dummy OTA server endpoints/recording. |
| tests/ota-integration/test_compute_version.py | Unit tests for release-version computation script. |
| tests/ota-integration/test_boot.py | T1 boot smoke test: banner/login + agent heartbeat at expected version. |
| tests/ota-integration/target.py | Target ABC + QemuTarget implementation + Cm4Target stub. |
| tests/ota-integration/seed.py | Data-partition seeding utilities (config + Ed25519 key). |
| tests/ota-integration/requirements.txt | Python deps for the integration/unit tests. |
| tests/ota-integration/README.md | Local + CI usage docs and rationale/self-verification notes. |
| tests/ota-integration/pytest.ini | Defines unit and qemu markers and default pytest opts. |
| tests/ota-integration/image_ops.py | WIC decompress + rootfs partition extraction (bz2 + sha256) + loop attach helper. |
| tests/ota-integration/helpers.py | wait_until polling helper (avoid fixed sleeps). |
| tests/ota-integration/dummy_server.py | Standalone dummy OTA server (stdlib) implementing agent protocol surface. |
| tests/ota-integration/conftest.py | Fixtures for artifacts, qemu target lifecycle, and dummy server factory. |
| scripts/release.sh | Convert release helper into a thin gh workflow run dispatcher. |
| scripts/l0a-fstab-uuid-lint.sh | L0a static guard: flag --use-uuid fstab injection risks + recipe UUID fstab writes. |
| scripts/compute-release-version.sh | Compute next YYYY.MM.DD-HH[a-z] version based on existing tags. |
| meta-oe5xrx-remotestation/recipes-core/images/oe5xrx-remotestation-image.bb | Add L0b /etc/fstab UUID-mount assertion in ROOTFS postprocess. |
| docs/superpowers/specs/2026-07-11-boot-ota-integration-test-design.md | Design/spec for harness, gates, and rationale. |
| docs/superpowers/plans/2026-07-11-boot-ota-integration-test.md | Detailed implementation plan and protocol reference. |
| .github/workflows/release.yml | Rework to workflow_dispatch, compute version, gate publish on boot/OTA tests. |
| .github/workflows/ci.yml | Run L0a on every PR + add OTA-integration unit tests. |
| .github/workflows/boot-ota-test.yml | Reusable workflow to run QEMU boot/OTA tests (runner parametrized). |
| .github/workflows/boot-ota-spike.yml | Feasibility spike workflow to measure TCG boot timing/capacity. |
| .github/workflows/boot-ota-pr.yml | Path-filtered PR workflow: build x86 + run T1 boot smoke. |
| .github/actions/compute-version/action.yml | Composite action wrapping version computation script. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Extending shellcheck to all of scripts/ surfaced pre-existing intentional
warnings in run-qemu.sh (word-splitting ${CPU_FLAGS} = SC2086). Lint the
three new scripts explicitly instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The L0a step runs after 'kas dump', which clones upstream layers (poky, meta-*) into the workspace; those legitimately use --use-uuid and produced 20+ false positives. Scope the scan to git-tracked files (git ls-files), with a find fallback for non-git trees (unit-test temp dirs). Reproduced the pollution locally and verified the fix ignores untracked upstream wks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pass the latest published release as slot A so boot-critical PRs get real cross-build OTA coverage (not just the release gate). Bootstrap-safe: no release yet -> T2 skips, T1 only. Matches the spec's PR-path cadence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Draft PRs no longer trigger the Hetzner build + QEMU test, so Copilot-loop / WIP pushes on a draft don't re-run everything. types adds ready_for_review so marking the PR ready fires the run; each job is if: !draft. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- decompress_wic handles .wic.xz / .wic.gz (not just .bz2), matching the compression set release.yml may publish - boot-ota-test / boot-ota-spike: download slot A / release across the same extension set (bz2|xz|gz|wic) and pick the first present via nullglob - gh release list: --exclude-drafts --exclude-pre-releases so slot A is always a published release (boot-ota-pr, release, spike) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- boot-ota-test: build-under-test wic discovery also accepts .wic.xz/.wic.gz - boot-ota-test: fail fast if last_release_tag is set but no slot-A asset downloaded (prevents a silently-empty output making T2 skip = false green) - dummy_server: return 400 on a malformed JSON body instead of silently treating it as empty (surfaces protocol drift) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Assert the serial banner version by EXACT match in T1 and T2 (banner_re captures the tag token). A bidirectional substring check could false-green e.g. 2026.07.11-15 vs 2026.07.11-15a, weakening the gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- compute-release-version.sh: arity guards for --now / --existing (set -u) - l0a lint: recipe scan also catches /dev/disk/by-uuid/ fstab entries, not just UUID= (parity with L0b) - boot-ota-test / boot-ota-spike: install grub-common alongside grub2-common so grub-editenv is present across Ubuntu runner versions - tests for both new guards Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
.github/workflows/release.yml:186
actions/checkout@v6appears to reference a non-existent major version for the official checkout action, which would prevent the publish job from running. Use an existing supported major version (e.g.,actions/checkout@v7).
steps:
- uses: actions/checkout@v6
- dummy_server: stream the download payload (shutil.copyfileobj) instead of reading the whole rootfs into memory; accurate Content-Length - dummy_server: merge the path-derived result_pk LAST so a client body can't override it (authoritative pk); added a test Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… ANSI The T1/T2 gate failed with banner 'dev\x1b' != 'dev': getty emits ANSI escapes on the console and (\S+) greedily swallowed the trailing ESC byte. The exact-match assertion (round 4) then correctly rejected it. Restrict the capture to the tag charset [A-Za-z0-9._-]+ (what stamp_release enforces) so it stops before the escape. Added a unit test reproducing the ANSI-suffixed line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment on lines
+29
to
+36
| # Collect our own wks + recipe files (git-tracked only, else find). | ||
| if git -C "$root" rev-parse --is-inside-work-tree >/dev/null 2>&1; then | ||
| mapfile -t wks_files < <(git -C "$root" ls-files -- '*.wks' '*.wks.in' | sed "s|^|${root%/}/|") | ||
| mapfile -t recipe_files < <(git -C "$root" ls-files -- '*.bb' '*.bbappend' | sed "s|^|${root%/}/|") | ||
| else | ||
| mapfile -t wks_files < <(find "$root" \( -name '*.wks' -o -name '*.wks.in' \)) | ||
| mapfile -t recipe_files < <(find "$root" \( -name '*.bb' -o -name '*.bbappend' \)) | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two OTA-safety bugs shipped in releases and only bit at runtime on a real cross-build A/B OTA: #36 (slot B never relabelled
root_b→ GRUBsearch --labelfailed → unbootable) and #37 (ESP mounted by FAT UUID → OTA'd slot mismatched the on-disk ESP →/boottimeout → emergency mode → rollback). No existing check booted the image or performed an OTA, so nothing caught them. This adds exactly that test — plus cheap guards and a release gate.Design + plan (committed on the branch):
docs/superpowers/specs/2026-07-11-boot-ota-integration-test-design.mddocs/superpowers/plans/2026-07-11-boot-ota-integration-test.mdWhat
Dummy OTA server (
tests/ota-integration/dummy_server.py) — stdlib, speaks the agent's real protocol (heartbeat / check / download / status / commit, verified againststation-manager/station_agent+apps/deployments), records every interaction. The agent runs its complete real OTA code path against it. Standalone by design: the agent verifies only a sha256 on the bz2 rootfs and signs requests with Ed25519 (which a test double needn't verify), so no station-manager dependency is needed.Targetabstraction (target.py) —QemuTargetnow;Cm4Targetis a stub so the CM4 HIL bench is a drop-in, not a rewrite (flash/seed/power/console/dut_server_url/boot_markers).Tests:
OE5XRX Remote Station <tag>+login:over serial AND agent check-in at the expected version.Guards: L0a static lint (
scripts/l0a-fstab-uuid-lint.sh, every PR) flags any wks--use-uuidmount without--no-fstab-update; L0bROOTFS_POSTPROCESSassertion fails the build on any device-UUID/etc/fstabmount.Pipeline:
ci.yml— L0a +unittests every PR (seconds).boot-ota-pr.yml— path-filtered boot-critical PRs → build x86 + T1.boot-ota-test.yml— reusable, runner-parametrized (GH-hostedubuntu-latestdefault, TCG; flip to Hetzner viarunner_label).release.yml— reworked to one-buttonworkflow_dispatchthat computes its own version (YYYY.MM.DD-HH[a-z], composite action) and gates publish on T1+T2;action-gh-releasecreates the tag;dry_runskips sign/tag/publish.scripts/release.sh→ thingh workflow rundispatcher.boot-ota-spike.yml— TCG boot feasibility spike (decides GH-hosted vs Hetzner).Verification
yamllint+shellcheckclean. Theqemutests skip without an image + root.boot-ota-pr.ymland the release gate. Open feasibility item: whether GH-hosted TCG is fast/roomy enough — runboot-ota-spike.ymlto decide; if not, fliprunner_label.Notes / deferred
Please review — do not merge on my behalf.
🤖 Generated with Claude Code