Skip to content

fix(selfhost): stop the documented setup path producing an unbootable container (#9487 follow-up) - #9789

Merged
JSONbored merged 1 commit into
mainfrom
fix/empty-optional-secrets
Jul 29, 2026
Merged

fix(selfhost): stop the documented setup path producing an unbootable container (#9487 follow-up)#9789
JSONbored merged 1 commit into
mainfrom
fix/empty-optional-secrets

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

The bug

Two correct-in-isolation behaviours combine into a broken onboarding path:

  1. scripts/selfhost-init-secrets.sh deliberately creates zero-byte placeholders for the four secrets it cannot generate, because they come from an external party. secrets/README.md says so outright: it "creates only an empty placeholder for the four externally-issued ones."
  2. Compose refuses to start the stack unless each secret file exists, so operators cannot simply delete them.
  3. ops: two misfiring alerts, a blind spot on visual capture, a 44% false-positive gate, and three hardening gaps (from live edge-nl-01 inspection) #9487 then made an empty secret file fatal at boot.

Net effect: follow the documented setup, start the container, crash-loop. Every new self-hoster who isn't using all four capabilities hits this.

Observed live on the ORB upgrading to orb-v3.6.0-beta.1 — an unused GitHub App private key (a 0-byte placeholder, no App configured, loopover-orb handles auth here) took the whole instance down:

{"level":"error","event":"selfhost_secret_file_empty","var":"GITHUB_APP_PRIVATE_KEY_FILE"}
Error: Secret file for GITHUB_APP_PRIVATE_KEY_FILE (/run/secrets/github_app_private_key) is empty

The fix

For those four variables only, an empty file means "not configured": skip it, leave the target var genuinely unset, and log a named selfhost_secret_file_empty_optional warning.

This is not a revert to the pre-#9487 behaviour it superficially resembles. That set the target to an empty string, which every downstream nonBlank() read as unconfigured with no signal whatsoever. Here the var is left truly unset and the operator gets a warning naming it at boot.

Every other secret keeps #9487's fail-closed behaviour exactly. The init script writes a real random value for those, so an empty file there can only mean a truncated write — which is precisely the bug #9487 was written to catch (a truncated GITHUB_WEBHOOK_SECRET booting an instance that silently rejected every webhook). An invariant test pins that.

An unreadable file stays fatal for all four too: missing is not the same as deliberately empty.

Why not the alternatives

  • Don't default the env var in compose${VAR:-default} substitutes on empty as well as unset, so it cannot be cleared from .env; and dropping the default breaks everyone who does use the capability.
  • Make the operator delete the placeholder — compose won't start without the file.
  • Keep it fatal and document it — the failure is a crash-loop at boot with no prior warning, on the very first start after following our own instructions.

Tests

7 new cases, including a regression that reproduces the exact ORB crash (empty App key + real webhook secret → boots, App key unset, webhook secret loaded) and the invariant that a self-generated secret's empty file still throws. 26 pass.

docs:drift-check and selfhost:env-reference:check green; secrets/README.md documents the split.

… container

scripts/selfhost-init-secrets.sh deliberately creates ZERO-BYTE placeholders for
the four externally-issued secrets it cannot generate (github_app_private_key.pem,
orb_enrollment_secret.txt, pagerduty_routing_key.txt, claude_code_oauth_token.txt),
and compose refuses to start unless those files exist. #9487 then made an empty
secret file fatal at boot.

Together those mean: run the documented setup, start the container, crash-loop.
Observed live on the ORB, where an unused GitHub App key took the instance down on
upgrade to orb-v3.6.0-beta.1.

For those four ONLY, an empty file now means "not configured": skip it, leave the
target var genuinely unset, and log a named warning. Strictly better than the
pre-#9487 behavior it resembles -- that set an empty string every nonBlank() read
as unconfigured with no signal at all.

Every other secret keeps #9487's fail-closed behavior exactly. An empty file there
can only be a truncated write, which is the bug it was written to catch.
@loopover-orb

loopover-orb Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Important

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏳ LoopOver is waiting…

LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.48%. Comparing base (0fa52ed) to head (4f45c4b).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9789      +/-   ##
==========================================
- Coverage   90.30%   89.48%   -0.82%     
==========================================
  Files         913      913              
  Lines      113595   113599       +4     
  Branches    26963    26964       +1     
==========================================
- Hits       102584   101658     -926     
- Misses       9682    10851    +1169     
+ Partials     1329     1090     -239     
Flag Coverage Δ
backend 94.09% <100.00%> (-1.48%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/selfhost/load-file-secrets.ts 100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes

@JSONbored
JSONbored merged commit c368d7e into main Jul 29, 2026
7 checks passed
@JSONbored
JSONbored deleted the fix/empty-optional-secrets branch July 29, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant