Add hermetic ACME e2e harness with Pebble, fixing two provisioning bugs - #127
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the first end-to-end ACME test to Sōzune: Pebble (Let's Encrypt's test CA) issues a real certificate over a real HTTP-01 challenge, hermetically — everything in Docker, no public ACME, no host port 80. Building it surfaced a provisioning bug, and reviewing the fix surfaced a second; both are fixed here.
Why this exists
HTTP-01 and DNS-01 have run in production with no e2e coverage since they shipped. This lays down a Pebble-based harness that exercises a full issuance cycle, and is structured to extend to DNS-01 and (later) TLS-ALPN-01.
The harness
tests/e2e/acme/— a self-contained Docker compose (Pebble + a backend + Sōzune on one network) andrun-acme.sh. Not wired intorun-all.sh: it pulls an image and builds Sōzune, so it's run on demand. It asserts Sōzune provisions a cert for the routed host and serves the Pebble-issued cert on 443.A few integration details worth calling out:
SOZUNE_ACME_TEST_CAenv var (see below). The root that matters is Pebble's minica (which signs its HTTPS directory endpoint), not the ACME issuance root at/roots/0— a distinction that cost a while to pin down.The test hook
SOZUNE_ACME_TEST_CA, read only inaccount_builder(). When set, ACME calls go through instant-acme'sbuilder_with_rootso a test PKI is trusted; unset (production) the path is byte-for-byte the oldAccount::builder(). Becausebuilder_with_rootreplaces the whole trust store, the variable is refused against a public Let's Encrypt directory — a leaked test env var must never weaken trust for a real CA.Bug 1 — a transient failure stranded a cert for 12h
Found by the harness. The per-hostname backoff was computed (60s, 5min, 15min, …) but never consulted between cycles: the renewal loop only woke every 12h or on a new entrypoint. So a first-attempt failure — e.g. the challenge route not yet in place at cold start — waited out the full 12h before its 60s retry ever fired. Fixed by waking the loop at the soonest pending backoff.
Bug 2 — the fix for bug 1 could spin at 100% CPU
Found in review. Waking on the soonest backoff means a due entry returns
sleep(0). If that entry's hostname no longer needs a cert, no success/failure is recorded, the entry lives on, and the loop busy-spins. The nastiest trigger:save_certificatesucceeds butcert_tx.sendfails — the attempt records a failure, yet the cert is now on disk soneeds_certificateturns false forever. Fixed by clearing backoff when a hostname no longer needs a cert, and dropping entries for hostnames removed from config.Both behaviours are covered by unit tests (
soonest_retry_reports_the_nearest_pending_backoff,retain_hostnames_drops_entries_no_longer_in_config).Not committed
tests/e2e/acme/pebble-ca.pemis fetched at runtime (minica mints a fresh root per image build) and gitignored.