Skip to content

Add hermetic ACME e2e harness with Pebble, fixing two provisioning bugs - #127

Merged
Shine-neko merged 1 commit into
mainfrom
feat/acme-e2e-pebble
Jul 20, 2026
Merged

Add hermetic ACME e2e harness with Pebble, fixing two provisioning bugs#127
Shine-neko merged 1 commit into
mainfrom
feat/acme-e2e-pebble

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

Adds the first end-to-end ACME test to Sōzune: Pebble (Let's Encrypt's test CA) issues a real certificate over a real HTTP-01 challenge, hermetically — everything in Docker, no public ACME, no host port 80. Building it surfaced a provisioning bug, and reviewing the fix surfaced a second; both are fixed here.

Why this exists

HTTP-01 and DNS-01 have run in production with no e2e coverage since they shipped. This lays down a Pebble-based harness that exercises a full issuance cycle, and is structured to extend to DNS-01 and (later) TLS-ALPN-01.

The harness

tests/e2e/acme/ — a self-contained Docker compose (Pebble + a backend + Sōzune on one network) and run-acme.sh. Not wired into run-all.sh: it pulls an image and builds Sōzune, so it's run on demand. It asserts Sōzune provisions a cert for the routed host and serves the Pebble-issued cert on 443.

A few integration details worth calling out:

  • Sōzune trusts Pebble's directory root via a new SOZUNE_ACME_TEST_CA env var (see below). The root that matters is Pebble's minica (which signs its HTTPS directory endpoint), not the ACME issuance root at /roots/0 — a distinction that cost a while to pin down.
  • Pebble validates on ports 80/443 (config), and resolves the challenged host to the Sōzune container by Docker network alias, so no external DNS is involved.

The test hook

SOZUNE_ACME_TEST_CA, read only in account_builder(). When set, ACME calls go through instant-acme's builder_with_root so a test PKI is trusted; unset (production) the path is byte-for-byte the old Account::builder(). Because builder_with_root replaces the whole trust store, the variable is refused against a public Let's Encrypt directory — a leaked test env var must never weaken trust for a real CA.

Bug 1 — a transient failure stranded a cert for 12h

Found by the harness. The per-hostname backoff was computed (60s, 5min, 15min, …) but never consulted between cycles: the renewal loop only woke every 12h or on a new entrypoint. So a first-attempt failure — e.g. the challenge route not yet in place at cold start — waited out the full 12h before its 60s retry ever fired. Fixed by waking the loop at the soonest pending backoff.

Bug 2 — the fix for bug 1 could spin at 100% CPU

Found in review. Waking on the soonest backoff means a due entry returns sleep(0). If that entry's hostname no longer needs a cert, no success/failure is recorded, the entry lives on, and the loop busy-spins. The nastiest trigger: save_certificate succeeds but cert_tx.send fails — the attempt records a failure, yet the cert is now on disk so needs_certificate turns false forever. Fixed by clearing backoff when a hostname no longer needs a cert, and dropping entries for hostnames removed from config.

Both behaviours are covered by unit tests (soonest_retry_reports_the_nearest_pending_backoff, retain_hostnames_drops_entries_no_longer_in_config).

Not committed

tests/e2e/acme/pebble-ca.pem is fetched at runtime (minica mints a fresh root per image build) and gitignored.

@Shine-neko
Shine-neko merged commit 5827a56 into main Jul 20, 2026
3 checks passed
@Shine-neko
Shine-neko deleted the feat/acme-e2e-pebble branch July 20, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant