ssh-per-host-access-blocks-lab — OpenSSH per-host user access blocks

A self-contained, reproducible lab that proves the PKI Manager's per-host user access blocks (decision-016): you can block one user on one host while she keeps access to every other host, and restore her by unblocking — with no certificate re-issue, no password, and no manual file copying. Enforcement is a composed, Host-CA-signed, ECIES-encrypted per-host KRL that each host pulls itself with krl-client.

Companion to pki-manager-web — this lab is a black-box integration harness for that project's SSH Certificate Manager (the "mother project"). It drives a live PKI Manager (pki.joor.net) and exercises the per-host-access-block feature end-to-end against real sshd. The design it validates lives in pki-manager's decision-016 and the krl-client host puller ships from that repo.

                         pki.joor.net   (the PKI — CAs, cert issuance, blocks)
                          │  operator REST (admin token): create-ca, register, issue, block/unblock
                          │  POST /krl   (no auth): ECIES-encrypted composed per-host KRL
   ┌──────────────────────┼───────────────────────────────────┐
   │                      │                                    │
┌──┴──────────┐   cert   ┌┴─────────────────┐        ┌─────────┴────────┐
│ sshclient   │─────────▶│ web1.acme.example│         │ web2.acme.example│
│ alice, jane │  OR      │  BLOCK TARGET    │         │  CONTROL         │
│ break-glass │  raw key │ sshd + krl-client│         │ sshd + krl-client│
└─────────────┘          └──────────────────┘        └──────────────────┘
     block alice on web1  ─▶  alice denied on web1, still OK on web2, jane unaffected

Two programs drive everything, with a strict split of duties:

Program	Role	Talks to
pki_client.py	the operator — creates CAs, registers hosts, issues certs, blocks/unblocks. All REST (/api/v1/ssh/*).	pki.joor.net (admin OIDC token)
krl-client	the host puller — its only job: POST /krl → local ECIES decrypt with the host's ecdsa key → verify the Host-CA signature → atomically install /etc/ssh/revoked_keys.	pki.joor.net (no token)

About the outputs: every block below is captured from a real run against the live containers and pki.joor.net. Serials, fingerprints and timestamps differ per run — the shape and the behaviour are what to match.

---

Contents

1. What this proves
2. Concepts in 60 seconds
3. Prerequisites
4. Quickstart
5. Walkthrough with real output
6. How a host pull works (krl-client)
7. Break-glass — the backdoor that survives a block
8. The two sshd drop-ins
9. pki_client.py + krl-client + krl-decode.py reference
10. Cleanup
11. Notes, gotchas, troubleshooting
12. Meta — mother project, CI, versioning

---

1. What this proves

provision.sh runs three phases and asserts a login matrix at each. The point is the selectivity of a block:

Phase	Operator action	alice→web1	alice→web2	jane→web1	break-glass→web1
A baseline	(none)	✅	✅	✅	✅
B block	block(alice, web1)	⛔	✅	✅	✅
C unblock	unblock(alice, web1)	✅	✅	✅	✅

• B: alice denied on web1 but OK on web2 — "loses this host, keeps everywhere else." Her certificate stays active; only web1's KRL denies it.
• B: jane OK on web1 — the block is per-identity, not host-wide.
• B: break-glass OK — a raw authorized_keys key is not a certificate, so no KRL/block can touch it (the deliberate backdoor).
• C: alice OK again — access restored purely by lifting the block and letting the host pull the fresh KRL. No re-issue.

Expected tail:

== PASS: per-host block denies on web1 only, break-glass survives, unblock restores ==

---

2. Concepts in 60 seconds

Term	What it is
User CA (acme-users)	signs people (user certs). Servers trust it via TrustedUserCAKeys.
Host CA (acme-hosts)	signs servers (host certs) and signs each composed KRL. Clients trust it via @cert-authority; krl-client verifies KRL signatures against it.
Block (not a revocation)	"deny THIS identity on THIS host." The cert stays active (valid elsewhere); a row in ssh_host_blocks is resolved into that host's KRL.
Composed per-host KRL	host-CA revocations ∪ all user-CA revocations ∪ resolve(active blocks on this host). Each host gets a different one. resolve(alice) = her unexpired cert serials + the SHA256 fingerprint of every key ever certified for her.
ECIES distribution	the KRL payload is encrypted to the host's own ecdsa-sha2-nistp256 host key (P-256 ECDH + HKDF-SHA256 + AES-256-GCM). The host decrypts locally — the KMS is never involved, the private key never leaves the box.
Anti-rollback	krl-client accepts a KRL only if its signed header number is strictly greater than the installed one. KRL numbers are globally monotonic (a single shared allocator across all lineages).

OpenSSH re-reads RevokedKeys on every publickey auth, so an atomically installed KRL takes effect on the next login with no sshd reload.

---

3. Prerequisites

• Docker + Docker Compose, uv, and a host python3.

• Network access to pki.joor.net and iam.joor.net (Keycloak).

• Admin password — Bitwarden item pki.joor.net. Export it:

  export PKI_PASSWORD=$(bash ~/.claude/skills/bitwarden/scripts/bw_exec.sh get password "pki.joor.net")

• The backend must run with SSH_ECIES_ENABLED=true (and SSH_HOST_KRL_SERVE at its default true), and its ssh_krl_seq allocator row must be seeded (fixed in pki-manager 3.6.1; see §11). Verify readiness in one call — 404/400 means ready, 501 means ECIES is disabled:

  curl -sS -o /dev/null -w "%{http_code}\n" -X POST https://pki.joor.net/api/v1/external/ssh/krl \
    -H 'Content-Type: application/json' -d '{"host_id":"probe.example.com"}'

---

4. Quickstart

cd ~/ssh-cert-test
# 0) build + stage the host puller once (it is git-ignored — it's a product of the
#    pki-manager repo, and the Dockerfile COPYs ./krl-client into the host image):
( cd ~/miimetiq3/pki-manager/krl-client && make build-static )
cp ~/miimetiq3/pki-manager/krl-client/dist/krl-client-linux-amd64 ./krl-client

docker compose up -d --build     # sshclient + sshweb1 + sshweb2 (Ubuntu 26.04)
export PKI_PASSWORD=$(bash ~/.claude/skills/bitwarden/scripts/bw_exec.sh get password "pki.joor.net")
./provision.sh                   # provisions everything and runs the 3 phases

provision.sh is idempotent — re-run it freely (it reuses the CAs, host records, identities and host keys it created; the ecdsa host key is cached in artifacts/ so it keeps matching the server's registered pubkey). The rest of this document is the same flow, explained.

---

5. Walkthrough with real output

5.1 Setup (steps 1–4)

The two CAs. acme-users signs people, acme-hosts signs servers and the KRLs. Both are ECDSA-P256; user/host keys are ed25519 / ecdsa respectively.

uv run python3 pki_client.py create-ca --type user --label acme-users
uv run python3 pki_client.py create-ca --type host --label acme-hosts

Trust anchors, three files (not two). Trust is split by direction, and krl-client adds a third consumer:

• User CA → each server's /etc/ssh/ssh-user-ca.pub (TrustedUserCAKeys).
• Host CA → client's known_hosts @cert-authority line (verify the host cert).
• Host CA → each server's /etc/ssh/ssh-host-ca.pub — the key krl-client verifies the composed KRL's signature against (composed KRLs are Host-CA signed; this is the BLK-10 trust-anchor reconciliation).

Hosts register with their ECDSA key. ECIES is P-256-only, so each host is registered with its ssh_host_ecdsa_key.pub — that public key becomes the ECIES recipient, and its private half is what krl-client decrypts with. The issued host cert is therefore an ecdsa host cert:

/etc/ssh/ssh_host_ecdsa_key-cert.pub:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com host certificate
        Signing CA: ECDSA SHA256:Lxx6wqlg8e5q21+baw7VyiUDE4lhtnaAgHvtU7UipJE (using ecdsa-sha2-nistp256)
        Key ID: "web1.acme.example-2026-07-05"
        Principals: web1.acme.example, web1

Users. alice (the block target) and jane (the control) each get a user cert with principal admins, mapped to the deploy account:

/root/.ssh/id_alice-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Signing CA: ECDSA SHA256:+fYXhmh5p4m998L9U34S4nkg5tBPu96V8Ao6r5oGosk (using ecdsa-sha2-nistp256)
        Key ID: "alice@acme.example"
        Serial: 7
        Valid: from 2026-07-05T00:35:56 to 2026-07-12T00:35:56   (~1 week)
        Principals: admins

The merged, effective server config shows certificate auth and the break-glass authorizedkeysfile live simultaneously:

pubkeyauthentication yes
trustedusercakeys /etc/ssh/ssh-user-ca.pub
hostcertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
authorizedprincipalsfile /etc/ssh/auth_principals/%u
revokedkeys /etc/ssh/revoked_keys
authorizedkeysfile .ssh/authorized_keys

5.2 TEST A — baseline (everyone reaches every host)

Each host pulls its own composed KRL (no blocks yet), then every login succeeds. Success here proves the whole ECIES pipeline: encrypt→pull→local-decrypt→ verify→install.

== [5] TEST A — baseline pull, then everyone logs in ==
   [web1] outcome=updated krl_number=2
   [web2] outcome=updated krl_number=3
   PASS  alice (cert)             web1.acme.example  -> ok
   PASS  alice (cert)             web2.acme.example  -> ok
   PASS  jane  (cert)             web1.acme.example  -> ok
   PASS  break-glass (raw)        web1.acme.example  -> ok

5.3 TEST B — block alice on web1 only

The operator blocks; the server synchronously rebuilds web1's composed KRL (alice's serial + key fingerprint), Host-CA-signed, with a higher number:

uv run python3 pki_client.py block --host-id "$WEB1_HID" --identity-id "$ALICE_IID" \
  --reason "lab demo: block on web1 only"

Both hosts pull — but web2's KRL is unchanged (up_to_date / 304), which is exactly why alice keeps access there:

== [6] TEST B — block alice on web1 (per-host deny; certs valid elsewhere) ==
   blocked: alice@acme.example on web1.acme.example -> new KRL # 4 blockCount 1
   [web1] outcome=updated     krl_number=4
   [web2] outcome=up_to_date  krl_number=0
   PASS  alice (cert) BLOCKED     web1.acme.example  -> deny
   PASS  alice (cert) elsewhere   web2.acme.example  -> ok
   PASS  jane  (cert) unaffected  web1.acme.example  -> ok
   PASS  break-glass survives     web1.acme.example  -> ok
   host-access view of web1 (operator's read-only status):
     distribution state: effective (served version matches current)
     BLOCKED: alice@acme.example — lab demo: block on web1 only

web1's freshly-pulled KRL marks alice's certificate REVOKED (it lists her serial and her key fingerprint), while it is untouched on web2:

docker exec ssh-cert-web1 ssh-keygen -Qf /etc/ssh/revoked_keys /tmp/alice-cert.pub
# /tmp/alice-cert.pub (alice@acme.example): REVOKED

That only tests one cert. To list everything a revoked_keys file revokes (serials, fingerprints, key-ids), use krl-decode.py — see §9.

sshd's own log on web1 confirms it was the KRL — not the test harness — that denied her. provision.sh starts each host's sshd with -E /var/log/sshd.log -o LogLevel=VERBOSE (the containers run no syslog daemon, so AUTH-facility messages would otherwise be dropped), and prints the decisive line:

Authentication key ED25519-CERT SHA256:P6f06A… revoked by file /etc/ssh/revoked_keys

distribution state: effective means the version web1 last fetched matches the current composed head — "served to the puller", not a guarantee of install (true confirmation would need a puller callback; short cert TTLs remain the primary revocation mechanism).

5.4 TEST C — unblock alice on web1 (restore)

== [7] TEST C — unblock alice on web1 (restore) ==
   unblocked -> new KRL # 5 blockCount 0
   [web1] outcome=updated krl_number=5
   PASS  alice (cert) RESTORED    web1.acme.example  -> ok
   sshd log on web1 — alice's cert accepted again after the unblock:
     | Accepted certificate ID "alice@acme.example" (serial 31) signed by ECDSA CA … via /etc/ssh/ssh-user-ca.pub

The KRL numbers climb monotonically (2 → 3 → 4 → 5), so anti-rollback is satisfied for free: every pull carries a strictly greater signed header number.

Baseline reset (step 4b). Before TEST A, provision.sh clears any lingering blocks for alice and jane on both hosts (unblocking a not-blocked identity is a no-op). This makes the run self-healing against a block left by manual UI/API testing or a crashed prior run — otherwise a stray block would fail TEST A/B before the demo starts.

---

6. How a host pull works (krl-client)

krl-client runs on each host (baked into the image at /usr/local/bin/krl-client). One run:

read state ─▶ POST /krl (If-None-Match: cached krl_version) ─┬─ 304 ─▶ up-to-date, exit 0
                                                             └─ 200 ─▶ ECIES decrypt (host ecdsa key)
   ─▶ parse + validate (host_id / valid_until / krl_version==sha256(krl) / anti-rollback)
   ─▶ verify detached Host-CA signature (ECDSA-P256/DER over sha256(krl))  ◀── BEFORE install
   ─▶ atomic install 0444 root:root  ─▶ persist state  ─▶ exit 0

Because every default derives from the backend's ssh-config.ts, the lab runs it with just the URL and host id — the ecdsa host key (/etc/ssh/ssh_host_ecdsa_key), the Host-CA anchor (/etc/ssh/ssh-host-ca.pub) and the install target (/etc/ssh/revoked_keys) are all already the right defaults:

krl-client --server-url https://pki.joor.net --host-id web1.acme.example

Stable exit codes let a scheduler alert precisely: 0 ok/up-to-date, 2 network, 3 decrypt, 4 signature, 5 expired payload, 6 host mismatch, 8 anti-rollback, 9 not-provisioned/disabled, 10 rate-limited. In production a systemd timer runs it every ~15 min; the lab drives it explicitly per phase so the tests are deterministic.

Fail-closed on unsigned KRLs. If Host-CA signing fails server-side the KRL is persisted with a null signature; krl-client then refuses it (exit 4, keeps the last-good KRL) unless --allow-unsigned is set.

---

7. Break-glass — the backdoor that survives a block

A raw authorized_keys key on deploy is a fallback that does not depend on the PKI. Certificate auth and a raw key are both the publickey method, so sshd accepts either. Crucially, a block cannot touch it: the KRL revokes certificates and key fingerprints, and the break-glass key is neither alice's key nor a cert — so when alice's cert is denied on web1, the break-glass key still logs in (TEST B, last row). That is the whole point of a break-glass path — and exactly why it must be tightly scoped and audited (it also bypasses the principal model). To kill it centrally you would list its fingerprint in the KRL.

---

8. The two sshd drop-ins

Drop-ins load in lexical order; single-valued directives are first-value-wins.

File	Source	Role
50-ssh-authorized-keys.conf	this lab	break-glass: PubkeyAuthentication yes + AuthorizedKeysFile .ssh/authorized_keys. Sorts before 60- so its value wins over any later AuthorizedKeysFile none.
60-ssh-ca.conf	PKI Manager (issued.sshdConfig)	certificate auth: HostCertificate, TrustedUserCAKeys, AuthorizedPrincipalsFile, RevokedKeys (all on the ecdsa host key).

---

9. pki_client.py + krl-client + krl-decode.py reference

pki_client.py (operator; every subcommand prints JSON; needs PKI_PASSWORD):

Command	REST	Purpose
list-cas	GET /ssh/cas	list the SSH CAs
create-ca --type user|host [--label L]	POST /ssh/cas	create a CA (one active per type)
trust-anchors	GET /ssh/trust-anchors	User CA + Host CA public keys
register-host --fqdn F --addresses a,b --pubkey-file P [--no-issue]	POST /ssh/hosts (+ /hosts/issue)	register a host (ecdsa pubkey) + issue its cert
issue-host-cert --host-id ID [--ca-id ID]	POST /ssh/hosts/issue	(re)issue a host cert
create-identity --subject S [--email E]	POST /ssh/identities	create an identity
issue-user-cert --identity-id ID --pubkey-file P --principals a,b [--ttl N]	POST /ssh/users/issue	sign a user cert
block --host-id ID --identity-id ID [--reason R]	POST /ssh/blocks	block the identity on that host (regenerates its KRL synchronously)
unblock --host-id ID --identity-id ID	POST /ssh/blocks/unblock	lift the block (row kept for audit)
host-access --host-id ID	GET /ssh/hosts/:id/access	who can reach the host + distribution state + active blocks

krl-client (host puller): --server-url (required), --host-id (default hostname -f), --host-key (default /etc/ssh/ssh_host_ecdsa_key), --ca-pubkey (default /etc/ssh/ssh-host-ca.pub), --krl-file (default /etc/ssh/revoked_keys), --state-dir (default /var/lib/krl-client), --dry-run, --allow-unsigned. Built from pki-manager/krl-client (make build-static).

krl-decode.py (inspector; lists what a revoked_keys file revokes). OpenSSH itself can only test one cert against a KRL — ssh-keygen -Qf <krl> <cert.pub> prints REVOKED or nothing — and has no dump mode. This decodes the binary PROTOCOL.krl format and enumerates the whole file: cert serials grouped by signing CA, revoked key SHA256 fingerprints, key-ids, and explicit keys.

# containers have no python3, so copy the file out and decode on the host:
docker cp ssh-cert-web1:/etc/ssh/revoked_keys /tmp/rk
python3 krl-decode.py /tmp/rk

KRL  header#=53  ...  comment: pki-manager ssh host web1.acme.example composed KRL
  revoked cert serials  (CA ecdsa-… SHA256:+fYXhmh5…): 1, 3, 5, 7, 9, 11, …, 37
  revoked key fingerprint: SHA256:GCR68tRD1thnp8WdU6MyYLcDkeAbvjfAFTxs+uGcoRI   ← alice's key
summary: 19 serial(s), 0 key-id(s), 5 fingerprint(s), 0 explicit key(s)

The fingerprints match what ssh-keygen -lf <pubkey> reports and what sshd logs on a rejection, so you can confirm exactly why a login was denied. Blocking one identity typically yields many entries — every not-yet-expired serial and every key ever certified for it (belt-and-braces; verified OpenSSH fact 3).

---

10. Cleanup

docker compose down          # remove the three containers + network
# rm -rf artifacts           # (optional) remove generated keys/certs/host-key cache

Records on pki.joor.net (CAs acme-*, hosts web1/web2.acme.example, identities alice/jane@acme.example) remain — remove them from the UI/DB to reset the instance (there is no REST retire/delete for CAs/hosts).

---

11. Notes, gotchas, troubleshooting

• ECIES needs an ecdsa host key. ed25519-only hosts cannot do P-256 ECIES — /krl returns 404 ECIES_KEY_UNSUPPORTED. The lab registers each host with its ssh_host_ecdsa_key.pub and regenerates unique host keys per container (the image bakes identical keys into both; the ecdsa key is then cached in artifacts/ so a recreated container still matches the server's registered pubkey).
• ssh_krl_seq must be seeded. The global KRL-number allocator is a single-row table seeded by migration 0008. If its row is missing, all KRL generation fails (ssh_krl_seq allocator row missing) → /krl returns 503 NO_KRL. Fixed in pki-manager 3.6.1; if you hit it on an older build, seed it: INSERT INTO ssh_krl_seq (id, value) VALUES (1, COALESCE((SELECT MAX(krl_number) FROM ssh_krls), 0));
• No fleet token needed. Operator actions use the admin OIDC token; the host's POST /krl is unauthenticated (ECIES means only the target host can decrypt). The register-host-pubkey endpoint (which does want a fleet token) is unnecessary here — the ecdsa pubkey is stored at host registration.
• /ssh/* public routes are SPA-shadowed on this deployment (GET /ssh/host-ca-keys returns HTML). The lab sources the Host-CA anchor from /api/v1/ssh/trust-anchors (proxied, works) instead.
• Effective ≠ installed. distribution state: effective means the current version was served to the puller, not that it is installed. Short cert TTLs (+1w users) remain the primary revocation mechanism; blocks add immediacy.
• NTP matters. Each ECIES payload carries a short valid_until; a host whose clock has drifted past --clock-skew (default 300s) rejects every pull (exit 5). The lab's containers share the WSL2 kernel clock, so this is a non-issue.
• Blocks are per-host and per-identity. A block on web1 never changes web2's KRL (its next pull is a 304); blocking alice never affects jane. The fingerprint entry over-blocks only if two identities share the same public key (warned via warnings.sharedKeyCollisions at block time).
• Racy sshd restart is handled by restart_sshd, which waits for :22 to free, starts sshd, then confirms it is listening (a silent bind failure would otherwise slip past set -e).

---

12. Meta — mother project, CI, versioning

• Mother project: oriolrius/pki-manager-web. This repo is a black-box harness for its SSH Certificate Manager; the feature under test is specified in that project's decision-016, and the krl-client host puller is built from its krl-client/ module.
• Nothing sensitive is published. artifacts/ (private host keys, issued certs, identity ids) and the built ./krl-client binary are git-ignored. CI (.github/workflows/ci.yml) enforces it on every push/PR: gitleaks scans the full history for secrets/private keys (.gitleaks.toml allowlists only public SSH keys and fingerprints in the docs), and a guard step fails the build if any private-key / artifacts/ / binary path is ever tracked.
• Versioning is SemVer via Commitizen. Commits follow Conventional Commits (feat: → minor, fix: → patch, !/BREAKING CHANGE → major); CI lints PR commit messages with cz check. Cut a release with cz bump (updates .cz.toml, tags vX.Y.Z, writes CHANGELOG.md). Config in .cz.toml.