mayfly-integration — Mayfly Platform Integration Environment

A hermetic, deterministic Docker environment that runs the whole Mayfly platform — server + agent + helper + a GitHub stand-in — on one isolated network.

This is the standalone platform repository (git@github.com:mayfly-sh/mayfly-integration.git), extracted from the former top-level integration/ directory in milestone 014A (implements ADR-0026). It is the canonical home for all platform-level integration: Docker assembly, docker compose, bootstrap/provisioning, the mock GitHub service, the test runner, acceptance / lifecycle / failure-injection / chaos / performance / benchmark suites, fixtures, generated artifacts, the version-compatibility matrix, future Kubernetes manifests, integration docs, and the platform CI pipeline.

Dependency direction is one-way (ADR-0026): mayfly-integration consumes the crate repositories (mayfly-server, mayfly-agent, mayfly-helper, mayfly-cli); no crate depends on it. It introduces no library import — every image builds the real crate source, reconstituted as siblings at build time.

Repository docs

In-repo guides live under docs/: architecture · developer setup · running locally · running CI · workspace layout · ownership · version compatibility. Validated crate revisions: compatibility/versions.lock + compatibility/matrix.md.

The cross-repo engineering memory ("one shared brain") stays in the shared .cursor/ at the workspace root (NOT vendored into this repo):

• Canonical design: .cursor/outputs/analysis/architecture/integration-environment.md
• Decision records: .cursor/outputs/decisions/ADR-0014-integration-environment-architecture.md, .cursor/outputs/decisions/ADR-0026-mayfly-integration-repository.md (this extraction)
• Status (milestone 010F): the environment proves the happy path (010C), failure behaviour (010D), the production deployment profile on real systemd (010E), and now does so with real TLS verification and digest-pinned images (010F). tests/run.sh (CI profile, 11/11) runs the 010B infra checks, the 010C happy-path e2e (tests/e2e.sh), then the 010D failure suite (tests/failures.sh). tests/run-systemd.sh (systemd profile) runs tests/systemd.sh (71/71) and tests/install.sh (28/28). tests/security-checks.sh (9/9) statically asserts no test hooks / insecure defaults / elevated caps reach the ship path.
• TLS (010F, R-INT-1 closed): the server presents a CA-issued certificate whose SAN includes the mayfly-server DNS name (fixtures/tls/), loaded via the production cert_path/key_path path. The agent pins the CA via tls_ca_path (baked at /opt/mayfly/integration-ca.crt) and verifies the chain with rustls; the test-runner + the server's healthcheck trust the CA from the system store. allow_insecure_tls and every curl -k are gone. Regenerate the test PKI with fixtures/tls/regen.sh (TEST-ONLY, 100-year validity).
• Images (010F, R-INT-2 closed): all base images are @sha256 digest-pinned in .env.example + the Dockerfiles. Bump via docker buildx imagetools inspect.
• CI: the canonical pipeline is .github/workflows/ci.yml (see .cursor/outputs/analysis/release/ci-guide.md). musl static builds (BL-012) run there via cross — the macOS dev host can compile but not cross-link musl.

Topology

              ┌─────────────┐        ┌──────────────────────────────────────┐
              │ mock-github │◀──────▶│ mayfly-server (HTTPS :8443, SQLite,   │
              │  :8080      │  http  │ CA, CA-issued TLS) health:/api/v1/health│
              └─────────────┘        └──────────────────────────────────────┘
                     ▲                          ▲ https (CA-verified since 010F)
                     │                                │
              ┌──────┴────────────────────────────────┴─────────────────────┐
              │ managed-host (one container, shared PID ns)                   │
              │   sshd  ·  mayfly-helper (root, UDS /run/mayfly/helper.sock)  │
              │           mayfly-agent (unprivileged "mayfly" uid)            │
              └──────────────────────────────────────────────────────────────┘
              ┌─────────────┐
              │ test-runner │  in-network probes + assertions (the only judge) │
              └─────────────┘

All four roles share the mayfly-net bridge with service-name DNS aliases.

Prerequisites

• Docker Engine 24+ and the Docker Compose v2 plugin (docker compose).
• ~2 GB free disk for images; the first build compiles all three Rust binaries.
• The sibling crate repos available for image builds (the build context is the superproject root, context: ..). See Workspace modes below.

Workspace modes (dev vs CI)

Images build the real crate source, so the build needs the sibling crate repos beside this one. scripts/workspace.sh reconstitutes that layout and keeps the build-context .dockerignore in sync. URLs/refs are configurable via env (defaults: github.com/mayfly-sh + the pins in compatibility/versions.lock).

# Development: use sibling repos already checked out in the workspace
#   mayfly/{mayfly-integration,mayfly-server,mayfly-agent,mayfly-helper,mayfly-cli}
./scripts/workspace.sh dev      # verifies siblings exist, syncs .dockerignore

# CI / fresh machine: clone the siblings via git into the workspace root
./scripts/workspace.sh ci       # clones server/agent/helper/cli at pinned refs
# overrides: MAYFLY_GIT_BASE, MAYFLY_<CRATE>_REPO, MAYFLY_<CRATE>_REF

See docs/workspace-layout.md and docs/running-ci.md for details.

Quick start

cd mayfly-integration

# 0. Reconstitute the superproject (dev mode uses workspace siblings)
./scripts/workspace.sh dev

# 1. Bootstrap (idempotent: writes .env from .env.example, makes artifacts/,
#    syncs the build-context .dockerignore)
./scripts/bootstrap.sh

# 2. Build all images
./scripts/build.sh

# 3. Bring up and block until every service is healthy (no fixed sleeps)
./scripts/up.sh

# 4. Run the COMPLETE suite: infra checks + the 010C happy-path e2e.
#    Reuse already-built images with MAYFLY_SKIP_BUILD=1.
MAYFLY_SKIP_BUILD=1 ./tests/run.sh

# 5. Tear down. Keeps volumes by default; --volumes for a clean slate.
./scripts/down.sh            # stop, keep volumes
./scripts/teardown.sh        # stop + remove volumes (clean rerun)

To run only the end-to-end happy path (it performs its own staged bring-up and down -v):

cd mayfly-integration
./tests/e2e.sh                       # builds images, runs the full happy path
MAYFLY_SKIP_BUILD=1 ./tests/e2e.sh   # reuse already-built images
MAYFLY_KEEP_UP=1   ./tests/e2e.sh    # leave the stack up afterwards to inspect

End-to-end happy path (tests/e2e.sh, milestone 010C)

The e2e uses a staged bring-up so an enrollment token can be minted before the agent starts (otherwise the agent would race ahead with no valid token):

1. Bring up mock-github + mayfly-server + test-runner; wait for health.
2. The test-runner authenticates through the deterministic device flow and calls POST /api/v1/admin/machines/enrollment-tokens to mint a real token.
3. Bring up managed-host with that token injected → the agent enrolls.
4. Poll predicates until the agent converges (identity persisted, generation applied, TrustedUserCAKeys written, server synced_generation ≥ 1).
5. Issue a real SSH user certificate and SSH in with only the certificate.
6. Evaluate the 16 required assertions and tear down (down -v).

It asserts every required stage:

#	Assertion	Source of truth
1	enrollment completed	machines row in server DB
2	identity persisted	agent identity/machine.json == server machine_id
3	heartbeat accepted	agent runtime_status.json + server last_seen
4	bundle downloaded	audit bundle.downloaded
5	bundle signature valid	TOFU pin present + applied generation ≥ 1
6	helper received apply request	helper log line
7	TrustedUserCAKeys updated	managed file contains a CA key
8	sshd configuration valid	sshd -t
9	sshd reload succeeded	helper apply log
10	helper VerifyState succeeds	helper-ping.py verify_state
11	bundle ACK accepted	audit bundle.applied
12	fleet rollout updated	synced_generation + admin rollout API (100%)
13	certificate issued	audit certificate.issued + session log
14	certificate principal correct	audit metadata + cert principal
15	SSH login succeeds (cert-only)	ssh exit 0 + marker; no authorized_keys
16	audit chain updated	contiguous chain_position + lifecycle events

Artifacts land in artifacts/: ssh-session.log (issued cert + login), fleet-status.json, e2e-summary.json, and *.e2e.log service logs.

Example: issued certificate + certificate-only SSH login

[e2e] certificate issued: principal=integration-bot fingerprint=SHA256:1Lehscv…
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Signing CA: ED25519 SHA256:LEFlu89Y…   Key ID: "mayfly-ca"
        Principals:
                integration-bot
[e2e] attempting SSH as 'integration-bot@managed-host' using ONLY the issued certificate
[e2e] SSH certificate login SUCCEEDED
MAYFLY_SSH_OK user=integration-bot host=80f28ab845f3

fleet-status.json after rollout: {"latest_generation":1,"total_machines":1, "online":1,"rollout_percentage":100.0,…}.

Profiles (ADR-0014)

Profile	How	What it proves	Privilege
CI (default)	docker compose ...	deterministic bring-up; sshd reload via systemctl-shim.sh	low (cap_drop: ALL + curated cap_add)
systemd (fidelity)	MAYFLY_PROFILE=systemd ...	real PID-1 systemd + the hardened units + a real systemctl reload	higher (SYS_ADMIN, SETPCAP, host cgroup)

# systemd fidelity profile — the full 010E lane (builds images, runs both suites,
# self-managed bring-up/teardown). Works on Docker Desktop (LinuxKit cgroup v2).
./tests/run-systemd.sh
MAYFLY_SKIP_BUILD=1 ./tests/run-systemd.sh   # reuse already-built images

The CI profile is the canonical, fully verified path. The systemd profile is opt-in and validates the production sandbox (closes D-010 in 010E). The systemd test container is granted SYS_ADMIN + SETPCAP and the host cgroup mount so that systemd can set up per-unit mount namespaces and drop the agent unit's capability bounding set — privileges real PID 1 holds intrinsically. The production units are unchanged and least-privilege (R-INT-6); the canonical CI profile stays low-privilege.

Systemd-fidelity validation (tests/systemd.sh + tests/install.sh, milestone 010E)

tests/run-systemd.sh builds the systemd managed-host image and a bare install-host image, then runs both suites against real systemd as PID 1 (cgroup v2, journald, real service management):

Suite	Group	What it proves
systemd.sh	S1 units	helper.service/agent.service active; ordering (helper before agent); Restart=on-failure + RestartSec; auto-restart after SIGKILL; clean stop/start; daemon-reload; enable; status; journal logging
systemd.sh	S2 socket	helper.sock 0660 root:mayfly; recreated after restart; stale socket cleaned up; agent survives helper restart
systemd.sh	S3 hardening	effective sandbox via systemctl show + systemd-analyze security (helper 3.8 / agent 1.4, both OK): NoNewPrivileges, ProtectSystem, ProtectHome, ProtectProc, PrivateTmp, PrivateDevices, CapabilityBoundingSet, SystemCallFilter, IPAddressDeny, UMask, RemoveIPC, KeyringMode — without breaking runtime
systemd.sh	S4 recovery	restart server/agent/helper individually → auto-recovery; identity persists; no generation regression; socket recreated; SSH still works
systemd.sh	S6 logging	journald has startup/restart lines; no capability token / private key leakage
systemd.sh	S7 filesystem	ownership/permissions of /etc/mayfly-agent, /run/mayfly, /var/lib/mayfly (0700 — the agent re-asserts stricter than the unit), identity/
systemd.sh	S8 boot	fresh boot → services auto-start; helper before agent; socket ready; enrollment + bundle recovery; SSH functional
systemd.sh	S9 stress	3× restart cycles → deterministic recovery, no stale sockets, no zombies, no degradation
systemd.sh	G0 e2e	certificate-only SSH login through the real systemctl reload ssh
install.sh	I install	real deploy/install.sh on a bare systemd host: users/dirs/perms, units enabled, services active
install.sh	I idempotency	repeated install is idempotent; reinstall keeps working
install.sh	I uninstall	uninstall.sh preserves state; uninstall.sh --purge removes state + the mayfly user/group

./tests/run-systemd.sh                        # both suites, full build + teardown

The agent installer writes identity_dir=/var/lib/mayfly/identity so the machine key lands under the agent-writable state dir; the hardened unit runs ProtectSystem=strict (so /etc is read-only at runtime). This was a real install bug surfaced by 010E — see ADR-0016 §2.

Containers

Service	Image	Role
mayfly-server	mayfly-integration/docker/server/Dockerfile	control plane; renders config.yaml (GitHub + Keycloak providers), auto-bootstraps a CA, self-signs dev TLS, serves HTTPS :8443 (published to 127.0.0.1:8443)
mock-github	mayfly-integration/docker/mock-github/Dockerfile	deterministic Device-Flow/REST GitHub stand-in (:8080, stdlib only)
mock-keycloak	mayfly-integration/docker/mock-keycloak/Dockerfile	deterministic Keycloak/OIDC stand-in (014B): discovery + JWKS + device grant + RS256-signed JWTs the server verifies (:8080, PyJWT+cryptography)
managed-host	mayfly-integration/docker/managed-host/Dockerfile{,.systemd}	real sshd + root mayfly-helper + unprivileged mayfly-agent, co-located
test-runner	mayfly-integration/docker/test-runner/Dockerfile	curl/jq/ssh toolbox + the real mayfly CLI (014B); idle, exec'd into by the tests

Filesystem layout (managed-host, ADR-0014 §11)

Path (named volume)	Owner / mode	Contents
/etc/mayfly-agent (mayfly-etc)	root:mayfly 0750	config.toml, helper.token (0640), helper.env, agent identity dir lives under state
/run/mayfly (mayfly-run)	root:mayfly 2750	helper.sock (0660, group mayfly)
/var/lib/mayfly (mayfly-state)	mayfly:mayfly 0750	agent state + identity/
/etc/ssh (ssh-config)	system	sshd_config (+ Include), host keys, sshd_config.d/90-mayfly.conf, mayfly/trusted_user_ca_keys
server /data (server-data)	mayfly:mayfly	config.yaml (sets cert_path/key_path), mayfly.db, ca/ (CA-issued TLS material is baked at /etc/mayfly/tls)

Bootstrap & configuration generation

Two deterministic, idempotent layers:

1. Host-side (scripts/bootstrap.sh): creates .env (from .env.example) and artifacts/, and syncs the build-context .dockerignore to the superproject root via scripts/workspace.sh ensure. Every value in .env is a fixed TEST-ONLY fixture.
2. In-container provisioning (docker/managed-host/provision-host.sh, docker/server/entrypoint.sh): create users/dirs/permissions, generate the capability token once, pin the helper SO_PEERCRED uid, render config.toml / config.yaml, install the sshd drop-in, and generate host + dev-TLS keys. Re-running never regenerates an existing token or host key.

Health checks (no arbitrary sleeps — ADR-0014 §14)

Service	Check
mock-github	GET /healthz → 200
mayfly-server	GET https://localhost:8443/api/v1/health → status:"ok"
managed-host	sshd running and helper.sock answers an authenticated Ping and mayfly-agent running
ordering	depends_on: { condition: service_healthy }; up.sh uses --wait

scripts/up.sh blocks on compose --wait; tests poll predicates via scripts/lib.sh:poll_until.

Logging

Each service logs single-line JSON to stdout. scripts/logs.sh collects per-component logs (timestamped) into artifacts/<UTC-timestamp>/; down.sh flushes logs automatically before stopping.

Infrastructure tests

tests/run.sh runs the suite (CI profile by default):

Test	Needs stack	Asserts
test_compose_config.sh	no	both profiles parse; 4 roles present
test_bootstrap_idempotent.sh	no	.env created once, unchanged on re-run
test_startup_ordering.sh	no	service_healthy dependency edges declared
test_bringup_health.sh	yes	all healthy; server/mock-github/sshd reachable; /ready ok
test_config_generation.sh	yes	config/token(0640 root:mayfly,64 hex)/socket(0660)/drop-in/identity generated
test_volume_persistence.sh	yes	marker + sqlite db survive a restart
test_cleanup.sh	yes	teardown removes all containers + volumes
e2e.sh (010C)	self-managed	the complete happy path; 16 required assertions (see above)
failures.sh (010D)	self-managed	failure injection & lifecycle validation; 30 assertions (see below)

test_startup_ordering.sh checks test-runner depends on the control plane but not on managed-host — the 010C e2e must mint a token through the test-runner before managed-host starts, so that edge would deadlock it.

Failure injection & lifecycle validation (tests/failures.sh, milestone 010D)

The canonical failure-validation suite. It drives the real components through faults using deterministic, predicate-polled, isolated staged bring-ups (each group does its own down -v), and proves the security-relevant properties: fail-closed, rollback when required, no partial writes, generation never advances incorrectly, audit records, deterministic recovery, agent/helper synchronized.

Group	Scenario	Key assertions
D1	injected sshd reload failure	helper rolls back; TrustedUserCAKeys absent; agent + server generation stay 0; bundle.rollback audited; cert SSH login FAILS (fail-closed)
D2	invalid enrollment token	agent stays alive + retries; identity never persisted; 0 machines server-side; cert SSH login FAILS before any apply
D3	bad certificates	wrong-principal, untrusted-CA, and expired certs are all rejected; a valid cert still logs in (control)
D4	helper live negatives	helper rejects an invalid capability token; VerifyState detects TrustedUserCAKeys drift
D5	runtime restart recovery	restart managed-host + server → auto re-convergence; identity / generation / trusted-CA persist; SSH keeps working
D6	network partition	server unreachable → agent survives + backs off; reconnect → auto-converges; SSH works again

Failure logic a real component cannot emit end-to-end (crafted invalid bundles, replay/skew, IPC framing) is owned by the crates' unit/integration tests; the full mapping is in .cursor/outputs/analysis/release/failure-matrix.md.

Test-only fault-injection hooks (never created in production): a reload-fail sentinel in mayfly-helper/deploy/docker/systemctl-shim.sh (armed by MAYFLY_INJECT_RELOAD_FAIL=1 via the provisioner), a mayfly-intruder login account, and a bad_token probe in helper-ping.py.

./tests/failures.sh                       # builds images, runs all 6 groups
MAYFLY_SKIP_BUILD=1 ./tests/failures.sh   # reuse already-built images

Troubleshooting

• Daemon not running: docker info must succeed. On macOS start Docker Desktop.
• Server unhealthy, "unknown field ... MAYFLY_ ...": the server treats every MAYFLY_* env var as a config key. Only valid nested keys (MAYFLY_SERVER__HOST, MAYFLY_GITHUB__*, …) are allowed; do not set MAYFLY_CONFIG here.
• Agent shows repeated enrollment rejected/retry on a plain up: expected — a bare scripts/up.sh injects only the structurally-valid placeholder token from .env (not in the server DB), so the agent retries (and stays alive). Bring-up health does not require a successful enroll. The 010C e2e mints a real token and brings managed-host up with it, so it enrolls for real.
• e2e.sh fails at "helper VerifyState succeeds": the on-disk sshd_config.d/90-mayfly.conf must match mayfly-helper's render_dropin byte-for-byte (the helper compares them and fails closed on drift). The shipped mayfly-helper/deploy/sshd/90-mayfly.conf is pinned to that rendering by a unit test (deploy_dropin_asset_matches_rendered); keep them in sync.
• e2e.sh can't read the audit DB: the server image bundles sqlite3 (010C) and the e2e queries /data/mayfly.db read-side; ensure the server image was rebuilt after pulling 010C.
• failures.sh D6 "heartbeat fails during partition" flakes: detection is bounded by the agent's 30s heartbeat request timeout (a stale keep-alive blocks until then), so the suite polls with a ~95s window (R-INT-5). If it still flakes, the docker network disconnect did not take effect — check the derived network name/container id in the D6 step.
• failures.sh D1 never sees the rollback: the reload-fail sentinel must be armed before the agent starts — set MAYFLY_INJECT_RELOAD_FAIL=1 on managed-host (the suite does this for group C only); the provisioner drops /run/mayfly/inject/reload-fail, which systemctl-shim.sh checks on reload.
• helper ping failed: No module named 'json': the managed-host image needs the full python3 package (not python3-minimal).
• systemd profile won't start: it needs a cgroup-v2 engine and the container to run with cgroup: host + SYS_ADMIN + SETPCAP (the run-systemd.sh lane sets these). It does run on Docker Desktop (LinuxKit cgroup v2). If a unit fails with status=218/CAPABILITIES, the container is missing SETPCAP — systemd (PID 1) needs it to drop the agent unit's empty CapabilityBoundingSet. Use the CI profile for day-to-day work; the systemd profile is the fidelity gate.
• Inspect a service: docker compose -f docker-compose.yml logs <svc> or ./scripts/logs.sh <svc>.

Determinism guarantees (ADR-0014 §16)

Fixed fixtures (token, passphrase, signing-key seed, GitHub identity), low intervals + zero jitter, predicate polling (never blind sleeps), idempotent bootstrap, and down.sh --volumes for a clean slate between runs.

All secrets here are test-only. Never reuse them anywhere real.