feat: dblab provisioning module + env-create honesty gate (#127)

[samo-agent]

Summary

Closes the gap from #127: SAMO clients are now dblab-ready by default at provision time. The hardening-only cloud-init now gains an optional dblab module that provisions the full minimal €0 profile verified in #127.

• src/cloudinit/dblab.ts — new Module implementation. Generates a deterministic cloud-init fragment that:

  • Installs Docker CE from the official apt repo + zfsutils-linux.
  • Creates a sparse 10 GB loopback ZFS pool (dblab) on the root disk with a systemd persistence service.
  • Caps ZFS ARC at 256 MB via /etc/modprobe.d/zfs.conf (applied live + baked into initrd).
  • Creates a 2 GB swapfile persisted in /etc/fstab.
  • Writes /root/.dblab/engine/configs/server.yml with the minimal profile: port 2345, maxCloneCount=2, maxIdleMinutes=20160, cloneAccessAddresses=127.0.0.1, portPool 6000–6099, containerResources.memory=1073741824 (1 GB), shared_buffers=128MB, source = host PG via Docker bridge 172.17.0.1:5432, logical retrieval jobs.
  • Generates a random verification token at provision time (never hardcoded).
  • Writes the busybox-grep workaround startup.sh.
  • Adds the Docker bridge subnet to pg_hba.conf so the engine can dump prod Postgres.
  • Runs dblab_server:4.1.3 on 127.0.0.1:2345 only (--privileged, --restart on-failure, rshared).
  • Installs the dblab CLI v4.1.3, inits it, waits for healthz, triggers the initial refresh.
  • Masks ssh.socket to prevent the Ubuntu 24.04 restart loop.
  • NEVER disables fail2ban, NEVER deletes the UFW SSH rule (SAMO client package: every client gets prod + dblab previews + auto-deploy #127 critical lesson baked in as constants + tested).

• src/commands/preview.ts — resolveModules now has a MODULE_REGISTRY map. "dblab" resolves to dblabModule. Unknown names still error.

• src/commands/env.ts — EnvExecDeps gains optional dblabPreflight?. runEnvCreate gates on it when db=dblab: BLOCKED/UNKNOWN/throw → exit 1 + clear error, no env record written (the samograph broken path: fake record written, clone never existed).

Root-cause analysis (why the gate matters)

samograph's manual dblab install wrote env records without verifying the engine was healthy. The reconciler saw needDeploy=false on those records and never retried the broken env. The gate is inserted BEFORE the DNS step and BEFORE any record write, so the only way an env record exists is if the engine was confirmed READY at create time.

fail2ban / control-plane exemption

The dblabModule cloud-init fragment contains zero fail2ban or UFW commands. The hardening module (always rendered first) writes jail.local with trustedIps in ignoreip; provision.ts auto-injects the control-plane egress IP into trustedIps before rendering. The combined output keeps fail2ban enabled and the control-plane whitelisted.

Tested assertions:

• never toContain("systemctl stop fail2ban")
• never toContain("systemctl disable fail2ban")
• never toMatch(/systemctl\s+(stop|disable)\s+fail2ban/)
• never toContain("ufw delete")
• toContain("systemctl enable --now fail2ban") (from hardening module)
• toContain("ignoreip") + toContain("91.99.233.145") within 200 chars

TDD

RED commit a03f09f: 2 test files, 0 source changes, CI red on missing module import + missing gate.
GREEN commit 79a0e45: implementation, 1423/1423 tests pass, tsc clean.

What real-provision smoke test is still needed post-merge

1. Provision a fresh cx22 with --module dblab and walk the 19 steps from SAMO client package: every client gets prod + dblab previews + auto-deploy #127 manually on the live VM.
2. Verify curl http://127.0.0.1:2345/healthz returns {"edition":"community",...}.
3. Run samohost env preflight <vm> → READY.
4. Run samohost env create <vm> <app> --branch smoke --db dblab → exit 0 + env record written.
5. Backfill: samorev-client + game-changers VMs (currently db=none) need the module applied and a first refresh with their prod database name substituted into server.yml.

Not in this PR (deferred, filed confidentially)

• #124 findVm ignores lifecycleState (pre-existing)
• #125 trigger swallows preview failures (pre-existing)
• Production wiring of dblabPreflight in defaultEnvExecDeps() (requires the healthz curl probe via SSH runner — low-risk follow-up once the module is deployed and smoke-tested)

Closes #127 provisioning checklist item.

🤖 Generated with Claude Code

[samo-agent]

DBLab provisioning fixes — 4 root-cause bugs addressed (#128)

Reference: samograph dead-engine root cause (task wdgv5zs85)

Commits added

SHA	Description
4e21325	test(RED): 4 failing tests pinning the bugs
b831d93	fix(dblab): GREEN — all 4 fixes implemented

---

Fix 1 (CRITICAL): poolManager.preSnapshotSuffix: "_pre"

Without this key, DBLab v4 runs:

zfs list -t snapshot -H -o name | grep -v ''

BusyBox grep (Alpine base image in the postgresai/dblab-server container) rejects an empty -v pattern with exit 2 → DBLab interprets this as "no available pools" → the engine cannot attach to the ZFS pool → zero clones, dead engine. This is exactly what happened on samograph's first provision.

Fix 2: selectedPool: (correct yaml struct tag)

pool: dblab was silently ignored by DBLab's config parser. The correct key is selectedPool: per the v4.1.3 schema. Harmless but necessary for the pool to actually be selected.

Fix 3: Source DB config-driven via makeDblabModule({ sourceDb })

Preview envs must connect to <app>_template, not <app>_prod. Samograph's incident proved that pointing the retrieval source at samograph_prod caused the snapshot to fail to initialise (wrong roles/schema for preview). The new factory bakes the correct DB name into server.yml at provisioning time:

// When provisioning a samograph DBLab VM:
const mod = makeDblabModule({ sourceDb: "samograph_template" });

The existing dblabModule export uses a ${DBLAB_SOURCE_DB} placeholder for back-compat.

Fix 4: defaultEnvExecDeps().dblabPreflight wired to production

The dblabPreflight field was absent from defaultEnvExecDeps(), meaning the env-create honesty gate (which checks the engine before creating a clone) was silently skipped in production. A db=dblab env-create against a non-running engine would write a fake env record — the same broken path that samograph hit.

The production implementation runs the full DBLAB_PROBES audit script over a pinned SSH connection and reuses evaluateDblabPreflight() so the gate and samohost env preflight <vm> cannot diverge.

---

Test summary

89 dblab/gate/preflight tests pass. Pre-existing smol-toml package-not-found errors in this test environment are unaffected by these changes.

---

Not merged — awaiting samorev + smoke-provision on a real cx22.

[samo-agent]

Smoke-test YAML bug fixes — RED/GREEN

Smoke-provision on a real VM produced [FATAL] failed to parse config. Root-caused 4 structural bugs in buildServerYml() vs the known-good samograph server.yml (ssh -p 2223 samo@116.203.249.135).

Commits pushed

SHA	Role
11f1f02	RED — YAML-parse + schema validation test (test/dblab-schema-yaml-128.test.ts) using js-yaml; 14 assertions fail on current PR head for the exact structural reasons
8b807e4	GREEN — implementation fixes; all 1449 tests pass

Bugs fixed

BUG-1 provision.cloneAccessAddresses was a YAML sequence - "127.0.0.1" — DBLab v4.1.3 expects a scalar string "127.0.0.1".

BUG-2 retrieval structure wrong:

• skipStartRefresh moved from retrieval.spec.skipStartRefresh → retrieval.refresh.skipStartRefresh
• jobs moved from inline-options objects inside retrieval.spec.jobs → flat string list at retrieval.jobs
• Job options now correctly at retrieval.spec.<job>.options

BUG-3 losetup hardcoded /dev/loop0; Ubuntu 24.04 + snapd occupy loop0–loop2. Changed to losetup -f --show (dynamic) with $LOOP_DEV used in zpool create. dblab-loopback.service now uses losetup -j check + losetup -f on re-attach.

BUG-4 ${DBLAB_SOURCE_USER} placeholder was never substituted → logicalDump connection sent a literal ${DBLAB_SOURCE_USER} to Postgres, causing silent refresh failure behind || true. Now substituted with adminUser from ProvisionSpec at build time via buildServerYml(opts, adminUser).

What the new test does differently

Previous tests used toContain() string checks which let invalid YAML structure through. The new test/dblab-schema-yaml-128.test.ts file:

1. Calls makeDblabModule(opts).cloudInitFragment(spec) to get the raw fragment
2. Extracts server.yml content from writeFiles
3. Parses it with js-yaml.load() — fails immediately if YAML is malformed
4. Asserts correct Go/DBLab schema shapes: scalar vs list, exact key nesting, correct values

Do NOT merge — awaiting a fresh real-VM re-smoke on the updated branch.

[samo-agent]

Fresh-VM gap analysis — GAP-1 + GAP-2 closed (TDD RED→GREEN)

SSH ground-truth source: samograph 116.203.249.135 (live, clone-verified)

Gaps found by diffing module output vs samograph working config

#	Gap	Root cause	Fix
GAP-1	selectedPool: "dblab"	DBLab v4 scans mountDir for subdirectories. Value must be dataset name "dblab_pool" (at /var/lib/dblab/dblab_pool), not ZFS pool name "dblab". Using "dblab" → looks for /var/lib/dblab/dblab which doesn't exist → "no available pools" loop.	selectedPool: "dblab_pool"
GAP-2	UFW clone-port rules absent	No ufw allow for Docker bridge → ports 6000-6099. UFW default-deny blocks engine health probes; 100 ports × ~3s TCP timeout = ~5 min stall → engine never healthy.	Added ufw allow proto tcp from 172.17.0.0/16 to any port 6000:6099 + port 5432 (host PG). Ground truth: samograph rule 17.
GAP-3	Missing poolManager.dataSubDir/clonesMountSubDir	DBLab needs these to locate data/clone directories within the pool.	Added dataSubDir: "data", clonesMountSubDir: "clones"
GAP-4	Missing databaseContainer section	Clone containers need docker image + memory config.	Added databaseContainer.dockerImage + containerConfig
GAP-5	retrieval.spec.logicalDump missing dockerImage, containerConfig, dumpLocation, source.type: remote	Logical dump job won't execute without these.	Added all 4 matching samograph
GAP-6	retrieval.spec.logicalRestore/Snapshot had empty options: {}	Restore/snapshot containers need image config.	Added dockerImage, containerConfig, dumpLocation
GAP-7	Top-level containerConfig.containerResources section	Not in DBLab v4 schema; not in samograph.	Removed
FALSE-POS	toContain("selectedPool: dblab") in dblab-fixes-128.test.ts	String is a substring of "dblab_pool" → passes regardless of value.	Updated to quoted literal

TDD commits

• RED d180568 — 5 failing tests (3 × GAP-1 YAML-parsed, 2 × GAP-2 UFW regex); 19 existing tests still pass
• GREEN 6949dfc — all 1457 tests pass (0 fail)

Test file extended

test/dblab-schema-yaml-128.test.ts: two new describe blocks — GAP-1 (3 YAML-parsed exact assertions) + GAP-2 (5 UFW assertions including regression guards for fail2ban and control-plane SSH rule).