From 5a26b839ebd0f7ae84c89b792fadfc5d0ef93b41 Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Mon, 11 May 2026 10:14:57 -0400 Subject: [PATCH 1/3] docs: add index README and YAML frontmatter to every doc Add docs/README.md indexing every document in docs/ with a shared TYPE/CONNECTION vocabulary used consistently across all m-dev-tools repos. Add YAML frontmatter (created, last_modified, revisions, doc_type) to every existing doc. Existing frontmatter is merged, not replaced. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/README.md | 22 ++ docs/m-cli-integration-research.md | 351 +++++++++++++++++++ docs/m-engine-implementation-plan.md | 507 +++++++++++++++++++++++++++ 3 files changed, 880 insertions(+) create mode 100644 docs/README.md create mode 100644 docs/m-cli-integration-research.md create mode 100644 docs/m-engine-implementation-plan.md diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..9ef18d6 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,22 @@ +--- +created: 2026-05-11 +last_modified: 2026-05-11 +revisions: 0 +doc_type: [REFERENCE] +--- + +# m-test-engine — Documentation Index + +> First-pass index generated 2026-05-11. Labels follow the shared vocabulary below; the same vocabulary is used across all m-dev-tools repos. + +## Vocabulary + +Each doc is labeled `[TYPE · type? · connection · connection?]`. + +**Types** — `HISTORY` · `ARCHITECTURE` · `DESIGN` · `ADR` · `SPEC` · `REFERENCE` · `GUIDE` · `TUTORIAL` · `ROADMAP` · `PLAN` · `RESEARCH` · `SURVEY` · `GAP-ANALYSIS` · `STATUS` · `EXPLAINER` · `NOTES` · `WORKED-EXAMPLE` · `SETUP` · `INTEGRATION` · `PROPOSAL` · `BUILD-LOG` · `CHANGELOG` · `POSTMORTEM` + +**Repo connections** — `history` · `function` · `design` · `architecture` · `planning` · `implementation` + +## Top-level + +- **`m-cli-integration-research.md`** — `[RESEARCH · DESIGN · architecture · planning]` Pre-design synthesis tracing the m-cli/m-test-engine contract, discovery layers, manifest sketch, and `m engine` subcommand family. diff --git a/docs/m-cli-integration-research.md b/docs/m-cli-integration-research.md new file mode 100644 index 0000000..8666560 --- /dev/null +++ b/docs/m-cli-integration-research.md @@ -0,0 +1,351 @@ +--- +created: 2026-05-11 +last_modified: 2026-05-11 +revisions: 0 +doc_type: [RESEARCH, DESIGN] +--- + +# m-cli ↔ m-test-engine integration — background research + +**Status**: research / pre-design. No code committed. +**Audience**: m-test-engine maintainer + m-cli maintainer + future AI agents that bootstrap M projects. +**Source**: Synthesised from a design conversation that began with a user-run `m doctor` showing three unactionable warnings on a host with no local YottaDB. + +--- + +## 1. Statement of the problem + +`m doctor` is m-cli's environment self-check. On a fresh host without a locally-installed YottaDB it prints something like: + +``` + ! WARN ydb_dist not set + hint: Source your YottaDB env script (e.g. `source $ydb_dist/ydb_env_set`) or export `ydb_dist=/usr/local/lib/yottadb/r2.07`. + ! WARN ydb_routines not set + hint: Export `ydb_routines` to your routine search path, ... + ✓ OK parser tree-sitter-m loaded + ✓ OK keywords 323 M language keywords loaded from m-standard + ! WARN ydb_binary no `ydb` binary found + hint: Install YottaDB or set $YDB to the binary path. ... +``` + +Three problems compound here: + +1. **The hints are not actionable.** The first hint suggests `source $ydb_dist/ydb_env_set`, which is circular when `$ydb_dist` itself is not set. The example version `r2.07` is hard-coded and may not be current. No discovery is attempted. +2. **Three warnings describe one root cause** ("no YottaDB reachable") but are printed as independent failures, leaving the user to deduce the relationship. +3. **The Docker-first reality is invisible.** The canonical m-cli runtime path for users without root or YDB packaging support is the `m-test-engine` Docker container. `m doctor` currently checks only the locally-installed YottaDB path and does not mention the Docker path at all, even though m-cli's own engine layer (`src/m_cli/engine.py:170-220`, `DockerEngine`) already prefers it on hosts with no local YDB. + +The narrow ask was "make `m doctor` more actionable." The broader question this surfaces is: **what is the contract between m-cli and m-test-engine, and what artifacts would make every interaction along it concrete?** + +The conversation that followed traced that question through engine discovery, manifest design, CLI ergonomics, ecosystem precedents (Go, Rust, Python, Node, .NET, etc.), and AI-tooling discoverability. This document captures the reasoning and the candidate solutions for later design work. + +--- + +## 2. Concepts + +### 2.1 The two engines that matter + +| Engine | Class | When used | Status today | +|---|---|---|---| +| **LocalEngine** | locally-installed YottaDB | Linux servers with root, CI runners with apt access | Working; what `m doctor` checks today | +| **DockerEngine** (m-test-engine) | YottaDB inside a container | Mac developers, hosts without root, ephemeral CI, reproducible local dev | Working in code (`src/m_cli/engine.py:DockerEngine`), invisible to `m doctor` | +| SSHEngine | remote YottaDB over SSH | Legacy vista-meta maintainer workflow only | Working; kept for backward compat | + +The user's assertion was direct: *m-cli users will be using the Docker version of m (YottaDB) via the m-test-engine.* That makes DockerEngine the **canonical** runtime — not the fallback. `m doctor` and every actionable hint should be oriented around it. + +### 2.2 The current contract is two hardcoded strings + +The entire existing m-cli ↔ m-test-engine integration is: + +```python +# src/m_cli/engine.py:182-183 +container: str = "m-test-engine" +bind_root: Path = field(default_factory=lambda: Path("/work")) +``` + +Container name and bind mount path. That's the whole protocol. There is: + +- No declared image name or registry +- No declared compose file location +- No version handshake +- No healthcheck definition shared between sides +- No way for m-cli to introspect what's installed +- No way for m-cli to know "the image you have is older than the contract I speak" + +Every fixable warning depends on data m-cli does not currently have. + +### 2.3 Discovery surface — three layers + +To make `m doctor` actionable for Docker users and to enable an `m engine` subcommand family, m-test-engine should publish itself through three layered surfaces: + +**Layer 1 — Vendored manifest (`dist/m-test-engine.json`).** A static JSON file shipped *in the m-test-engine repo* and vendored into `m-cli/dist/m-test-engine.json` at m-cli release time. Sketch: + +```json +{ + "image": "ghcr.io/m-dev-tools/m-test-engine", + "default_tag": "latest", + "container": "m-test-engine", + "bind_mount": "/work", + "compose_file": "docker/compose.yaml", + "repo_url": "https://github.com/m-dev-tools/m-test-engine", + "min_docker": "20.10", + "ydb_version": "r2.02", + "protocol": 1 +} +``` + +This layer is the highest-leverage change. It works **before anything is installed**, so it powers the "no Docker, no image, no container" warning paths. + +**Layer 2 — OCI image labels.** Once the image is pulled, m-cli reads `docker image inspect`: + +```dockerfile +LABEL org.m-dev-tools.m-test-engine.protocol="1" +LABEL org.m-dev-tools.m-test-engine.bind-mount="/work" +LABEL org.m-dev-tools.m-test-engine.ydb-version="r2.02" +LABEL org.m-dev-tools.m-test-engine.image-rev="" +``` + +This layer enables **version-mismatch detection** — the image carries its own description; m-cli can warn "image speaks protocol 1, m-cli expects protocol 2, run `m engine upgrade`." + +**Layer 3 — Healthcheck + introspection entrypoint.** Two parts, both inside the container: + +- A Docker `HEALTHCHECK` directive in the Dockerfile, so `docker inspect --format '{{.State.Health.Status}}'` returns `starting | healthy | unhealthy` automatically. +- A small `mte` command on `$PATH` inside the container that prints structured JSON: `{"ok": true, "ydb_dist": "...", "uptime_s": 1234, "release": "r2.02", "globals_count": 17}`. + +This layer enables **continuous monitoring** (`m engine watch`) and a structured `m engine status --verbose`. + +### 2.4 Root-cause grouping in `m doctor` + +Independent of discovery, the diagnose output should collapse correlated failures. Today three warnings describe one cause. With discovery in place, the doctor output becomes a decision tree: + +``` +runtime engine: m-test-engine (Docker) + ✓ docker installed (28.5.1) + ✓ docker daemon reachable + ✓ user in docker group + ✓ image ghcr.io/m-dev-tools/m-test-engine:latest present + ✗ container m-test-engine not running + fix: m engine start + - (skipped) bind-mount /work → $PWD [needs container running] + - (skipped) ydb inside container [needs container running] + +source tools (fmt / lint / lsp): ready +runtime tools (test / coverage): blocked — see above + +alternative engines: + local yottadb: not installed (only if you want to skip Docker) + ssh (vista-meta): not configured (legacy, maintainer-only) +``` + +Each check declares prerequisites; if a prerequisite fails, downstream checks are reported as `SKIPPED — waiting on ` instead of running and exploding with a noisier secondary failure. + +### 2.5 The `m engine` subcommand family + +The natural shape, once the discovery layers exist: + +``` +m engine status # JSON or text: detected? running? healthy? +m engine install # docker pull +m engine start # docker run / docker start (smart) +m engine stop # docker stop +m engine restart # stop + start +m engine logs [--follow] # docker logs +m engine shell # docker exec -it ... bash +m engine exec '' # one-shot M command, exit code matters +m engine watch --interval 5 # long-running health poll (TAP/JSON lines) +m engine upgrade # pull newest tag → recreate container +m engine reset # destructive, --confirm required +m engine version # image label + release_name.txt +``` + +`m doctor` becomes a thin facade over `m engine status` for the runtime section, and every WARN line in doctor output points at the *exact* `m engine ` that fixes it — not free-form prose. + +--- + +## 3. Comparisons with other language CLIs + +The question "where does runtime/engine management belong?" has been answered differently across ecosystems. The pattern split is informative. + +| CLI | Build/test in core | Runtime/engine mgmt | Pattern | +|---|---|---|---| +| **Go** (`go`) | yes | `go tool ` (curated, in-tree) | Monolithic with namespaced extensions; no third-party `go` subcommands | +| **Rust** (`cargo` + `rustup`) | yes (`cargo`) | separate binary (`rustup`) | Split: build tool vs. toolchain management | +| **.NET** (`dotnet`) | yes | `dotnet-install.sh` external; `dotnet tool install` for tools | SDK install outside; tools pluginable | +| **Elixir** (`mix`) | yes | external (`asdf-erlang`, `kerl`) | Build in core, BEAM management separate | +| **Node** (`npm`/`pnpm`) | yes | `nvm`/`volta`/`asdf` external; `pnpm env use` recent exception | Historically separate, slowly converging | +| **Python** (`uv`) | yes | `uv python install` built-in | Modern trend: bundle into one binary | +| **Deno** (`deno`) | yes | `deno upgrade` built-in | One binary, self-managed | +| **Bun** (`bun`) | yes | `bun upgrade` built-in | One binary, self-managed | +| **Docker** | — | `docker compose` ships as plugin alongside `docker` | Plugin model, namespaced subcommand | + +**Two clusters emerge:** + +- **Toolchains touched once per developer-decade** (Rust, Erlang, .NET SDK) → separate management CLI. The friction of `rustup` is amortized over years. +- **Runtimes touched per-project or per-CI-run** (uv's Python, deno, bun) → bundled into the primary CLI. Friction matters because the operation is frequent. + +**Where m-test-engine sits.** It is not a once-per-decade toolchain. It will be started, restarted, debugged, pulled-fresh, and reset constantly during normal m-cli use — installs on a new dev's laptop, container recreations after image upgrades, CI environment provisioning, ad-hoc shells for debugging. That places it firmly in the `uv` / `deno` / `bun` bucket: **bundle into the primary CLI.** + +The `cargo`/`rustup` split would only fit if M development meant "set up YDB once, forget about it." The whole point of the Docker engine, conversely, is that setup *is* the recurring friction — and the recurring friction is exactly what a first-class subcommand should absorb. + +--- + +## 4. Proposed options and solutions + +### 4.1 Where does `m engine` live? — core vs. plugin + +**Option A — Core subcommand.** +Engine management ships in m-cli core, alongside `fmt`, `lint`, `test`, `doctor`. The built-in Docker driver targets m-test-engine. + +| Pros | Cons | +|---|---| +| `m doctor --fix` and `m engine ` live together — every WARN points to a real, always-present command | Pushes m-cli further into "runtime CLI" territory (it started as source-level tooling) | +| Discoverable: beginners see `m engine` in `m -h` with zero extra installs | Adds a Docker-shell-out namespace that is mostly thin wrappers | +| Engine protocol version tracked in lockstep with the rest of core | Tests need subprocess stubbing for `docker` calls | +| Cross-platform install hints reuse existing platform detection | If alternative engines (IRIS, podman) emerge, core grows | +| The manifest (`dist/m-test-engine.json`) already wants to live alongside `dist/repo.meta.json` | — | + +**Option B — Plugin (m-cli-extras-style).** +Engine management ships as an out-of-tree subcommand discovered via the `m_cli.plugins` entry-point group. + +| Pros | Cons | +|---|---| +| Keeps core minimal and source-tool-focused | Chicken-and-egg: `m doctor` says "run `m engine install`" → "command not found" → user has to know about plugins | +| Multiple engine plugins can coexist (`m-cli-docker-engine`, `m-cli-iris-engine`, `m-cli-podman-engine`) | Plugin API (`PLUGIN_API_VERSION=1`, `src/m_cli/plugins.py`) does not yet expose hooks for `m doctor` to delegate fixes | +| Iterates independently of m-cli releases | Splits documentation, install instructions, version compatibility | +| Mirrors the m-cli ↔ m-test-engine repo boundary at the package layer | Real users hit a worse first-run experience | + +**Recommended hybrid.** +Put **`m engine` in core** with a thin built-in **Docker driver**, and define an `m_cli.engine_drivers` entry-point group so out-of-tree drivers can register: + +```python +class DockerDriver(EngineDriver): + name = "docker" + def install(self): ... + def start(self): ... + def status(self) -> Status: ... + +# Out-of-tree: pip install m-cli-iris-engine +class IrisDriver(EngineDriver): + name = "iris" + ... +``` + +This mirrors the `cargo` + `cargo-foo` plugin pattern, scoped to drivers instead of arbitrary subcommands. Keeps discoverable-by-default behaviour while leaving room for non-Docker engines without forking core. + +### 4.2 Impact on AI / agent discoverability + +m-cli is already unusually agent-friendly. It ships `dist/commands.json` (from `m capabilities --json`), `dist/repo.meta.json`, `dist/lint-rules.json`, `dist/fmt-rules.json`, and an `m capabilities` subcommand specifically so coding agents have a single machine-readable surface. Putting `m engine` in core plugs into that surface for free; the plugin route bypasses it. + +**Existing AI-readable surfaces:** + +| Artifact | Who reads it | What it answers | +|---|---|---| +| `dist/commands.json` | Agents, IDE plugins | "What subcommands and flags exist?" | +| `dist/repo.meta.json` | Org-wide AI tooling | "What's the verification contract for this repo?" | +| `dist/lint-rules.json` / `dist/fmt-rules.json` | Agents, doctors, LSP | "What rules can fire? What are their IDs/severity?" | +| `m capabilities --json` | Live introspection | Same as commands.json, but runtime-current | +| `m doctor --json` | Agents reasoning about env | Structured health, not free-form prose | +| `CLAUDE.md` / `AGENTS.md` | LLM session context | Project conventions, layout, dos and don'ts | + +**What changes when `m engine` is core:** + +1. **It auto-appears in `dist/commands.json`.** No agent change required. Plugin route is invisible until `pip install m-cli-extras`; a fresh-checkout agent would never propose `m engine install` because it cannot know the command can exist. + +2. **`m doctor --json` gains structured fix verbs.** Free-form `hint` strings require an LLM round-trip to interpret. Typed fields are agent-actionable: + + ```json + { + "name": "engine", + "status": "WARN", + "message": "m-test-engine container not running", + "fix": { "command": ["m", "engine", "start"], "destructive": false } + } + ``` + +3. **Error → action edges become introspectable.** `EngineNotConfigured` in `src/m_cli/engine.py:399` can carry a `recovery: ["m", "engine", "install"]` field. Agents walk failure-mode → command → verify as a state machine instead of pattern-matching prose. + +4. **`dist/m-test-engine.json` joins the `dist/` namespace.** Tier-1 contract for the engine, sibling to `dist/repo.meta.json`. AI tooling that already trusts `dist/` gets engine info without learning a new lookup. + +5. **CLAUDE.md / AGENTS.md scales naturally.** Adding `m engine` to the existing "Engine support" section is one paragraph. Plugin route requires either a new "Optional plugins" section that agents skim past, or a pre-flight `pip install` that breaks reproducibility. + +6. **Bootstrap recipes become four lines.** Agents seeing a fresh `m new` project can do, deterministically: + + ```bash + m engine install + m engine start + m test + ``` + + …and verify each step from JSON exit data. With a plugin, that recipe acquires an "if-the-plugin-is-installed-then" branch LLMs handle poorly. + +7. **MCP servers / IDE integrations get one source of truth.** Any future m-cli MCP server, or `tree-sitter-m-vscode`'s command palette, walks `dist/commands.json`. Core commands light up automatically. + +8. **`m doctor --fix` becomes safe for autonomous execution.** Because the fix verbs are core, well-typed, and idempotent, an autonomous agent can run `m doctor --fix` without prompting the user — the fix surface is bounded. + +9. **Skill generation from manifest.** `~/claude/skills/m-stdlib/` is already auto-generated from m-stdlib's manifest. The same pattern applies: a `~/claude/skills/m-engine/` skill could be generated from `dist/m-test-engine.json` + the engine slice of `commands.json`, auto-loading in any Claude Code session that touches an M project. + +**Honest costs of the core path for AI discovery:** + +- `dist/commands.json` grows by a few KB. Mitigation: agents can `jq '.commands.engine'` for a slice. +- The capabilities surface has to track Docker's semantics. When `docker compose v3` shifts, m-cli chases. Plugin would absorb this independently. + +**Concrete additions to capture the AI win:** + +1. **Add `recovery` edges to `m doctor`'s JSON schema** — `{ "fix": { "command": [...], "verify": [...] } }` per failing check. +2. **Add `m engine capabilities --json`** mirroring the top-level capabilities pattern, so the engine namespace is independently inspectable. +3. **Add a `verbs` section to `dist/m-test-engine.json`** declaring which `m engine ` commands are safe for autonomous execution vs. require `--confirm` (e.g. `reset` is destructive, `status` is read-only). Lets agent harnesses gate the destructive ones at the policy layer. + +### 4.3 Staging proposal + +A phased rollout that ships value at every step and decouples the m-test-engine and m-cli changes wherever possible: + +1. **Phase 1 — Vendored manifest only.** Add `dist/m-test-engine.json` to both repos. Rewrite `m doctor` hints to consume it. No Docker image changes. Concrete `docker pull` / `docker run` strings appear in every WARN. ~1–2 days. **Single highest-leverage change.** + +2. **Phase 2 — `m engine` subcommand family in core.** Skeleton subcommand wrapping `docker` calls, driven entirely by the vendored manifest. `m doctor --fix` delegates to it. Adds the `EngineDriver` protocol and the entry-point group for out-of-tree drivers. ~3–5 days. + +3. **Phase 3 — OCI labels + `HEALTHCHECK` (m-test-engine side).** Enables `m engine status` to report `healthy/unhealthy`, version-mismatch detection, and `m engine watch` cadence based on the native healthcheck. + +4. **Phase 4 — `mte` introspection command (m-test-engine side).** Enables `m engine status --verbose` with structured globals/uptime/release info. Unlocks richer `m doctor --json` payloads for agents. + +5. **Phase 5 — Skill / MCP integration.** Auto-generated `~/claude/skills/m-engine/` from `dist/m-test-engine.json`. Optional `m-cli`-MCP server entry that exposes engine verbs to Claude Code natively. + +Phase 1 is independent of every later phase and is the single change with the highest signal-to-noise return. Phases 3–5 are non-blocking once Phase 1's manifest ships. + +--- + +## 5. Open questions + +These were raised in the conversation and remain unresolved: + +1. **Where does `dist/m-test-engine.json` live as source of truth?** + - Option A — m-test-engine repo, vendored into m-cli at release time (simpler; m-cli releases on its own cadence). + - Option B — separate `m-test-engine-meta` Python package (decouples cadences; m-test-engine releases bump the contract independently). + +2. **Canonical image registry.** + - `ghcr.io/m-dev-tools/m-test-engine` is the natural default given the GitHub org, but no image has been published yet. Confirm the registry + naming convention before vendoring. + +3. **Compose vs. `docker run`.** + - `m engine start` could shell out to `docker compose -f ` or to plain `docker run`. Compose is more declarative; `docker run` has fewer prerequisites. Decide which the manifest's `start_cmd` should point at. + +4. **Bind-mount semantics for monorepos.** + - Today `bind_root = /work` assumes one project per container. For developers working across multiple m-* repos in one session, do we accept "one container per `cd`" or build multi-mount support into the manifest? + +5. **Protocol version bump policy.** + - When does a change to the contract require a `protocol` bump vs. a minor field addition? Need a written policy before the first paying consumer (`m doctor`) starts asserting on it. + +6. **`EngineDriver` entry-point name.** + - `m_cli.engine_drivers` is one candidate; `m_cli.engines` is shorter. Lock the name before publishing a Phase 2 plugin API since it becomes part of `PLUGIN_API_VERSION`'s contract. + +--- + +## 6. Summary + +The thread of reasoning, compressed: + +1. `m doctor` is unactionable for the Docker-first majority of m-cli users because it has no model of the Docker engine path. +2. The fix is not better prose hints — it is **publishing the m-cli ↔ m-test-engine contract as machine-readable artifacts** that doctor, an `m engine` subcommand, and AI agents all consume from the same source. +3. Three discovery layers (vendored manifest, OCI labels, container-side introspection) cover the no-install, post-install, and runtime states respectively. +4. An **`m engine` subcommand family in core** absorbs the recurring friction of managing the container. The Rust `cargo`/`rustup` split fits long-lived toolchains; the `uv` / `deno` / `bun` "bundle the runtime" pattern fits frequently-touched runtimes — and m-test-engine is firmly in the latter group. +5. Putting `m engine` in core extends m-cli's existing manifest-driven, machine-readable, agent-bootable stance to runtime management. The plugin route works for humans but creates a hidden seam that AI tooling reliably trips over. +6. **Phase 1 (vendored manifest + rewritten doctor hints)** is the smallest first move with the largest delta and is independent of every later phase. + +The single decision that unlocks every subsequent improvement is the publication of `dist/m-test-engine.json`. Everything downstream — doctor actionability, the `m engine` subcommand, agent bootstrapping, version handshaking — depends on that one file existing. diff --git a/docs/m-engine-implementation-plan.md b/docs/m-engine-implementation-plan.md new file mode 100644 index 0000000..22f1cbc --- /dev/null +++ b/docs/m-engine-implementation-plan.md @@ -0,0 +1,507 @@ +--- +created: 2026-05-11 +last_modified: 2026-05-11 +revisions: 0 +doc_type: [PLAN, DESIGN] +supersedes: none +informs: m-cli-integration-research.md +--- + +# `m engine` — implementation plan + +**Status**: design-approved, ready for Phase 1 implementation. +**Scope**: the `m engine` subcommand family in m-cli core, the +`dist/m-test-engine.json` manifest published by this repo, and the +companion changes to m-cli's `m doctor` that consume it. +**Predecessor**: [`docs/m-cli-integration-research.md`](m-cli-integration-research.md) +captures the rationale, ecosystem comparisons, and option analysis that +led to the decisions below. This plan does not restate that material; it +records the *what* and *when*, with the decisions section pinning every +open question. + +--- + +## 1. Decisions on open questions + +Each entry below corresponds to a numbered open question in +`m-cli-integration-research.md §5`. Recording them here makes the plan +self-contained and audit-friendly for future contributors. + +### 1.1 Source of truth for `dist/m-test-engine.json` — **Option A** + +The manifest is authored and versioned in **this repo** (m-test-engine) +and vendored into `m-cli/dist/m-test-engine.json` at m-cli release time. + +- **Why**: m-cli already has a vendoring discipline (`dist/repo.meta.json`, + `dist/commands.json`, etc.); reusing the same pattern avoids inventing + a second metadata-package release pipeline. m-cli releases ride on + their own cadence and pull in the latest published manifest. +- **Mechanic**: m-cli's `make manifest` target gains a step that copies + `m-test-engine/dist/m-test-engine.json` from a pinned tag (recorded in + m-cli's lockfile or `dist/repo.meta.json` `dependencies` block). No + network fetch at runtime — vendoring is a build-time artifact. +- **Drift gate**: m-cli's `make check-manifest` asserts the vendored copy + byte-matches the pinned upstream tag. + +### 1.2 Canonical image registry — **`ghcr.io/m-dev-tools/m-test-engine`** + +Confirmed. GHCR matches the org domain on GitHub, supports anonymous +pulls, and inherits org-level access control. No separate Docker Hub +account or rate-limit footprint to manage. + +- First published tag: `:r2.02` (matches the current `ydb_version` in + the manifest sketch). +- Floating tag: `:latest` tracks the most recent stable r-release. +- Multi-arch: linux/amd64 only initially; arm64 added when Mac-on-arm + consumers materialise (build matrix already supports it via + `docker buildx`). + +### 1.3 Compose vs. `docker run` — **compose-first, run-fallback** + +**Decision**: `m engine start` shells out to `docker compose -f ` +as the primary path. Plain `docker run` is the documented fallback for +hosts without the compose plugin, constructible deterministically from +the manifest fields. + +**Pros/cons that drove the call** — the user's stated priorities were +simplicity, maintainability, and minimal drift as Docker evolves: + +| Aspect | `docker compose` | `docker run` | +|---|---|---| +| Declarativeness | One reviewable `compose.yml` file; diffs read like config | Configuration lives in code (flags assembled per call) | +| Drift over time | Compose v2 schema is stable; Docker has committed to it long-term (v1 was retired 2023) | CLI flags are the most stable Docker surface — ~10 years backward-compatible | +| Set-and-forget | Yes — edit the file, restart the container, done | Partially — flag changes touch m-cli code | +| Dependency surface | Requires `docker compose` plugin (bundled with modern Docker since 2022) | Only requires `docker` CLI | +| Multi-container readiness | Trivial (add a service) | Manual orchestration | +| Maintainability across the m-* repos | The compose file lives in m-test-engine and every consumer points at the same one | Every consumer assembles its own flag list — divergence risk | +| Failure mode visibility | Compose surfaces healthchecks, depends_on, restart-policy out of the box | Has to be re-implemented per call site | + +**Why compose wins for this project**: m-test-engine already ships +`compose.yml` as its canonical contract; pointing every consumer at that +same file means the configuration lives in *one* place and edits +propagate without coordinated multi-repo changes. Compose v2 is now +shipped *as part of* Docker Engine and Docker Desktop, so the +"prerequisite" cost is approximately zero on any host that has Docker +in the first place. Compose's schema has been remarkably stable since +the v2 rewrite (2022); the deprecations that do happen (e.g. the +`version:` top-level field) are non-breaking warnings. + +**Why `docker run` stays in the fallback slot**: minimal CI runners and +older Linux distros sometimes ship Docker Engine without the compose +plugin. The manifest carries enough fields (`image`, `container`, +`bind_mount`, env vars) to reconstruct an equivalent `docker run` +invocation; m-cli detects compose-plugin absence and falls back +transparently. + +**Set-and-forget guarantee**: the manifest declares both +`compose_file: docker/compose.yml` and a `run_args` block. m-cli prefers +compose; on `docker compose version` failure it constructs the +equivalent `docker run` from `run_args` and the rest of the manifest. +Either path produces an identically-named, identically-mounted +container. + +### 1.4 Bind-mount semantics — **shared host `/m-work` directory** + +**Decision**: a single, shared host directory at `/m-work` is bind-mounted +into the container at `/m-work`. All m-* repos that participate (m-cli, +m-stdlib, m-test-engine itself, future m-* projects) are checked out or +symlinked under `/m-work/`, e.g.: + +``` +/m-work/ +├── m-cli/ +├── m-stdlib/ +├── m-test-engine/ +├── m-modern-corpus/ +└── ... +``` + +The container sees the same layout under `/m-work/`. `ydb_routines` is +configured (inside the container) to include the relevant routine +subdirs across all participating repos, so routines from m-stdlib are +callable from m-cli tests without re-mounting or restarting. + +- **Why this shape**: every m-* repo provides distinct capabilities + (m-stdlib provides `^STDASSERT` / `^STDJSON` / `^STDREGEX`; m-cli + provides linting/formatting/runner; m-modern-corpus provides + calibration M source). They must coexist in the running engine to be + useful. A per-cwd `/work` mount silos them and forces "one container + per project" — which contradicts the canonical-runtime model. +- **Manifest field**: + ```json + "bind_mount": { + "host": "/m-work", + "container": "/m-work", + "mode": "rw" + } + ``` + (was: `"bind_mount": "/work"` — a single string. Promoted to an object + to carry host/container/mode.) +- **`m engine start` precondition**: `/m-work` must exist on the host. + If absent, m-cli prints an actionable hint: + ``` + ✗ host directory /m-work does not exist + fix: sudo install -d -o $USER -g $USER /m-work + cd /m-work && git clone https://github.com/m-dev-tools/m-cli + cd /m-work && git clone https://github.com/m-dev-tools/m-stdlib + ``` +- **m-cli implications**: m-cli's `engine.py` `DockerEngine` constructor + loses its per-instance `bind_root` arg in favour of the manifest's + shared mount. Engine discovery (`detect_engine`) becomes a singleton + per host, not per cwd. +- **Migration note for existing dev setups**: anyone with a working + `/work`-mounted setup needs a one-time move to `/m-work`. m-cli's + `m doctor` detects the legacy mount and emits a migration hint + (`✗ legacy /work mount detected — see docs/migration-to-m-work.md`). + +### 1.5 Protocol version bump policy — **semver-style, with explicit rules** + +The `protocol` field in `dist/m-test-engine.json` is a single integer +that m-cli treats as a compatibility handshake. Question 5 in the +research doc was left open ("advise on impact"). Here is the +recommendation and the policy that follows from it. + +**Impact of getting the policy wrong**: + +- **Bumping too aggressively** — every minor manifest change forces + every consumer to upgrade. `m doctor` starts firing + "protocol mismatch" warnings during normal release cycles, users + develop alarm-fatigue, and the field becomes ignored noise. +- **Bumping too conservatively** — silent contract drift. A field's + semantics change but the protocol number doesn't move, so m-cli keeps + using the old interpretation and behaves wrongly. This is the more + dangerous failure mode because it manifests as inscrutable bugs + rather than visible warnings. + +**Policy** (additive-by-default, strict on semantics): + +| Change | Bump `protocol`? | +|---|---| +| New **optional** field added | No | +| New **required** field added | Yes | +| Field renamed | Yes | +| Field removed | Yes | +| Field's *type* changes (string → object, etc.) | Yes | +| Field's *semantics* change (same name, new meaning) | Yes | +| Default value of an optional field changes | No (document in release notes) | +| New enum value added to an existing enum field | No, provided consumers tolerate unknown values | +| Enum value removed or repurposed | Yes | +| Documentation / comment / typo fix | No | + +**Consumer rules** (m-cli, future drivers): + +- m-cli **must tolerate unknown fields** in the manifest. Future + additive evolution stays unblocked. +- m-cli **must reject** a manifest whose `protocol` is *higher* than the + highest version it understands, with a clear "upgrade m-cli" hint. +- m-cli **may warn** when `protocol` is *lower* than expected (consumer + is newer than the manifest); behaviour is best-effort. + +**Expected cadence**: bumps are rare. Realistic expectation is one bump +per 12–24 months. Most evolution will be additive. + +**Initial state**: `protocol: 1` ships with Phase 1. + +### 1.6 `EngineDriver` entry-point group name — **`m_cli_engines`** + +Confirmed. Short, consistent with `m_cli.plugins` (the existing +entry-point group name), reads naturally as "m-cli engines". + +- Underscore-separated to match Python entry-point conventions. +- Locked as part of `PLUGIN_API_VERSION = 1` once Phase 2 ships. + +--- + +## 2. Phased rollout + +The research doc proposed five phases; this plan keeps that shape but +specifies the exit criteria, owners, and the cross-repo coordination +required for each. + +### Phase 1 — vendored manifest + actionable `m doctor` + +**Goal**: ship the manifest from this repo, vendor it into m-cli, and +rewrite `m doctor`'s Docker-path hints to consume it. No new +subcommands, no Docker image changes. + +**Deliverables in m-test-engine**: + +- `dist/m-test-engine.json` — hand-authored, validated against a JSON + Schema at `dist/m-test-engine.schema.json`. Fields exactly as decided + above (`image`, `default_tag`, `container`, `bind_mount` object, + `compose_file`, `repo_url`, `min_docker`, `ydb_version`, `protocol`, + `run_args`). +- `make check-manifest` — schema-validates `dist/m-test-engine.json`, + asserts `verified_on` is within 90 days, and asserts the referenced + `compose_file` path exists. +- README pointer to the manifest as the public machine-readable contract. + +**Deliverables in m-cli**: + +- `dist/m-test-engine.json` vendored from this repo at a pinned tag. +- `m doctor` rewritten so every WARN in the Docker engine path quotes + the exact `docker pull` / `docker compose -f up -d` / + `docker exec m-test-engine ...` command derived from the manifest. +- `m doctor --json` schema extended with `fix.command: [...]` and + `fix.destructive: bool` per check (lays groundwork for autonomous + agents). +- Root-cause grouping: prerequisite-failed checks downstream report + `SKIPPED` rather than running and producing secondary failures. + +**Exit criteria**: + +- `m doctor` on a fresh Mac with Docker installed and no m-test-engine + pulled prints a four-line fix recipe that, when run verbatim, + resolves every WARN. +- `m doctor --json` validates against the new schema. +- m-cli's `make check-manifest` catches drift from the upstream + manifest. + +**Duration**: 1–2 days of focused work. Phase 1 is independent of every +later phase and is the single highest-leverage delivery. + +--- + +### Phase 2 — `m engine` subcommand family in m-cli core + +**Goal**: turn the WARN hints from Phase 1 into commands that actually +exist. `m doctor --fix` becomes safe and idempotent. + +**Deliverables in m-cli**: + +- New subcommand tree under `src/m_cli/engine/`: + - `m engine status` (text + `--json`) + - `m engine install` + - `m engine start` + - `m engine stop` / `restart` + - `m engine logs [--follow]` + - `m engine shell` + - `m engine exec ''` + - `m engine version` + - `m engine upgrade` + - `m engine reset --confirm` (destructive, opt-in) + - `m engine capabilities --json` (mirrors top-level `m capabilities`) +- `EngineDriver` protocol exported as a public API; built-in + `DockerDriver` is the only registered driver. +- `m_cli_engines` Python entry-point group declared and documented; no + out-of-tree drivers yet but the seam exists. +- `m doctor --fix` delegates to `m engine ` for every fixable + WARN; refuses to run destructive verbs without explicit `--confirm`. +- `dist/commands.json` auto-grows to include the `engine` namespace + (downstream agents pick it up for free). + +**Exit criteria**: + +- `m engine status --json` is the canonical health check; `m doctor`'s + runtime section becomes a thin facade over it. +- All `m engine ` calls construct their `docker` / `docker compose` + invocations from the manifest — no hard-coded image names or paths in + Python. +- `m doctor --fix` on a fresh Mac with Docker installed runs to a green + state without manual intervention. + +**Duration**: 3–5 days. + +--- + +### Phase 3 — OCI labels + `HEALTHCHECK` (m-test-engine side) + +**Goal**: make the image self-describing once pulled, so m-cli can do +version-mismatch detection and `m engine status` can report real Docker +health. + +**Deliverables in m-test-engine**: + +- `Dockerfile` adds: + ```dockerfile + LABEL org.m-dev-tools.m-test-engine.protocol="1" + LABEL org.m-dev-tools.m-test-engine.bind-mount="/m-work" + LABEL org.m-dev-tools.m-test-engine.ydb-version="r2.02" + LABEL org.m-dev-tools.m-test-engine.image-rev="" + HEALTHCHECK CMD $ydb_dist/mumps -run %XCMD 'write "ok",!' || exit 1 + ``` +- `make smoke` extended to verify the label set and healthcheck + presence. +- Release process documents the `image-rev` propagation + (`docker buildx --build-arg GIT_SHA=$(git rev-parse HEAD)`). + +**Deliverables in m-cli**: + +- `m engine status` reads `docker image inspect` and surfaces + `protocol_mismatch` / `image_outdated` warnings derived from label + comparisons against the vendored manifest. +- `m engine version` prints both the manifest-declared expectation and + the image-reported actual. + +**Exit criteria**: + +- An intentionally-mismatched image (older tag pulled, newer manifest + vendored) produces a clear "run `m engine upgrade`" WARN. +- `docker inspect --format '{{.State.Health.Status}}'` returns + `healthy` after `m engine start` completes. + +**Duration**: 1–2 days, mostly on the m-test-engine side; m-cli side is +small once Phase 2's `status` infrastructure is in place. + +--- + +### Phase 4 — `mte` container-side introspection + +**Goal**: structured, rich introspection from inside the container, so +`m engine status --verbose` reports more than just "running / healthy". + +**Deliverables in m-test-engine**: + +- `mte` shell script (or compact M routine) on `$PATH` inside the + container. `mte status --json` prints: + ```json + { + "ok": true, + "ydb_dist": "/opt/yottadb/r2.02", + "release": "r2.02", + "uptime_s": 1234, + "globals_count": 17, + "routines_count": 412, + "mounted_repos": ["m-cli", "m-stdlib", "m-modern-corpus"] + } + ``` +- Tests in `make smoke` assert `mte status --json` produces valid JSON. + +**Deliverables in m-cli**: + +- `m engine status --verbose` runs `docker exec m-test-engine mte status + --json` and folds the output into its report. +- `m engine watch --interval 5s` streams `mte status --json` lines for + live monitoring (TAP-like format for CI; JSON-lines for tools). + +**Exit criteria**: + +- `m engine status --verbose` on a healthy container shows mounted + repos, routine count, uptime — answering "is the engine ready for + *my* repo's tests?" not just "is it up?". + +**Duration**: 2–3 days. + +--- + +### Phase 5 — Skill / MCP integration + +**Goal**: extend the existing manifest-driven AI-discoverability stance +to the engine namespace, so Claude Code and other agents bootstrap m-* +projects without bespoke instructions. + +**Deliverables**: + +- Auto-generated `~/claude/skills/m-engine/SKILL.md` driven by + `dist/m-test-engine.json` + the `engine` slice of + m-cli's `dist/commands.json` (`make skill-install` target in this + repo, mirroring m-stdlib's existing pattern). +- `dist/m-test-engine.json` gains a `verbs` section declaring which + `m engine ` commands are safe for autonomous execution vs. + require `--confirm`: + ```json + "verbs": { + "status": { "destructive": false, "read_only": true }, + "start": { "destructive": false, "read_only": false }, + "reset": { "destructive": true, "requires_confirm": true }, + ... + } + ``` +- Optional: m-cli MCP server registers the safe verbs as MCP tools so + Claude Code can drive the engine natively without shelling out. + +**Exit criteria**: + +- A fresh Claude Code session in any m-* repo auto-loads the m-engine + skill and offers `m engine install` / `start` / `status` as actions. +- The verb-safety classification gates destructive operations at the + agent-harness layer, not at human-prose-warning layer. + +**Duration**: 2–3 days, parallelisable with Phase 4. + +--- + +## 3. Risks and mitigations + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| Docker compose v2 schema deprecation breaks `compose.yml` mid-cycle | Low | Medium | The `run_args` fallback in the manifest reconstructs `docker run`. Schema deprecations in v2 have been non-breaking warnings; we'd notice via `make smoke` long before users do. | +| `/m-work` migration friction for existing `/work`-mounted devs | Medium | Low | `m doctor` detects legacy `/work`, prints a one-step `mv` / re-symlink hint. Document in `docs/migration-to-m-work.md`. Only relevant for current maintainers; new users land directly on `/m-work`. | +| Manifest drift between m-test-engine and m-cli's vendored copy | Medium | High | m-cli's `make check-manifest` byte-compares against the pinned upstream tag; CI gate. Vendoring pin recorded in m-cli's `dist/repo.meta.json` `dependencies` block. | +| GHCR rate-limits or outages | Low | Medium | Anonymous pulls are 1000/hr per IP — well above realistic dev usage. For CI, document GHCR token auth as an opt-in. | +| Protocol bump churn surprises users | Low | Medium | Policy in §1.5 is conservative-by-default; expected cadence is 12–24 months. Every bump documented in `CHANGELOG.md` with a migration recipe. | +| m-cli grows into "yet another Docker orchestrator" | Medium | Medium | Scope discipline: `m engine` shells out to `docker` / `docker compose`; it does not reimplement them. Anything beyond start/stop/exec/status belongs in compose, not in Python. The `EngineDriver` seam keeps the door open for non-Docker engines without bloating core. | +| `mte` introspection script leaks YDB internals or PII | Low | Low | Output is structured JSON with a fixed allowlist (no `$ZGBLDIR`, no env dump). Phase 4 ships with a schema for `mte status --json` and tests pin the field set. | +| Bind-mount of host `/m-work` exposes too much filesystem to the container | Low | Low | `/m-work` is a user-controlled directory containing only m-* repos. Mount mode is `rw` (consumers need to write build artifacts). Document the security model in README. | + +--- + +## 4. Benefits realised + +Mapping back to the research doc's framing — what does the +status-quo unblock once each phase lands? + +| Benefit | Phase that delivers it | +|---|---| +| `m doctor` produces actionable, copy-pasteable fix recipes | 1 | +| AI agents bootstrap m-* projects from `dist/commands.json` alone | 1 (manifest) + 2 (`m engine` in `commands.json`) | +| Version-mismatch detection between image and m-cli | 3 | +| Single shared engine across all m-* repos via `/m-work` | 1 (manifest) + 2 (start command) | +| `m doctor --fix` autonomous-execution safe | 2 (typed fixes) + 5 (verb safety classes) | +| Continuous health monitoring (`m engine watch`) | 4 | +| Out-of-tree engines (IRIS, podman) without forking core | 2 (`m_cli_engines` entry point) | + +--- + +## 5. Cross-repo coordination + +Phase ordering reflects dependency between this repo and m-cli: + +``` +this repo (m-test-engine) m-cli +───────────────────────── ───── +Phase 1a: ship manifest ───► Phase 1b: vendor + rewrite doctor + Phase 2: m engine subcommand family +Phase 3a: labels + healthcheck ─► Phase 3b: status reads labels +Phase 4a: mte introspection ───► Phase 4b: status --verbose + Phase 5: skill + MCP +``` + +Phase 1a (this repo) is the only blocker for Phase 1b (m-cli). After +that, m-cli can iterate independently through Phase 2 without further +changes here. Phases 3 and 4 require small coordinated bumps but neither +breaks any earlier deliverable. + +--- + +## 6. Out of scope + +Explicitly **not** part of this plan: + +- IRIS engine support (the `m_cli_engines` entry-point seam admits it + later, but no IRIS driver ships in core). +- Podman as a Docker drop-in (same — seam exists, driver doesn't). +- Multi-arch image (arm64) — added when arm64 consumers materialise. +- SSH transport changes — `SSHEngine` remains the legacy maintainer + path; not modified by this plan. +- VistA-specific extras inside the container (no FileMan, no Kernel — + m-test-engine's existing guardrail stands). +- m-cli replacing `docker` / `docker compose` with a Python Docker SDK. + Shell-out keeps the dependency surface minimal and the behaviour + trivially auditable. + +--- + +## 7. Success metric + +A new contributor on a fresh laptop, after `git clone m-cli`, runs: + +```bash +m doctor --fix +m test +``` + +…and sees a green test suite without reading any documentation, without +manually pulling images, and without setting environment variables. That +is the bar Phase 2 must clear. Every phase before contributes to it; +every phase after polishes it for agents. From e88e960c4504950dc54d73fd863d4d87839baca3 Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Mon, 11 May 2026 11:06:47 -0400 Subject: [PATCH 2/3] engine-contract: publish dist/m-test-engine.json (Phase 1a) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ships the machine-readable m-cli <-> m-test-engine contract decided in docs/m-engine-implementation-plan.md so m-cli's `m doctor` and the upcoming `m engine` subcommand family have a single, version-handshaked surface to read from. * dist/m-test-engine.json — protocol 1, GHCR image ref, bind_mount object (host=/m-work, container=/m-work, mode=rw), compose-first + run_args fallback per plan §1.3 * dist/m-test-engine.schema.json — JSON Schema 2020-12, additive-only * tools/check-manifest.py — extended to validate both manifests: schema-shape, compose_file resolves on disk, verified_on within 90d * dist/repo.meta.json — enrolled engine_contract in exposes; bumped verified_on to 2026-05-11 * README.md — new "Machine-readable contract" section pointing consumers at the manifest + schema + drift gate * docs/m-engine-tracker.md — live progress tracker; locks the plan doc once stages move off `todo` Notes / followups recorded in tracker narrative log: - /m-work declared in manifest is target state; compose.yml and Dockerfile still bind /work — runtime cutover deferred to Phase 2 - ghcr.io image not yet published; Phase 3 covers the publish workflow Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 21 +++ dist/m-test-engine.json | 33 +++++ dist/m-test-engine.schema.json | 130 +++++++++++++++++++ dist/repo.meta.json | 9 +- docs/m-engine-tracker.md | 231 +++++++++++++++++++++++++++++++++ tools/check-manifest.py | 154 ++++++++++++++++++---- 6 files changed, 547 insertions(+), 31 deletions(-) create mode 100644 dist/m-test-engine.json create mode 100644 dist/m-test-engine.schema.json create mode 100644 docs/m-engine-tracker.md diff --git a/README.md b/README.md index 5a1a03f..801f0a4 100644 --- a/README.md +++ b/README.md @@ -76,6 +76,27 @@ filesystem is directly visible inside the container. `m-stdlib`'s test runner uses the same transport once Track A3 of the [m-dev-tools-todo plan](../m-dev-tools-todo.md) lands. +## Machine-readable contract + +[`dist/m-test-engine.json`](dist/m-test-engine.json) is the canonical +machine-readable contract between this repo and its consumers +(primarily `m-cli`'s `m doctor` and the upcoming `m engine` subcommand +family). It declares the image registry, default tag, container name, +bind-mount layout, compose file path, minimum Docker version, YottaDB +release, and a `run_args` block sufficient to reconstruct an equivalent +`docker run` invocation when the compose plugin is unavailable. + +The schema lives alongside it at +[`dist/m-test-engine.schema.json`](dist/m-test-engine.schema.json). +`make check-manifest` validates both `dist/repo.meta.json` and +`dist/m-test-engine.json` — schema-shape, `compose_file` resolves on +disk, and `verified_on` is within 90 days. + +Consumers vendor `dist/m-test-engine.json` at release time (m-cli pins +to a tagged release of this repo). The protocol bump policy and the +broader integration design are recorded in +[`docs/m-engine-implementation-plan.md`](docs/m-engine-implementation-plan.md). + ## Configuration | Env var | Default | Purpose | diff --git a/dist/m-test-engine.json b/dist/m-test-engine.json new file mode 100644 index 0000000..4016c5d --- /dev/null +++ b/dist/m-test-engine.json @@ -0,0 +1,33 @@ +{ + "$schema": "https://raw.githubusercontent.com/m-dev-tools/m-test-engine/main/dist/m-test-engine.schema.json", + "protocol": 1, + "image": "ghcr.io/m-dev-tools/m-test-engine", + "default_tag": "r2.02", + "container": "m-test-engine", + "bind_mount": { + "host": "/m-work", + "container": "/m-work", + "mode": "rw" + }, + "compose_file": "docker/compose.yml", + "repo_url": "https://github.com/m-dev-tools/m-test-engine", + "min_docker": "20.10", + "ydb_version": "r2.02", + "run_args": { + "hostname": "m-test-engine", + "working_dir": "/m-work", + "restart": "unless-stopped", + "volumes": [ + { + "name": "m-test-engine-globals", + "target": "/data" + } + ], + "command": [ + "tail", + "-f", + "/dev/null" + ] + }, + "verified_on": "2026-05-11" +} \ No newline at end of file diff --git a/dist/m-test-engine.schema.json b/dist/m-test-engine.schema.json new file mode 100644 index 0000000..1e3d1e4 --- /dev/null +++ b/dist/m-test-engine.schema.json @@ -0,0 +1,130 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://raw.githubusercontent.com/m-dev-tools/m-test-engine/main/dist/m-test-engine.schema.json", + "title": "m-test-engine manifest", + "description": "Public machine-readable contract for the m-test-engine YottaDB Docker container. Consumed by m-cli's `m doctor` and the `m engine` subcommand family. See docs/m-engine-implementation-plan.md for design rationale and the protocol bump policy.", + "type": "object", + "additionalProperties": false, + "required": [ + "protocol", + "image", + "default_tag", + "container", + "bind_mount", + "compose_file", + "repo_url", + "min_docker", + "ydb_version", + "run_args", + "verified_on" + ], + "properties": { + "$schema": { + "type": "string", + "format": "uri" + }, + "protocol": { + "type": "integer", + "minimum": 1, + "description": "Contract version. Bumped when a field is removed, renamed, or has its semantics changed. Additive evolution (new optional fields) does NOT bump." + }, + "image": { + "type": "string", + "description": "Canonical image reference without tag (e.g. ghcr.io/m-dev-tools/m-test-engine). Combined with default_tag for the full reference." + }, + "default_tag": { + "type": "string", + "description": "Default tag to pull when no override is given (e.g. r2.02)." + }, + "container": { + "type": "string", + "description": "Container name. Public contract — m-cli's DockerEngine, m-stdlib's test runner, and every `docker exec` call reference this exact name." + }, + "bind_mount": { + "type": "object", + "additionalProperties": false, + "required": ["host", "container", "mode"], + "description": "Single shared bind mount. Host directory contains all participating m-* repo checkouts; container sees identical layout.", + "properties": { + "host": { + "type": "string", + "description": "Absolute host filesystem path." + }, + "container": { + "type": "string", + "description": "Absolute in-container path of the bind mount." + }, + "mode": { + "type": "string", + "enum": ["ro", "rw"], + "description": "Mount mode." + } + } + }, + "compose_file": { + "type": "string", + "description": "Repo-relative path to the canonical compose.yml. Primary path for `m engine start`." + }, + "repo_url": { + "type": "string", + "format": "uri", + "description": "Repo URL for issue links and source-of-truth pointers." + }, + "min_docker": { + "type": "string", + "pattern": "^\\d+\\.\\d+(\\.\\d+)?$", + "description": "Minimum Docker Engine version required (e.g. 20.10)." + }, + "ydb_version": { + "type": "string", + "description": "YottaDB release shipped in the image (e.g. r2.02). Sourced from the image base." + }, + "run_args": { + "type": "object", + "additionalProperties": false, + "required": ["hostname", "working_dir", "restart", "volumes", "command"], + "description": "Fallback args for `docker run` when the compose plugin is unavailable. Consumer reconstructs an equivalent invocation from these fields + image + bind_mount.", + "properties": { + "hostname": { + "type": "string" + }, + "working_dir": { + "type": "string" + }, + "restart": { + "type": "string", + "enum": ["no", "always", "on-failure", "unless-stopped"] + }, + "volumes": { + "type": "array", + "description": "Named Docker volumes beyond the primary bind_mount.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["name", "target"], + "properties": { + "name": { + "type": "string", + "description": "Named Docker volume identifier." + }, + "target": { + "type": "string", + "description": "Absolute in-container mount target." + } + } + } + }, + "command": { + "type": "array", + "items": { "type": "string" }, + "description": "Container entrypoint command (overrides Dockerfile CMD if set)." + } + } + }, + "verified_on": { + "type": "string", + "format": "date", + "description": "ISO date the manifest was last verified against the live image. Stale manifests >90 days fail check-manifest." + } + } +} diff --git a/dist/repo.meta.json b/dist/repo.meta.json index 50ba023..d42acdd 100644 --- a/dist/repo.meta.json +++ b/dist/repo.meta.json @@ -6,11 +6,12 @@ "language": ["dockerfile"], "license": "AGPL-3.0", "agent_instructions": "AGENTS.md", - "verified_on": "2026-05-10", + "verified_on": "2026-05-11", "exposes": { - "lifecycle": "dist/lifecycle.json", - "dockerfile": "docker/Dockerfile", - "compose": "docker/compose.yml" + "lifecycle": "dist/lifecycle.json", + "engine_contract": "dist/m-test-engine.json", + "dockerfile": "docker/Dockerfile", + "compose": "docker/compose.yml" }, "verification_commands": ["make smoke", "make check-manifest"], "status": "active", diff --git a/docs/m-engine-tracker.md b/docs/m-engine-tracker.md new file mode 100644 index 0000000..646eed0 --- /dev/null +++ b/docs/m-engine-tracker.md @@ -0,0 +1,231 @@ +--- +created: 2026-05-11 +last_modified: 2026-05-11 +revisions: 0 +doc_type: [TRACKER, LIVE] +locks: m-engine-implementation-plan.md +--- + +# `m engine` — implementation tracker + +**Status**: live. This is the single source of truth as the +`m engine` work progresses. + +**Relationship to the plan**: +[`docs/m-engine-implementation-plan.md`](m-engine-implementation-plan.md) +is **locked** as of the first row in this tracker moving off `todo`. +Decisions in the plan are not re-litigated here; if implementation +discovers a reason to change a decision, that goes in the narrative +log below and the plan gets a revision bump in a *follow-up* edit, not +silently. + +**How to use this document**: + +1. Update the table when a stage changes status. Keep the `Notes` + cell to one short line — if longer context is needed, append a + date-stamped entry to the narrative log and write `see 2026-MM-DD` + in the cell. +2. Append to the narrative log; never rewrite history. If a prior + entry was wrong, add a new entry that corrects it. +3. Status values: `todo` · `in-progress` · `blocked` · `done` · + `deferred` · `cancelled`. +4. When a stage's discoveries imply a *later* stage needs to change, + note it on the affected later row's Notes cell *and* in the log. + +--- + +## Progress table + +| Phase | Stage | Repo | Status | Notes / discoveries / followup | +|---|---|---|---|---| +| **1** — vendored manifest + actionable `m doctor` | | | | | +| 1a.1 | Author `dist/m-test-engine.json` + `dist/m-test-engine.schema.json` | m-test-engine | done | shipped 2026-05-11; manifest passes full schema validation. see 2026-05-11 | +| 1a.2 | `make check-manifest` validates schema + paths + 90-day freshness | m-test-engine | done | extended `tools/check-manifest.py`; 90-day gate tested by backdating | +| 1a.3 | README pointer to manifest as public machine-readable contract | m-test-engine | done | new "Machine-readable contract" section between consumer-usage and Configuration | +| 1b.1 | Vendor `dist/m-test-engine.json` into m-cli at pinned tag | m-cli | done | Makefile vendoring rule via `M_TEST_ENGINE`; drift gate is existing `git diff --exit-code dist/` | +| 1b.2 | Rewrite `m doctor` Docker-path hints to consume manifest | m-cli | done | 5 new Docker checks (installed/daemon/image/container/bind-mount) emit manifest-derived fix commands | +| 1b.3 | Extend `m doctor --json` schema with `fix.command` + `fix.destructive` | m-cli | done | `Fix` dataclass + JSON surface; prerequisites also exposed in JSON | +| 1b.4 | Root-cause grouping: prerequisite-failed checks report `SKIPPED` | m-cli | done | `Status.SKIPPED` + `prerequisites` field on `Check`; runner skips downstream when prereq non-OK | +| **2** — `m engine` subcommand family | | | | | +| 2.1 | Subcommand skeleton under `src/m_cli/engine/` (status/install/start/stop/restart/logs/shell/exec/version/upgrade/reset/capabilities) | m-cli | todo | | +| 2.2 | `EngineDriver` protocol + built-in `DockerDriver` | m-cli | todo | | +| 2.3 | Declare `m_cli_engines` entry-point group; document in `PLUGIN_API_VERSION` | m-cli | todo | name locked per plan §1.6 | +| 2.4 | `m doctor --fix` delegates to `m engine `; refuses destructive verbs without `--confirm` | m-cli | todo | | +| 2.5 | `dist/commands.json` auto-includes `engine` namespace (verify) | m-cli | todo | should be automatic via argparse registry | +| 2.6 | Compose-detection + `docker run` fallback path | m-cli | todo | per plan §1.3 | +| **3** — OCI labels + `HEALTHCHECK` | | | | | +| 3a.1 | Dockerfile `LABEL org.m-dev-tools.m-test-engine.*` (protocol, bind-mount, ydb-version, image-rev) | m-test-engine | todo | | +| 3a.2 | Dockerfile `HEALTHCHECK CMD $ydb_dist/mumps -run %XCMD 'write "ok",!'` | m-test-engine | todo | | +| 3a.3 | `make smoke` extended to verify label set + healthcheck presence | m-test-engine | todo | | +| 3a.4 | Release process: propagate `image-rev` via `docker buildx --build-arg GIT_SHA=...` | m-test-engine | todo | | +| 3b.1 | `m engine status` reads `docker image inspect`; surfaces `protocol_mismatch` / `image_outdated` WARN | m-cli | todo | | +| 3b.2 | `m engine version` prints manifest-expected vs image-reported | m-cli | todo | | +| **4** — `mte` container-side introspection | | | | | +| 4a.1 | `mte status --json` on `$PATH` inside container (ydb_dist, release, uptime_s, globals_count, routines_count, mounted_repos) | m-test-engine | todo | | +| 4a.2 | `make smoke` asserts `mte status --json` is valid JSON with allowlisted fields | m-test-engine | todo | | +| 4b.1 | `m engine status --verbose` folds `mte status --json` into report | m-cli | todo | | +| 4b.2 | `m engine watch --interval 5s` streams `mte status --json` lines | m-cli | todo | | +| **5** — skill / MCP integration | | | | | +| 5.1 | Auto-generated `~/claude/skills/m-engine/SKILL.md` from manifest + `commands.json` engine slice (`make skill-install`) | m-test-engine | todo | mirrors m-stdlib pattern | +| 5.2 | `verbs` section in `dist/m-test-engine.json` (destructive / read_only / requires_confirm per verb) | m-test-engine | todo | requires manifest schema bump — decide if protocol bump per plan §1.5 | +| 5.3 | (Optional) m-cli MCP server registers safe verbs as MCP tools | m-cli | todo | | + +--- + +## Narrative log + +Append-only. One entry per non-trivial discovery, blocker, or decision +that emerged during implementation. Date-stamp every entry. Don't +rewrite — supersede. + +### 2026-05-11 — tracker created + +Plan locked at `docs/m-engine-implementation-plan.md` rev 0. First +stage to move off `todo` triggers the lock; until then the plan can +still be edited freely if the user spots something. + +### 2026-05-11 — Phase 1a complete (m-test-engine side) + +Shipped `dist/m-test-engine.json` (manifest) + `dist/m-test-engine.schema.json` +(JSON Schema 2020-12). Extended `tools/check-manifest.py` to validate +both manifests under one `make check-manifest` gate. README gained a +"Machine-readable contract" section pointing at the manifest. Enrolled +the manifest into `dist/repo.meta.json` `exposes` as `engine_contract` +so it's discoverable through the tier-1 catalog. Bumped repo.meta's +`verified_on` to 2026-05-11. + +Validation states confirmed: +- basic-only path (no `jsonschema`): green +- full schema path (jsonschema in a temp venv): green for both manifests +- 90-day freshness gate fires correctly when `verified_on` is backdated + (tested with `2020-01-01` → exit 1 with clear error message) + +**Discoveries that affect later phases:** + +1. **`/m-work` vs `/work` bind-mount mismatch is intentional, not yet + resolved.** The new manifest declares `bind_mount.host = /m-work` + and `bind_mount.container = /m-work` per plan §1.4. The actual + `docker/compose.yml` and `docker/Dockerfile` still bind `/work` + (unchanged in this phase per plan: "no Docker image changes"). + `dist/lifecycle.json` also still reports `/work`. The runtime switch + happens in Phase 2 alongside m-cli's `m engine start` so the cutover + is atomic across all consumer paths. + + **Followup for Phase 2 (m-cli side, stage 2.1 or 2.6):** when + `m engine start` lands, the compose.yml + Dockerfile + lifecycle.json + must all flip to `/m-work` in the same commit, and a migration hint + must be wired into `m doctor` for users with legacy `/work` mounts. + +2. **GHCR image not yet published.** Manifest references + `ghcr.io/m-dev-tools/m-test-engine:r2.02` but no image exists at + that URL yet — locally the image is `m-test-engine:latest`, built + from `docker/Dockerfile`. This is fine for Phase 1 (manifest is + *published intent*; m-cli's hint copy can fall back to a + `make -C ~/projects/m-test-engine up` recommendation while the + GHCR pipeline is set up). + + **Followup for Phase 3 (stage 3a.4):** set up GHCR publish workflow + and produce the first tagged image. Until then, m-cli's `m engine + install` falls back to local-build via compose. + +3. **`exposes.engine_contract` is a new public-surface entry.** m-cli's + Phase 1b vendoring step (stage 1b.1) can use it as the canonical + discovery hop: read `dist/repo.meta.json` `exposes.engine_contract` + → fetch the path → vendor. Mirrors how other org tools discover + tier-1 artifacts. + +4. **jsonschema is opt-in, not required.** Followed the existing + convention: `make check-manifest` runs without dev deps and prints a + "skipping full schema validation" notice when jsonschema is absent. + Full validation confirmed once in a temp venv. No `pyproject.toml` + or `requirements.txt` introduced — this repo stays dependency-free + on the Python side. + +### 2026-05-11 — Phase 1b complete (m-cli side) + +Vendored `dist/m-test-engine.json` into `~/m-dev-tools/m-cli/dist/`. +Added a Makefile rule keyed on `$(M_TEST_ENGINE)` so a re-vendor +happens transparently when the upstream copy is newer; the existing +`make check-manifest` (`git diff --exit-code dist/`) is the drift gate. +Introduced `m_cli.engine_manifest` with `EngineManifest` / +`BindMount` / `Volume` / `RunArgs` dataclasses + protocol handshake +(rejects manifests claiming a higher protocol than this build supports). +Added `m_cli.doctor._runtime` shell-out probes (`docker_available`, +`docker_daemon_reachable`, `docker_image_present`, +`docker_container_running`, `path_exists`) — separated from the check +functions so tests monkeypatch the runtime surface without faking +subprocess directly. + +Doctor extensions: +- `Status.SKIPPED` added as a fourth status level +- `Fix` dataclass (command tuple + destructive bool) +- `Check.prerequisites: tuple[str, ...]` declares the chain +- `Check.fix: Fix | None` carries the copy-pasteable repair +- `run_all_checks()` skips downstream when any declared prereq is + non-OK and emits a SKIPPED Check pointing at the failed prereq +- 5 new Docker engine checks: `docker_installed`, `docker_daemon`, + `engine_image`, `engine_container`, `engine_bind_mount` — all derive + hint/fix from manifest fields +- Text output renders `hint:` and `fix:` lines under failing checks; + summary line now includes `N skipped` +- JSON output exposes `fix.command` (list) + `fix.destructive` (bool) + and `prerequisites` (list) per check + +Local YDB checks (`ydb_dist`, `ydb_routines`, `parser`, `keywords`, +`ydb_binary`) are unchanged — they don't declare prerequisites because +each handles its own missing-input case gracefully (e.g. +`check_ydb_binary` falls back to `$PATH`). + +Test surface: 18 new tests in `tests/test_doctor.py` + 11 in the new +`tests/test_engine_manifest.py`. Full suite still 1393 passing, 1 +skipped (pre-existing); lint + mypy clean. + +**Discoveries that affect later phases:** + +5. **Makefile default `M_TEST_ENGINE` is `$(HOME)/projects/m-test-engine`.** + This repo lives at `$HOME/m-dev-tools/m-test-engine` for this user, + so the wildcard returns empty and `make manifest` skips the vendor + refresh (treating the committed copy as authoritative). This is the + intended behaviour for users who don't have m-test-engine checked + out, but it means the user must pass `M_TEST_ENGINE=...` explicitly + when they want a re-vendor in this dev setup. + + **Followup for Phase 5 / docs:** when the broader docs land, note + that the `M_TEST_ENGINE` default reflects the canonical + `~/projects/m-test-engine` clone path documented in m-cli's + README. Multiple-checkout setups override per-invocation. + +6. **Local YDB checks still fire when Docker path is healthy.** Doctor + output today lists Docker checks first, then local YDB checks. When + Docker is fully OK, the local WARNs are noise (the canonical path + is healthy; alternative isn't needed). The plan's §2.4 sketch + suggested an "alternative engines:" grouped section — out of scope + for Phase 1b but worth pinning as a Phase 2 polish. + + **Followup for Phase 2 stage 2.1 (m engine status):** when + `m engine status` lands as the canonical health surface, refactor + `m doctor` to lead with engine status and demote the local YDB + block to an "alternative engines:" subsection that only emits + WARNs when the canonical path is unhealthy AND the user opted into + local YDB. + +7. **`shlex.join` quotes shell variables in fix commands.** Output + like `sudo install -d -o '$USER' -g '$USER' /m-work` quotes the + `$USER` shell var, so direct copy-paste of the fix line would + create a directory literally named `$USER`. The unquoted form + appears in `hint:`; the quoted argv form is correct for autonomous + exec (`subprocess.run(cmd_list)`). + + **Followup for Phase 2:** decide whether `fix.command` should + support `${env}` placeholders for the agent harness to expand, or + whether autonomous fixes should always produce argv with no shell + substitution. Lean toward the latter: it keeps the contract simple + and avoids a substitution-syntax decision. + +8. **Manifest loading is best-effort in `m doctor`.** If the vendored + copy is missing or malformed, `_manifest()` returns None and each + engine check emits a generic "manifest not loadable" WARN instead + of crashing. This degrades gracefully but means an agent walking + `m doctor --json` should treat that specific WARN as a higher + priority than the rest (fix the manifest first, then everything + downstream becomes resolvable). diff --git a/tools/check-manifest.py b/tools/check-manifest.py index f01d59a..55f3737 100755 --- a/tools/check-manifest.py +++ b/tools/check-manifest.py @@ -1,18 +1,21 @@ #!/usr/bin/env python3 -"""Phase-0 contract gate for dist/repo.meta.json. +"""Contract gates for dist/repo.meta.json and dist/m-test-engine.json. -Validates that: - 1. dist/repo.meta.json parses as JSON. - 2. Required fields from the org-level repo.meta.schema.json contract - are present. - 3. Each path under `exposes.*` resolves on disk. - 4. (Best-effort) full schema validation if jsonschema is available - and the canonical schema URL is reachable. +Two manifests live under dist/: -Exits 0 on success; non-zero with structured stderr on failure. + * dist/repo.meta.json — Phase-0 AI-discoverability tier-1 contract + shared across the m-dev-tools org. Schema + hosted in the org-level .github repo. + * dist/m-test-engine.json — m-cli ↔ m-test-engine integration contract + introduced by the m-engine implementation + plan (see docs/m-engine-implementation-plan.md). + Schema is local: dist/m-test-engine.schema.json. -Engine-free, no Node, no Python deps beyond the standard library -unless jsonschema happens to be installed. +Both are validated here so a single `make check-manifest` covers the +full publish surface. Each manifest is checked independently; either +failing exits non-zero. + +Engine-free, stdlib only (jsonschema is opt-in for full validation). """ from __future__ import annotations @@ -20,11 +23,14 @@ import json import sys import urllib.request +from datetime import date, datetime, timedelta, timezone from pathlib import Path -MANIFEST = Path("dist/repo.meta.json") +REPO_META = Path("dist/repo.meta.json") +ENGINE_MANIFEST = Path("dist/m-test-engine.json") +ENGINE_SCHEMA = Path("dist/m-test-engine.schema.json") -REQUIRED_FIELDS = ( +REPO_META_REQUIRED = ( "id", "repo", "role", @@ -36,21 +42,35 @@ "verification_commands", ) +# Stale manifests are a leading indicator of contract drift. The org +# smoke test rejects repo.meta.json older than 90 days; we apply the +# same threshold to the engine manifest. +FRESHNESS_DAYS = 90 -def main() -> int: - if not MANIFEST.exists(): - print(f"ERROR: {MANIFEST} not found", file=sys.stderr) - return 1 +def _load_json(path: Path) -> dict | None: + if not path.exists(): + print(f"ERROR: {path} not found", file=sys.stderr) + return None try: - data = json.loads(MANIFEST.read_text(encoding="utf-8")) + return json.loads(path.read_text(encoding="utf-8")) except json.JSONDecodeError as exc: - print(f"ERROR: {MANIFEST} is invalid JSON: {exc}", file=sys.stderr) + print(f"ERROR: {path} is invalid JSON: {exc}", file=sys.stderr) + return None + + +def _today() -> date: + return datetime.now(timezone.utc).date() + + +def validate_repo_meta() -> int: + data = _load_json(REPO_META) + if data is None: return 1 - missing = [f for f in REQUIRED_FIELDS if f not in data] + missing = [f for f in REPO_META_REQUIRED if f not in data] if missing: - print(f"ERROR: missing required fields: {missing}", file=sys.stderr) + print(f"ERROR: {REPO_META} missing required fields: {missing}", file=sys.stderr) return 1 fail = False @@ -66,14 +86,14 @@ def main() -> int: if fail: return 1 - # Best-effort full schema validation. Skipped silently if jsonschema - # isn't available (the canonical Track-A validator runs in the org - # smoke test against the same manifest). + # Best-effort full schema validation against the canonical org URL. + # The Track-A validator in the org smoke test is authoritative; this + # local check is a fast pre-push heuristic. try: from jsonschema import Draft202012Validator # type: ignore except ImportError: print( - "check-manifest: dist/repo.meta.json valid; " + f"check-manifest: {REPO_META} valid; " "all exposes.* present ✓ (jsonschema not installed — " "skipping full schema validation)" ) @@ -85,7 +105,7 @@ def main() -> int: schema = json.load(resp) except Exception as exc: # noqa: BLE001 print( - f"check-manifest: dist/repo.meta.json valid; all exposes.* " + f"check-manifest: {REPO_META} valid; all exposes.* " f"present ✓ (skipped live schema fetch: {exc})" ) return 0 @@ -98,11 +118,91 @@ def main() -> int: return 1 print( - "check-manifest: dist/repo.meta.json valid against org schema; " + f"check-manifest: {REPO_META} valid against org schema; " "all exposes.* present ✓" ) return 0 +def validate_engine_manifest() -> int: + data = _load_json(ENGINE_MANIFEST) + if data is None: + return 1 + + schema = _load_json(ENGINE_SCHEMA) + if schema is None: + return 1 + + # Cross-check that compose_file resolves on disk. The schema can't + # enforce filesystem existence; this is the manifest's equivalent of + # repo.meta.json's exposes.* check. + compose_path = Path(data.get("compose_file", "")) + if not compose_path.exists(): + print( + f"ERROR: compose_file payload missing on disk: {compose_path}", + file=sys.stderr, + ) + return 1 + + # Freshness: stale manifests are a leading indicator of drift + # between the published contract and the actual image / compose + # / Dockerfile. 90-day window matches the org repo.meta gate. + raw_verified = data.get("verified_on") + if not isinstance(raw_verified, str): + print( + f"ERROR: {ENGINE_MANIFEST} verified_on missing or non-string", + file=sys.stderr, + ) + return 1 + try: + verified = date.fromisoformat(raw_verified) + except ValueError as exc: + print( + f"ERROR: {ENGINE_MANIFEST} verified_on not ISO date: {exc}", + file=sys.stderr, + ) + return 1 + age = _today() - verified + if age > timedelta(days=FRESHNESS_DAYS): + print( + f"ERROR: {ENGINE_MANIFEST} verified_on={verified} is " + f"{age.days}d old (limit {FRESHNESS_DAYS}d). " + "Re-verify against the live image and bump verified_on.", + file=sys.stderr, + ) + return 1 + + # Full schema validation against the local schema file. No network + # fetch needed — schema ships in this repo. + try: + from jsonschema import Draft202012Validator # type: ignore + except ImportError: + print( + f"check-manifest: {ENGINE_MANIFEST} basic checks passed ✓ " + "(compose_file present; verified_on fresh; " + "jsonschema not installed — skipping full schema validation)" + ) + return 0 + + errors = list(Draft202012Validator(schema).iter_errors(data)) + if errors: + for err in errors: + path = "/".join(str(p) for p in err.absolute_path) or "" + print(f"SCHEMA ERROR at {path}: {err.message}", file=sys.stderr) + return 1 + + print( + f"check-manifest: {ENGINE_MANIFEST} valid against local schema; " + "compose_file present; verified_on fresh ✓" + ) + return 0 + + +def main() -> int: + rc_repo = validate_repo_meta() + rc_engine = validate_engine_manifest() + return rc_repo or rc_engine + + if __name__ == "__main__": sys.exit(main()) From 694094ecb0d7f11aee0ed15012bd251d3ffc2373 Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Mon, 11 May 2026 11:26:23 -0400 Subject: [PATCH 3/3] runtime: cut over from /work to /m-work bind mount (Phase 2 prep) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1a's manifest declared the canonical bind mount as /m-work but left compose.yml + Dockerfile + lifecycle.json pinned to /work. This commit closes that drift so a `docker compose up -d` produces a container whose runtime layout matches the published contract. * docker/compose.yml — bind ${M_TEST_ENGINE_BIND:-/m-work}:/m-work; working_dir: /m-work; restart: unless-stopped (matches manifest run_args.restart) * docker/Dockerfile — WORKDIR /m-work; prose updated for the shared- mount model * dist/lifecycle.json — mount_point /m-work, mount_source_default /m-work * dist/repo.meta.json — notes field references the new mount point * AGENTS.md + README.md — prose reflects the host /m-work shared working tree (all m-* repos as siblings under one host directory, visible identically inside the container) This is the runtime half of the bind-mount decision in docs/m-engine-implementation-plan.md §1.4. The m-cli side (DockerEngine.bind_root default + stage_routines prefix-mapping) lands in a coordinated m-cli commit. `make check-manifest` green; `docker compose config` validates the compose file shape. Smoke target requires a running container (kicked off via `m engine start` once the m-cli side lands). Tracker (docs/m-engine-tracker.md) updated with completed Phase 2 stages 2.0 (this cutover) + 2.1/2.2/2.3/2.5/2.6 on the m-cli side and three new followups (sudo-in-fix design, capabilities recursion, m_cli.engine package conversion). Co-Authored-By: Claude Opus 4.7 (1M context) --- AGENTS.md | 8 ++--- README.md | 16 +++++---- dist/lifecycle.json | 4 +-- dist/repo.meta.json | 2 +- docker/Dockerfile | 15 ++++---- docker/compose.yml | 25 +++++++------ docs/m-engine-tracker.md | 77 ++++++++++++++++++++++++++++++++++++---- 7 files changed, 110 insertions(+), 37 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0f6f9d1..433afd2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ location: ~/m-dev-tools/m-test-engine exposes: container_name: "m-test-engine" image_base: "yottadb/yottadb-base:latest-master" - mount_point: "/work" + mount_point: "/m-work" lifecycle: "make up | down | smoke | shell | clean | logs" consumes: @@ -38,7 +38,7 @@ docs: Minimal YottaDB Docker container for `m-cli` and `m-stdlib` testing. Long-running container exposing a YottaDB engine via `docker exec`. -Consumer projects bind-mount their source as `/work` and dispatch +Consumer projects coordinate through a shared `/m-work` bind mount and dispatch `docker exec m-test-engine $ydb_dist/mumps -run ...` commands. The full design rationale and lifecycle table is in `README.md`. @@ -49,7 +49,7 @@ The full design rationale and lifecycle table is in `README.md`. `compose.yml` plus thin Makefile wrappers (`up`, `down`, `smoke`, `shell`, `clean`, `logs`). - A container name (`m-test-engine`) and a bind-mount contract - (`/work` in container = consumer's `$PWD` on host). + (`/m-work` in container = host's `/m-work`, the shared m-* working tree). - A `dist/lifecycle.json` describing those facts in machine-readable form for the org-level AI-discoverability catalog. @@ -125,7 +125,7 @@ make check-docs-prose # docs/ holds only prose (this repo has no docs/ at all) the date only when the manifest changes materially (image bump, container rename, mount-point change). - **Container name and mount point are public contract.** - `m-test-engine` (name) and `/work` (mount) are referenced by + `m-test-engine` (name) and `/m-work` (mount) are referenced by consumer code paths in m-cli and m-stdlib. Renaming either is a breaking change requiring coordinated updates in those repos. - **Image base is pinned for a reason.** The Dockerfile pin diff --git a/README.md b/README.md index 801f0a4..cdf5214 100644 --- a/README.md +++ b/README.md @@ -17,9 +17,11 @@ YottaDB + VistA + RPC broker + FileMan + Octo SQL stack. For non-VistA M development, that's overkill: the tests just need a YottaDB engine. `m-test-engine` is exactly that — `yottadb/yottadb-base:latest-master` -with a keep-alive command. No SSH server, no VistA, no extras. The -consumer project's source bind-mounts in via `/work`; routines are -compiled and run from there with no SCP / staging round-trip. +with a keep-alive command. No SSH server, no VistA, no extras. All +participating m-* repos live under `/m-work` on the host (clones or +symlinks) and the container bind-mounts that single directory as +`/m-work`, so every repo's routines are simultaneously visible inside +one running engine — no per-cwd remount, no SCP / staging round-trip. For VistA-specific work, keep using vista-meta. This repo is the lightweight default for the rest of the toolchain. @@ -69,8 +71,8 @@ docker exec m-test-engine bash -lc \ ``` The `` is the in-container path to the consumer project's -routine dirs — derived by mapping the project root through the -`/work` bind mount. There's no SCP / SFTP step: the consumer's +routine dirs — derived by mapping the project root through the shared +`/m-work` bind mount. There's no SCP / SFTP step: every m-* repo's filesystem is directly visible inside the container. `m-stdlib`'s test runner uses the same transport once Track A3 of the @@ -101,7 +103,7 @@ broader integration design are recorded in | Env var | Default | Purpose | |-----------------------|---------|-----------------------------------------------------------| -| `M_TEST_ENGINE_BIND` | `$PWD` | Override the host directory mounted as `/work`. | +| `M_TEST_ENGINE_BIND` | `/m-work` | Override the host directory mounted as `/m-work`. | ## Optional: SSH overlay @@ -116,7 +118,7 @@ PR if you need it. | | m-test-engine | vista-meta | |---|---|---| | YottaDB | ✓ | ✓ | -| Bind-mount source code | ✓ (via `/work`) | ✓ (via `/home/vehu/dev/r`) | +| Bind-mount source code | ✓ (via `/m-work`) | ✓ (via `/home/vehu/dev/r`) | | VistA + FileMan | — | ✓ | | RPC broker / VistALink | — | ✓ | | Octo SQL / YDB GUI | — | ✓ | diff --git a/dist/lifecycle.json b/dist/lifecycle.json index 9812113..ffb4ef3 100644 --- a/dist/lifecycle.json +++ b/dist/lifecycle.json @@ -4,9 +4,9 @@ "hostname": "m-test-engine", "image": "m-test-engine:latest", "image_base": "yottadb/yottadb-base:latest-master", - "mount_point": "/work", + "mount_point": "/m-work", "mount_source_env": "M_TEST_ENGINE_BIND", - "mount_source_default": "$PWD", + "mount_source_default": "/m-work", "globals_volume": "m-test-engine-globals", "ports": [], "ssh_server": false, diff --git a/dist/repo.meta.json b/dist/repo.meta.json index d42acdd..6c9488d 100644 --- a/dist/repo.meta.json +++ b/dist/repo.meta.json @@ -15,5 +15,5 @@ }, "verification_commands": ["make smoke", "make check-manifest"], "status": "active", - "notes": "Container name 'm-test-engine' and mount point '/work' are public contract — m-cli's DockerEngine and m-stdlib's test runner reference them. Image base 'yottadb-base:latest-master' is pinned to match m-stdlib's CI." + "notes": "Container name 'm-test-engine' and mount point '/m-work' are public contract — m-cli's DockerEngine and m-stdlib's test runner reference them via `dist/m-test-engine.json`. Image base 'yottadb-base:latest-master' is pinned to match m-stdlib's CI." } diff --git a/docker/Dockerfile b/docker/Dockerfile index 910635d..128246e 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -5,11 +5,11 @@ # # docker exec m-test-engine bash -lc '$ydb_dist/mumps -run ^SUITE' # -# The companion compose.yml bind-mounts the consumer project's $PWD as -# /work, so a project's routine dirs (src/, routines/, tests/, etc.) -# become /work/src, /work/routines, /work/tests inside the container. -# m-cli's DockerEngine.stage_routines() generates the right -# `ydb_routines` value for those in-container paths. +# The companion compose.yml bind-mounts the host's /m-work directory +# (containing all m-* repo checkouts) as /m-work inside the container. +# A consumer project at /m-work/m-cli/ on the host becomes +# /m-work/m-cli/ inside the container; m-cli's DockerEngine assembles +# `ydb_routines` from the appropriate subdirs without per-cwd remounts. # # Pinned to yottadb-base:latest-master to match what m-stdlib's CI uses. @@ -26,8 +26,9 @@ RUN echo '. /opt/yottadb/current/ydb_env_set 2>/dev/null || true' \ && chmod +x /etc/profile.d/ydb-env.sh # Working directory matches the bind-mount target. The compose file's -# bind makes /work == the consumer's $PWD on the host. -WORKDIR /work +# bind makes /m-work == the host's /m-work directory (the shared +# m-* working tree). +WORKDIR /m-work # The container needs to stay running so `docker exec` calls can target # it. tail -f /dev/null is the canonical "do nothing forever" pattern; diff --git a/docker/compose.yml b/docker/compose.yml index b0d44bb..c376298 100644 --- a/docker/compose.yml +++ b/docker/compose.yml @@ -4,11 +4,14 @@ # # docker compose -f $(M_TEST_ENGINE)/docker/compose.yml up -d # -# (or use the consumer project's `make engine-up` target, which -# delegates here once Track A3 lands). +# Bind mount: /m-work in the container = the host's /m-work directory. +# The host /m-work is the shared working tree containing all m-* repo +# checkouts (m-cli, m-stdlib, m-test-engine, m-modern-corpus, ...) so a +# single running engine exposes routines from every participating repo +# without per-cwd container churn. Override with M_TEST_ENGINE_BIND +# (host path) if you keep your m-* repos elsewhere. # -# Bind mount: /work in the container = the caller's $PWD on the host. -# Override with the M_TEST_ENGINE_BIND env var to mount somewhere else. +# See `dist/m-test-engine.json` for the full machine-readable contract. name: m-test-engine @@ -20,17 +23,19 @@ services: image: m-test-engine:latest container_name: m-test-engine hostname: m-test-engine + restart: unless-stopped volumes: - # Bind the caller's working directory (or M_TEST_ENGINE_BIND - # override) as /work inside the container. - - ${M_TEST_ENGINE_BIND:-${PWD}}:/work + # Shared host bind: /m-work → /m-work. All m-* repos live under + # /m-work on the host (clones or symlinks). Override host side + # with M_TEST_ENGINE_BIND if your layout differs. + - ${M_TEST_ENGINE_BIND:-/m-work}:/m-work # Persistent globals so test data survives container restarts. - # m-cli/m-stdlib tests typically clear globals per-test, but the - # volume protects against accidental loss between sessions. + # Tests typically clear globals per-test, but the volume protects + # against accidental loss between sessions. - m-test-engine-globals:/data - working_dir: /work + working_dir: /m-work volumes: m-test-engine-globals: diff --git a/docs/m-engine-tracker.md b/docs/m-engine-tracker.md index 646eed0..e313562 100644 --- a/docs/m-engine-tracker.md +++ b/docs/m-engine-tracker.md @@ -47,12 +47,13 @@ silently. | 1b.3 | Extend `m doctor --json` schema with `fix.command` + `fix.destructive` | m-cli | done | `Fix` dataclass + JSON surface; prerequisites also exposed in JSON | | 1b.4 | Root-cause grouping: prerequisite-failed checks report `SKIPPED` | m-cli | done | `Status.SKIPPED` + `prerequisites` field on `Check`; runner skips downstream when prereq non-OK | | **2** — `m engine` subcommand family | | | | | -| 2.1 | Subcommand skeleton under `src/m_cli/engine/` (status/install/start/stop/restart/logs/shell/exec/version/upgrade/reset/capabilities) | m-cli | todo | | -| 2.2 | `EngineDriver` protocol + built-in `DockerDriver` | m-cli | todo | | -| 2.3 | Declare `m_cli_engines` entry-point group; document in `PLUGIN_API_VERSION` | m-cli | todo | name locked per plan §1.6 | -| 2.4 | `m doctor --fix` delegates to `m engine `; refuses destructive verbs without `--confirm` | m-cli | todo | | -| 2.5 | `dist/commands.json` auto-includes `engine` namespace (verify) | m-cli | todo | should be automatic via argparse registry | -| 2.6 | Compose-detection + `docker run` fallback path | m-cli | todo | per plan §1.3 | +| 2.0 | **/m-work runtime cutover** (compose.yml + Dockerfile + lifecycle.json + DockerEngine default) | both | done | added during Phase 2; closes the Phase 1 manifest/runtime drift | +| 2.1 | Subcommand skeleton under `src/m_cli/engine_cli.py` (status/install/start/stop/restart/logs/shell/exec/version/upgrade/reset/capabilities) | m-cli | done | 12 verbs wired; `m engine status --json` is canonical health surface | +| 2.2 | `EngineDriver` protocol + built-in `DockerDriver` | m-cli | done | injectable `runner` for tests; compose-first + docker-run fallback live | +| 2.3 | Declare `m_cli_engines` entry-point group; document in `PLUGIN_API_VERSION` | m-cli | done | `ENGINE_DRIVER_ENTRY_POINT_GROUP` constant + `discover_drivers()` seam; docs/plugin-development.md updated | +| 2.4 | `m doctor --fix` delegates to `m engine `; refuses destructive verbs without `--confirm` | m-cli | todo | deferred to follow-up commit — design still ambiguous for sudo'd fixes | +| 2.5 | `dist/commands.json` auto-includes `engine` namespace (verify) | m-cli | done | confirmed via `m capabilities --json` + drift gate captured the addition | +| 2.6 | Compose-detection + `docker run` fallback path | m-cli | done | implemented inside `DockerDriver._has_compose_plugin()` + `_start_via_run()` | | **3** — OCI labels + `HEALTHCHECK` | | | | | | 3a.1 | Dockerfile `LABEL org.m-dev-tools.m-test-engine.*` (protocol, bind-mount, ydb-version, image-rev) | m-test-engine | todo | | | 3a.2 | Dockerfile `HEALTHCHECK CMD $ydb_dist/mumps -run %XCMD 'write "ok",!'` | m-test-engine | todo | | @@ -229,3 +230,67 @@ skipped (pre-existing); lint + mypy clean. `m doctor --json` should treat that specific WARN as a higher priority than the rest (fix the manifest first, then everything downstream becomes resolvable). + +### 2026-05-11 — Phase 2 substantial deliverables landed + +Shipped, in order: `/m-work` runtime cutover (compose.yml + +Dockerfile + dist/lifecycle.json + dist/repo.meta.json notes + +README/AGENTS prose in m-test-engine; DockerEngine.bind_root default in +m-cli) → `EngineDriver` Protocol + `DockerDriver` (compose-first / +docker-run fallback; 19 tests) → `m engine` subcommand surface with +12 verbs (16 tests) → entry-point seam (`m_cli_engines`) → +capabilities verification. + +Phase 2.4 (`m doctor --fix` delegation) **deferred** — see followup #9 +below. The Phase 2 work as committed is a coherent shippable bundle: +the engine namespace is live (`m engine status --json`, +`m engine install`, `m engine start`, `m engine reset --confirm`, +etc.), driven entirely by the vendored manifest, with 35 new tests +and lint+mypy clean across the new modules. + +**Discoveries that affect later phases:** + +9. **`m doctor --fix` design still ambiguous for sudo'd fixes.** The + doctor checks shipped in Phase 1b emit `fix.command` arrays + including `["sudo", "systemctl", "start", "docker"]` and + `["sudo", "install", "-d", "/m-work"]`. Auto-executing sudo from + `--fix` is sketchy (password prompt, non-CI hosts, security + boundary). The plan's "delegates to m engine " framing + covers the docker-pull / docker-compose-up fixes cleanly but + doesn't help the sudo ones. + + **Followup for Phase 2 tail (stage 2.4) or Phase 5 polish:** + decide whether `--fix` is engine-verb-delegation only (with a + "manual: " line for non-engine fixes), or whether it should + shell out raw fix.command for non-sudo fixes only, or whether the + Fix dataclass should grow an `engine_verb: str | None` field that + `--fix` consults. Lean toward: add `engine_verb` field, `--fix` + invokes only those, print "manual:" for the rest. Keeps the + security boundary clear. + +10. **`m capabilities --json` doesn't recurse into nested subparsers.** + The `engine` namespace appears at the top level with `purpose`, + `options: []`, `examples: []` but no verb-by-verb breakdown. The + existing `m ci` namespace has the same shape (one nested action + `init`, not introspected). For AI-agent surface completeness, the + capabilities builder should eventually recurse — but that's a + cross-cutting change to `m_cli.capabilities`, not Phase 2-specific. + Workaround: `m engine capabilities --json` (the per-namespace verb) + emits a richer payload including verb safety classification. + + **Followup for Phase 5:** generalize the capabilities builder to + walk nested subparsers, so every namespace's verbs land in + `dist/commands.json` automatically. + +11. **`m_cli.engine` (module) vs `m_cli.engine_cli` / `m_cli.engine_driver` + (sibling modules).** Considered converting `m_cli.engine` from a + module to a package so the new engine surfaces could live under + `m_cli.engine.driver` / `m_cli.engine.cli`. Rejected for Phase 2 + because monkeypatches in existing tests address `m_cli.engine._has_*` + function bindings directly; a package conversion would break those. + Sibling modules keep change-blast small at the cost of slightly + awkward naming. Acceptable trade-off. + + **Followup for Phase 5 cleanup:** revisit the package conversion + when there's a maintenance window; the rename can land alongside + the capabilities-recursion change.