Merge: meta docs binary templates + config providers (the two neutral-docs follow-ups)

dmealing · claude · dmealing · commit 08e90a492f6d · 2026-06-02T08:15:56.000-04:00
Closes the two documented follow-ups from the neutral-metadata-docs feature so
'meta docs' is a fully standalone Tier-2 capability:

1. Framework doc templates now resolve inside the bun --compile meta binary. A
   generated plain-string module (embedded-templates.generated.ts, from the
   canonical templates/) is consulted as a fallback AFTER on-disk resolution, so
   dev/install layouts + adopter overrides are unaffected. tsc-safe; gated
   byte-identical to canonical (third representation alongside root + package copy).

2. meta docs loads metaobjects.config.ts providers (mirroring gen) so adopters
   with custom field/object types can generate docs — config stays OPTIONAL
   (config-less docs still work). A broken config is now surfaced with its real
   error instead of silently degrading. Also fixed a pre-existing shared
   config-loader bug (@metaobjectsdev/metadata missing from jiti's alias map),
   which affected gen/migrate/verify custom-provider configs too.

Co-Authored-By: Claude Opus 4.8 (1M context) &lt;noreply@anthropic.com&gt;
diff --git a/docs/superpowers/plans/2026-06-02-docs-binary-and-providers.md b/docs/superpowers/plans/2026-06-02-docs-binary-and-providers.md
@@ -0,0 +1,54 @@
+# `meta docs` binary templates + config providers — implementation plan
+
+> REQUIRED SUB-SKILL: superpowers:subagent-driven-development.
+
+**Goal:** Close the two documented follow-ups from the neutral-metadata-docs feature so `meta docs` is a fully standalone Tier-2 capability:
+1. Framework doc templates resolve inside the `bun --compile` `meta` binary (today they don't — only adopter-provided `templates/` work there).
+2. `meta docs` loads `metaobjects.config.ts` providers so adopters with custom field/object types can generate docs.
+
+**Architecture:** Both `codegen-ts` and `cli` build with `tsc` (the binary is built separately with `bun build --compile`). So embedding cannot use a `.mustache` text-import (tsc rejects it). Instead generate a plain-string module from the canonical `templates/`, imported by the framework provider with an on-disk-first / embedded-fallback chain. Config-provider loading mirrors `gen.ts`.
+
+**Tech:** TypeScript, bun 1.3.8, tsc builds, the existing `render()`/`projectProvider` chain.
+
+---
+
+## Task 1: Embedded framework-template fallback (fixes binary)
+
+**Intent:** `FrameworkTemplatesProvider` resolves the framework doc templates from the embedded map when the on-disk `templates/` dir is unavailable (the compiled binary). On-disk stays FIRST so dev/install layouts keep honoring local edits + adopter overrides.
+
+**Files:**
+- Create `scripts/generate-embedded-templates.ts` (bun script) — reads canonical `templates/docs/*.mustache`, writes `server/typescript/packages/codegen-ts/src/render-engine/embedded-templates.generated.ts` exporting `export const EMBEDDED_FRAMEWORK_TEMPLATES: Record<string, string> = { "docs/<name>.md": <content>, ... }` (key = the ref WITHOUT `.mustache`, matching how refs resolve; value = exact file text). Plain string literals only (tsc-safe). Header `// @generated from templates/docs/*.mustache — DO NOT EDIT; run scripts/generate-embedded-templates.ts`.
+- Extend `scripts/sync-doc-templates.sh` to ALSO invoke the generator (so one command syncs the package copy AND regenerates the embedded module from canonical).
+- Modify `server/typescript/packages/codegen-ts/src/render-engine/framework-provider.ts` — `FrameworkTemplatesProvider.resolve(ref)`: keep the on-disk lookup first; if it returns undefined, return `EMBEDDED_FRAMEWORK_TEMPLATES[ref]` (or undefined). Update the stale comment (lines ~33-35) that references a non-existent embedded fallback to describe the real one.
+- Test: `server/typescript/packages/codegen-ts/test/embedded-templates.test.ts` — (a) GATE: every canonical `templates/docs/*.mustache` has a matching embedded entry whose content is byte-identical (prevents drift); (b) the embedded map covers exactly the canonical set (no missing/extra); (c) simulate the binary case: a `FrameworkTemplatesProvider` whose on-disk dir is unresolved still resolves `docs/entity-page.md` + `docs/template-page.md` from the embedded map.
+
+- [ ] Step 1: Write `embedded-templates.test.ts` (drift gate + binary-fallback). Run → FAIL (generated module doesn't exist).
+- [ ] Step 2: Write `scripts/generate-embedded-templates.ts`; run it to produce `embedded-templates.generated.ts`. Wire it into `sync-doc-templates.sh`.
+- [ ] Step 3: Update `FrameworkTemplatesProvider.resolve` (on-disk → embedded fallback) + fix the stale comment.
+- [ ] Step 4: Run → PASS the new test + `bun test packages/codegen-ts` (existing docs/golden/byte-identity gates unaffected — on-disk still wins in dev). Report counts.
+- [ ] Step 5: Binary proof — `cd packages/cli && bun build ./bin/meta.ts --compile --outfile /tmp/meta-test --external @biomejs/wasm-bundler --external @biomejs/wasm-web`, then run `/tmp/meta-test docs <fixture-metadata> --out /tmp/docsout` from a dir with NO `templates/` and assert entity + template `.md` files are produced (framework templates now resolve from the embed). Report the result. (If building the full binary is too heavy/slow, at minimum assert the embedded provider resolves with on-disk forced-undefined — but attempt the real binary.)
+- [ ] Step 6: Commit. `feat(codegen-ts): embed framework doc templates so they resolve in the compiled binary`
+
+## Task 2: `meta docs` loads config providers (fixes custom types)
+
+**Intent:** `meta docs` loads `metaobjects.config.ts` (best-effort) and passes `providers` to `loadMemory`, so adopters with custom field/object types can generate docs — mirroring `gen.ts`.
+
+**Files:**
+- Modify `server/typescript/packages/cli/src/commands/docs.ts` — before `loadMemory`, best-effort `loadMetaobjectsConfig(projectRoot)` (import from `../lib/load-metaobjects-config.js`, as gen.ts does); if it succeeds, pass `{ providers: forgeConfig.providers }` to `loadMemory` (guard undefined, exactly like `gen.ts:42-43`). If config loading fails/absent, proceed config-less (docs must still work with no config — do NOT hard-fail; gen treats config as required, docs must NOT).
+- Test: `server/typescript/packages/cli/test/docs-command.test.ts` — add a case: a fixture project WITH a `metaobjects.config.ts` (or the config-loading mechanism) declaring a custom field/object type used in its metadata; `meta docs` loads + emits docs without a "type does not resolve" failure. Also keep/confirm the existing no-config case still works (config-less).
+
+- [ ] Step 1: Write the custom-type-with-config docs test (mirror how gen's config tests set up `loadMetaobjectsConfig`/a fixture config). Run → FAIL (docs currently ignores providers → custom type fails to load).
+- [ ] Step 2: Add best-effort config+providers loading to `docs.ts` (mirror gen.ts; non-fatal when absent).
+- [ ] Step 3: Run → PASS the new test + the existing no-config docs tests + `bun test packages/cli`. Report counts.
+- [ ] Step 4: Commit. `feat(cli): meta docs loads metaobjects.config.ts providers (custom types)`
+
+## Task 3: Closeout
+- [ ] Full suites: `bun test packages/codegen-ts packages/cli packages/render` green; counts.
+- [ ] Hygiene (merge-base diff): no private names/home paths/node_modules/bunfig/lockfile/dist.
+- [ ] Whole-branch code-review + simplifier; fix findings.
+- [ ] Forward-merge to main, push, remove worktree, delete branch, update memory.
+
+## Notes / guards
+- The embedded module is a GENERATED artifact gated against canonical `templates/` — edit root templates, run `sync-doc-templates.sh` (now also regenerates the embed). Three representations (root canonical, package bundled copy, embedded module) all gated byte-identical.
+- On-disk MUST stay first in `FrameworkTemplatesProvider` so adopter overrides + dev edits win; embedded is fallback only.
+- Don't touch native code generators, the render engine, other ports, or doc neutrality. Public-repo hygiene.
diff --git a/scripts/generate-embedded-templates.ts b/scripts/generate-embedded-templates.ts
@@ -0,0 +1,98 @@
+#!/usr/bin/env bun
+//
+// generate-embedded-templates.ts
+//
+// Reads the canonical doc templates at repo-root templates/docs/*.mustache and
+// emits a plain-string TS module
+//   server/typescript/packages/codegen-ts/src/render-engine/embedded-templates.generated.ts
+// mapping the provider resolve ref (path under templates/ WITHOUT the .mustache
+// suffix, e.g. "docs/entity-page.md") to the EXACT file contents.
+//
+// WHY a generated string module (not a `.mustache` text-import): codegen-ts
+// builds with `tsc`, which rejects unknown-extension text imports
+// (`import x from "./f.mustache" with { type: "text" }`). A plain TS module of
+// string literals compiles cleanly with tsc AND gets bundled into the
+// `bun build --compile` standalone `meta` binary — so the framework doc
+// templates resolve even where the on-disk templates/ dir is unavailable.
+//
+// Run via: bun run scripts/generate-embedded-templates.ts
+// (also wired into scripts/sync-doc-templates.sh so one command keeps the
+//  package copy AND this embedded module in sync with canonical.)
+//
+// A byte-identity drift test gates this output:
+//   server/typescript/packages/codegen-ts/test/embedded-templates.test.ts
+
+import { existsSync, readFileSync, readdirSync, writeFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+// Resolve the repo root relative to THIS script (walk up to the dir containing
+// both templates/ and server/). No hardcoded absolute paths.
+function findRepoRoot(start: string): string {
+  let dir = start;
+  while (true) {
+    if (existsSync(join(dir, "templates")) && existsSync(join(dir, "server"))) {
+      return dir;
+    }
+    const parent = dirname(dir);
+    if (parent === dir) {
+      throw new Error("Could not locate repo root (dir containing templates/ and server/)");
+    }
+    dir = parent;
+  }
+}
+
+const scriptDir = dirname(fileURLToPath(import.meta.url));
+const repoRoot = findRepoRoot(scriptDir);
+
+const canonicalDir = join(repoRoot, "templates", "docs");
+const outFile = join(
+  repoRoot,
+  "server",
+  "typescript",
+  "packages",
+  "codegen-ts",
+  "src",
+  "render-engine",
+  "embedded-templates.generated.ts",
+);
+
+const files = readdirSync(canonicalDir).filter((f) => f.endsWith(".mustache"));
+
+if (files.length === 0) {
+  console.error(`error: no *.mustache templates found under ${canonicalDir}`);
+  process.exit(1);
+}
+
+// ref = path under templates/ WITHOUT the .mustache suffix, matching how
+// FrameworkTemplatesProvider.resolve(ref) builds `<dir>/<ref>.mustache`.
+// e.g. templates/docs/entity-page.md.mustache -> "docs/entity-page.md".
+// Sorted by ref for deterministic, byte-stable output.
+const entries = files
+  .map((file) => {
+    const ref = `docs/${file.slice(0, -".mustache".length)}`;
+    const content = readFileSync(join(canonicalDir, file), "utf-8");
+    return { ref, content };
+  })
+  .sort((a, b) => (a.ref < b.ref ? -1 : a.ref > b.ref ? 1 : 0));
+
+const body = entries
+  // JSON.stringify emits a safe string literal — preserves newlines/quotes/
+  // unicode byte-for-byte.
+  .map((e) => `  ${JSON.stringify(e.ref)}: ${JSON.stringify(e.content)},`)
+  .join("\n");
+
+const source = `// @generated from templates/docs/*.mustache — DO NOT EDIT.
+// Regenerate: bun run scripts/generate-embedded-templates.ts (or scripts/sync-doc-templates.sh).
+//
+// Embedded framework doc templates so they resolve inside the
+// \`bun build --compile\` standalone \`meta\` binary, where the on-disk
+// \`templates/\` directory is unavailable. Keys are provider resolve refs
+// (path under templates/ minus the .mustache suffix).
+export const EMBEDDED_FRAMEWORK_TEMPLATES: Record<string, string> = {
+${body}
+};
+`;
+
+writeFileSync(outFile, source, "utf-8");
+console.log(`generated ${outFile} (${entries.length} template(s): ${entries.map((e) => e.ref).join(", ")})`);
diff --git a/scripts/sync-doc-templates.sh b/scripts/sync-doc-templates.sh
@@ -49,4 +49,10 @@ for dest in "${DESTS[@]}"; do
   done
 done
 
+# Regenerate the embedded-string TS module so the framework doc templates also
+# resolve inside the `bun build --compile` standalone `meta` binary (where the
+# on-disk templates/ dir is unavailable). Keeps the bundled package copy AND the
+# embedded module in sync with canonical from a single command. Idempotent.
+bun run "${SCRIPT_DIR}/generate-embedded-templates.ts"
+
 echo "doc templates synced from canonical root: ${SRC_DIR}"
diff --git a/server/typescript/packages/cli/src/commands/docs.ts b/server/typescript/packages/cli/src/commands/docs.ts
@@ -14,6 +14,7 @@
 import { resolve as resolvePath } from "node:path";
 import { mkdir, writeFile } from "node:fs/promises";
 import { log } from "../lib/log.js";
+import { loadMetaobjectsConfig } from "../lib/load-metaobjects-config.js";
 import { loadMemory, DEFAULT_METADATA_DIR } from "@metaobjectsdev/sdk";
 import { existsSync } from "node:fs";
 import { join } from "node:path";
@@ -88,10 +89,40 @@ export async function docsCommand(args: string[], cwd: string): Promise<number>
     : metaRoot;
   const outDir = resolvePath(metaRoot, flags.out);
 
-  // Load metadata standalone — same loader path as migrate/gen, NO gen config.
+  // Best-effort load of metaobjects.config.ts to pick up consumer-supplied
+  // providers (e.g. a project's custom field/object subtypes). Unlike `gen`,
+  // docs does NOT require a config — the Tier-2 "metadata alone" promise must
+  // hold for config-less projects. If the config is absent or invalid, fall
+  // back to defaults; the loader still surfaces a stable unknown-subtype error
+  // if the metadata genuinely uses an unregistered type.
+  let configProviders: NonNullable<Awaited<ReturnType<typeof loadMetaobjectsConfig>>["providers"]> | undefined;
+  // The config lives alongside metaobjects/ at the metadata root (metaRoot);
+  // projectRoot only diverges when --templates overrides the template lookup.
+  // Only attempt the load when the file is actually present: absence is the
+  // expected config-less case (stay silent), but a config that EXISTS yet fails
+  // to load is surfaced as a warning rather than silently degrading to
+  // provider-less docs — otherwise a custom-type project would later fail with a
+  // cryptic unknown-subtype error instead of the real config error.
+  if (existsSync(join(metaRoot, "metaobjects.config.ts"))) {
+    try {
+      const forgeConfig = await loadMetaobjectsConfig(metaRoot);
+      configProviders = forgeConfig.providers;
+    } catch (err) {
+      log.warn(
+        `docs: metaobjects.config.ts failed to load (${(err as Error).message}); ` +
+          `generating docs without its providers`,
+      );
+      configProviders = undefined;
+    }
+  }
+
+  // Load metadata standalone — same loader path as migrate/gen. Threads any
+  // consumer providers from the config so custom types resolve.
   let root;
   try {
-    root = await loadMemory(metaRoot);
+    root = await loadMemory(metaRoot, {
+      ...(configProviders !== undefined ? { providers: configProviders } : {}),
+    });
   } catch (err) {
     const msg = (err as Error).message;
     if (!existsSync(join(metaRoot, DEFAULT_METADATA_DIR))) {
diff --git a/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts b/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts
@@ -32,6 +32,13 @@ const CLI_PKG_PATHS: Record<string, { dist: string; src: string }> = {
     dist: "node_modules/@metaobjectsdev/codegen-ts/dist/index.js",
     src: "node_modules/@metaobjectsdev/codegen-ts/src/index.ts",
   },
+  // Consumer configs that ship custom MetaDataTypeProviders import the type
+  // primitives (TypeId, MetaField, TYPE_* …) from here. Aliased to the CLI's
+  // own copy so the user's project needn't declare it as a direct dependency.
+  "@metaobjectsdev/metadata": {
+    dist: "node_modules/@metaobjectsdev/metadata/dist/index.js",
+    src: "node_modules/@metaobjectsdev/metadata/src/index.ts",
+  },
   "@metaobjectsdev/codegen-ts/generators": {
     dist: "node_modules/@metaobjectsdev/codegen-ts/dist/generators/index.js",
     src: "node_modules/@metaobjectsdev/codegen-ts/src/generators/index.ts",
@@ -94,6 +101,7 @@ export async function loadMetaobjectsConfig(projectRoot: string): Promise<Metaob
     interopDefault: true,
     alias: {
       "@metaobjectsdev/codegen-ts": resolveCliPkg("@metaobjectsdev/codegen-ts"),
+      "@metaobjectsdev/metadata": resolveCliPkg("@metaobjectsdev/metadata"),
       "@metaobjectsdev/codegen-ts/generators": resolveCliPkg("@metaobjectsdev/codegen-ts/generators"),
       "@metaobjectsdev/codegen-ts-react": resolveCliPkg("@metaobjectsdev/codegen-ts-react"),
       "@metaobjectsdev/codegen-ts-tanstack": resolveCliPkg("@metaobjectsdev/codegen-ts-tanstack"),
diff --git a/server/typescript/packages/cli/test/docs-command.test.ts b/server/typescript/packages/cli/test/docs-command.test.ts
@@ -52,6 +52,67 @@ async function project(): Promise<string> {
   return root;
 }
 
+// Metadata using a CUSTOM field subtype (`field.geopoint`) that only a
+// consumer-supplied provider in metaobjects.config.ts can resolve.
+const CUSTOM_META = {
+  "metadata.root": {
+    package: "acme::geo",
+    children: [
+      {
+        "object.value": {
+          name: "Place",
+          children: [
+            { "field.string": { name: "name" } },
+            // Resolves ONLY when the config's geopoint provider is loaded.
+            { "field.geopoint": { name: "location" } },
+          ],
+        },
+      },
+    ],
+  },
+};
+
+// A metaobjects.config.ts that registers the `field.geopoint` subtype via a
+// consumer provider (mirrors how an adopter ships custom types to gen/migrate).
+const CUSTOM_CONFIG = [
+  `import { defineConfig } from "@metaobjectsdev/codegen-ts";`,
+  `import { entityFile } from "@metaobjectsdev/codegen-ts/generators";`,
+  `import { TypeId, TYPE_FIELD, MetaField } from "@metaobjectsdev/metadata";`,
+  `const geoProvider = {`,
+  `  id: "test-geo",`,
+  `  dependencies: ["metaobjects-core-types"],`,
+  `  registerTypes(registry) {`,
+  `    registry.register({`,
+  `      typeId: new TypeId(TYPE_FIELD, "geopoint"),`,
+  `      description: "A geographic point field",`,
+  `      factory: (typeId, name) => new MetaField(typeId, name),`,
+  `      childRules: [],`,
+  `      attributes: [],`,
+  `    });`,
+  `  },`,
+  `};`,
+  `export default defineConfig({`,
+  `  outDir: "out",`,
+  `  dialect: "sqlite",`,
+  `  generators: [entityFile()],`,
+  `  providers: [geoProvider],`,
+  `});`,
+].join("\n");
+
+/** Project root with metadata using a custom type + a config that provides it. */
+async function customTypeProject(): Promise<string> {
+  const root = await mkdtemp(join(tmpdir(), "meta-docs-custom-"));
+  dirs.push(root);
+  await mkdir(join(root, "metaobjects"), { recursive: true });
+  await writeFile(
+    join(root, "metaobjects", "meta.json"),
+    JSON.stringify(CUSTOM_META),
+    "utf8",
+  );
+  await writeFile(join(root, "metaobjects.config.ts"), CUSTOM_CONFIG, "utf8");
+  return root;
+}
+
 afterAll(async () => {
   for (const d of dirs) await rm(d, { recursive: true, force: true });
 });
@@ -120,6 +181,55 @@ describe("meta docs — standalone neutral metadata docs", () => {
     expect(summary).toContain(out);
   });
 
+  test("loads metaobjects.config.ts providers so custom types resolve", async () => {
+    const root = await customTypeProject();
+    const out = join(root, "out-custom");
+
+    // The metadata uses field.geopoint, which is NOT a core subtype — it only
+    // resolves via the provider declared in metaobjects.config.ts. Before docs
+    // loaded the config providers, this load failed with an unknown-subtype
+    // error and docs returned non-zero.
+    const code = await docsCommand([root, "--out", out], root);
+    expect(code).toBe(0);
+
+    const files = await readdir(out);
+    expect(files).toContain("Place.md");
+  });
+
+  test("a broken metaobjects.config.ts is surfaced (not silently swallowed), docs still generate", async () => {
+    // Basic metadata (no custom types) + a config that THROWS on load. Docs must
+    // still succeed config-less, but the real config error must be surfaced as a
+    // warning rather than silently degrading to provider-less docs.
+    const root = await project();
+    await writeFile(
+      join(root, "metaobjects.config.ts"),
+      "throw new Error('broken config boom');\n",
+      "utf8",
+    );
+    const out = join(root, "out-brokencfg");
+
+    const errLogged: string[] = [];
+    const origErr = console.error;
+    console.error = (...args: unknown[]) => {
+      errLogged.push(args.map(String).join(" "));
+    };
+    let code: number;
+    try {
+      code = await docsCommand([root, "--out", out], root);
+    } finally {
+      console.error = origErr;
+    }
+
+    expect(code).toBe(0); // config-less generation still works
+    const files = await readdir(out);
+    expect(files.length).toBeGreaterThan(0);
+    // The config failure is SURFACED (warning on stderr), not silently swallowed,
+    // and carries the loader's real diagnostic so the user can fix it.
+    const stderr = errLogged.join("\n");
+    expect(stderr).toContain("metaobjects.config.ts failed to load");
+    expect(stderr).toContain("generating docs without its providers");
+  });
+
   test("exits non-zero with a clear message when metadata is missing", async () => {
     const empty = await mkdtemp(join(tmpdir(), "meta-docs-empty-"));
     dirs.push(empty);
diff --git a/server/typescript/packages/codegen-ts/src/render-engine/embedded-templates.generated.ts b/server/typescript/packages/codegen-ts/src/render-engine/embedded-templates.generated.ts
diff --git a/server/typescript/packages/codegen-ts/src/render-engine/framework-provider.ts b/server/typescript/packages/codegen-ts/src/render-engine/framework-provider.ts
diff --git a/server/typescript/packages/codegen-ts/test/embedded-templates.test.ts b/server/typescript/packages/codegen-ts/test/embedded-templates.test.ts
