Skip to content

Commit b462644

Browse files
dmealingclaude
dmealing
and
claude
committed
docs(spec): Cloudflare D1 dialect design for meta migrate (TypeScript)
Adds a `dialect: "d1"` peer to `postgres`/`sqlite` that reuses the existing diff/emit pipeline and swaps three I/O seams: wrangler.toml-based connection, wrangler-CLI introspection, and Wrangler-native migration writer (`migrations/<seq>_<slug>.sql` + `.down/` sidecar). Includes an opt-in `--apply` hook that shells `wrangler d1 migrations apply` and a thin D1-safety post-pass over the SQLite emit (strip explicit txns, reject ATTACH/VACUUM, preserve PRAGMA bookends). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 102ff14 commit b462644

1 file changed

Lines changed: 283 additions & 0 deletions

File tree

‎docs/superpowers/specs/2026-05-24-meta-migrate-d1-dialect-design.md‎

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
# `meta migrate` — Cloudflare D1 dialect (TypeScript)
2+
3+
**Status:** Design — plan-of-record
4+
**Author:** Claude + Doug
5+
**Date:** 2026-05-24
6+
**Scope:** TypeScript (`@metaobjectsdev/migrate-ts` + `@metaobjectsdev/cli`)
7+
**Depends on:** existing `sqlite` dialect pipeline
8+
9+
## Goal
10+
11+
Add a `dialect: "d1"` to `meta migrate` so a TypeScript project targeting Cloudflare D1 can go from metadata to applied schema in one command, in Wrangler's native migration layout, without leaving the Wrangler-native workflow developers already use.
12+
13+
## Why a new dialect
14+
15+
D1 is SQLite at the SQL level, but it diverges from generic SQLite in three places that matter to `meta migrate`:
16+
17+
- **Connection.** Wrangler owns the database — there's no connection URL. The toolchain is `wrangler d1 execute <binding> [--local|--remote]`, configured in `wrangler.toml`.
18+
- **Migration file layout.** Wrangler expects forward-only `migrations/<seq>_<slug>.sql`, tracks applied migrations in a `d1_migrations` table, and ignores anything else.
19+
- **SQL constraints.** D1 rejects explicit `BEGIN/COMMIT/SAVEPOINT` in user SQL (it wraps each file itself), forbids `ATTACH`/`VACUUM`, and has per-statement size limits.
20+
21+
Modelling this as a flag on the existing `sqlite` dialect would fork three internal code paths on `target`. A new dialect keeps `sqlite` meaning "generic SQLite via libsql" and gives D1 a clean lane that other Wrangler-native dialects (hypothetical future Turso, Cloudflare Hyperdrive) can mimic.
22+
23+
## Forward-compatibility posture
24+
25+
The design preserves room for richer Postgres/Oracle work without locking us in:
26+
27+
- The diff/emit core (`expected-schema.ts`, `diff/`, `sql-type.ts`) is unchanged and stays state-based, not history-based. D1's `d1_migrations` history table is a sidecar concern owned by the `--apply` hook, not by the pipeline.
28+
- Output format is dialect-pluggable. A future `writeMigrationFlyway` (`V<n>__<slug>.sql`) or Liquibase writer slots in next to `writeMigrationD1` without touching the pipeline.
29+
- The CLI flag `--remote` is D1-specific now; if a future Postgres/Oracle design needs richer multi-environment support, the natural shape is `--env=<name>` of which `--remote` becomes a D1-vocabulary synonym. This is a forward note, not a commitment.
30+
31+
Deliberate constraint preserved (not introduced): `meta migrate` is single-target per invocation. The D1 design keeps that.
32+
33+
## Architecture
34+
35+
```
36+
metadata ──┐
37+
▼
38+
buildExpectedSchema(d1) ← d1 maps to sqlite shape
39+
│
40+
▼
41+
┌─────────────────┐ ┌────────────────┐ ┌─────────────────────────┐
42+
│ wrangler.toml │ │ diff() │ │ renderD1 = renderSqlite │
43+
│ resolver ├──► │ (shared) ├──► │ + D1 safety post-pass │
44+
└─────────────────┘ └────────────────┘ └─────────────────────────┘
45+
│ │
46+
▼ ▼
47+
introspectD1 (shell-out writeMigrationD1(up, down)
48+
to `wrangler d1 execute ├─ migrations/<seq>_<slug>.sql
49+
--json --command "..."`) └─ migrations/.down/<seq>_<slug>.sql
50+
│
51+
▼
52+
(optional, --apply)
53+
`wrangler d1 migrations apply`
54+
```
55+
56+
Three swap points; the diff/emit core is untouched.
57+
58+
## Connection + introspection
59+
60+
### Binding resolution
61+
62+
`meta migrate` finds the D1 database via Wrangler's own config. Resolution order:
63+
64+
1. `--d1 <BINDING>` CLI flag (explicit).
65+
2. `migrate.d1.binding` in `.metaobjects/config.json`.
66+
3. Auto: parse `wrangler.toml` (or `wrangler.jsonc`); if it has exactly one `[[d1_databases]]` entry, use it. If zero or 2+, error with a list of available bindings.
67+
68+
From the wrangler config block we capture `binding`, `database_name`, `database_id`, and `migrations_dir` (defaults to `migrations`).
69+
70+
`--db <url>` is rejected with a clear error for `dialect: "d1"` — Wrangler owns connection.
71+
72+
### Introspector
73+
74+
`migrate-ts/src/introspect/d1.ts` exposes `introspectD1(opts)` and returns the same `SchemaSnapshot` shape as `introspectSqlite`. It works by:
75+
76+
1. Issuing the same set of catalog queries against `sqlite_master`, `PRAGMA table_info`, `PRAGMA index_list`, `PRAGMA foreign_key_list` that `introspectSqlite` issues today.
77+
2. Running each via `execFile("wrangler", ["d1", "execute", "<binding>", "--local"|"--remote", "--json", "--command", "<sql>"], { cwd })` and parsing the JSON envelope (`{ results: [{ ... }] }`).
78+
3. Mapping rows back into `TableDescriptor` / `ColumnDescriptor` / `IndexDescriptor` / `FkDescriptor` using the same logic SQLite uses — only the transport differs.
79+
80+
`snapshot.meta.sqliteVersion` is populated from `SELECT sqlite_version()` over the same channel, so any version-gated decisions downstream (recreate-and-copy threshold) use D1's actual reported version.
81+
82+
### Why shell-out, not import wrangler as a library
83+
84+
Wrangler's programmatic API is unstable; shell-out insulates us from internal churn. Cost: fork overhead per query. Mitigations:
85+
86+
- Local introspection is negligible-cost (~ms per query).
87+
- Remote introspection makes one HTTPS round-trip per query; we print a per-query progress line and emit a one-line `wrangler d1 execute --remote` aggregate timing in the summary.
88+
- Future optimization (out of scope for v1): batch read queries via `wrangler d1 execute --file`.
89+
90+
### Error surfaces
91+
92+
- Wrangler not on PATH → "install wrangler: `npm i -D wrangler`".
93+
- No `wrangler.toml` in cwd or parents → "expected wrangler.toml; pass `--d1 <binding>` to bypass".
94+
- Binding not found → list available bindings.
95+
- Wrangler unauthenticated for `--remote` → forward wrangler's own error; suggest `wrangler login`.
96+
- Empty introspection on `--remote` → probe with `SELECT name FROM sqlite_master LIMIT 1` and distinguish "empty DB" from "wrong db_id".
97+
98+
### What stays shared with `sqlite`
99+
100+
`buildExpectedSchema(metadata, { dialect: "d1" })` maps internally to `"sqlite"` — D1 *is* SQLite for schema-shape purposes. Nothing in `expected-schema.ts`, `diff/`, or `sql-type.ts` knows D1 exists. The split is purely at I/O.
101+
102+
## Emit + D1-safety pass
103+
104+
### Wrapper structure
105+
106+
`migrate-ts/src/emit/d1.ts`:
107+
108+
```ts
109+
export function renderD1(changes, expectedSchema, actualMeta?): EmitResult {
110+
const sqliteResult = renderSqlite(changes, expectedSchema, actualMeta);
111+
return {
112+
up: applyD1SafetyPass(sqliteResult.up),
113+
down: applyD1SafetyPass(sqliteResult.down),
114+
recreatedTables: sqliteResult.recreatedTables,
115+
};
116+
}
117+
```
118+
119+
Both up and down get the safety pass — the down sidecar isn't run by Wrangler, but a human piping it through `wrangler d1 execute --file` needs it to be D1-safe too.
120+
121+
`emit/index.ts` dispatch adds `case "d1": return renderD1(...)`.
122+
123+
### Safety transforms
124+
125+
1. **Strip explicit transactions.** Remove `BEGIN TRANSACTION;` / `BEGIN;` / `COMMIT;` / `ROLLBACK;` at statement boundaries (case-insensitive). Wrangler wraps each file in its own transaction; nested ones are rejected.
126+
2. **Drop `SAVEPOINT` / `RELEASE` / `ROLLBACK TO`.** Same family. Defensive — not generated today.
127+
3. **Preserve `PRAGMA foreign_keys = OFF/ON`.** Wrangler sends a migration file as a single D1 batch, so PRAGMAs are expected to persist for the file's duration and the recreate-and-copy pattern should work as-is. **Verify during implementation:** the safety-pass test suite includes a recreate-and-copy fixture exercised end-to-end against `wrangler d1 execute --local` to confirm. If D1 silently drops the PRAGMA across statements in a batch, fall back to emitting per-statement FK-disable + re-enable bookends around each recreate-and-copy block (or, last resort, gate FK-touching DDL behind a typed error).
128+
4. **Reject `ATTACH DATABASE` / `DETACH DATABASE` / `VACUUM` at emit time** with a typed error. Not generated today; this is a future-proofing guard so a regression fails early with a clear message instead of at apply time.
129+
5. **Warn on statement size** if any statement exceeds 100 KB (the `wrangler d1 execute` limit). Schema DDL never gets close; data migrations could. Warn, don't block.
130+
131+
### Recreate-and-copy unchanged
132+
133+
`renderSqlite`'s version-gated decision (ALTER vs. recreate-and-copy via `__new_<table>`) uses `actualMeta.sqliteVersion`. `introspectD1` populates that from D1's actual version, so the right code path is selected automatically. No D1 special case.
134+
135+
### Testing the pass
136+
137+
Unit tests in `migrate-ts/test/emit/d1.test.ts` cover each transform with explicit in/out pairs (BEGIN stripped, ATTACH rejected with typed error, PRAGMA preserved verbatim, size warning fires above threshold). A snapshot test feeds `renderD1` a known `Change[]` and asserts the output matches the sqlite emit modulo the safety transforms. We do not duplicate `emit/sqlite.ts`'s suite.
138+
139+
### Why post-pass, not parallel emitter
140+
141+
Forking `renderSqlite` into `renderD1` would duplicate ~1000 lines and add maintenance liability every time SQLite emit evolves. The post-pass is ~50 lines and is text-equivalent to "sqlite minus what D1 rejects." If D1 ever needs structurally different DDL (not just text edits), we split then.
142+
143+
## Migration writer
144+
145+
### Layout
146+
147+
```
148+
migrations/
149+
├── 0001_init.sql ← up SQL (forward, Wrangler-discoverable)
150+
├── 0002_add-shipping.sql
151+
├── 0003_drop-legacy.sql
152+
└── .down/
153+
├── 0001_init.sql ← down SQL (sidecar; Wrangler doesn't recurse)
154+
├── 0002_add-shipping.sql
155+
└── 0003_drop-legacy.sql
156+
```
157+
158+
### Sequence assignment
159+
160+
Scan `migrations/*.sql` (top-level only), parse the leading `<digits>_` prefix, pick `max + 1`, pad to 4 digits. First-ever migration is `0001`. Pre-existing wrangler migrations from another source are honored — we slot in after them.
161+
162+
### Directory resolution
163+
164+
Precedence (first non-null wins):
165+
166+
1. `--out-dir` CLI flag.
167+
2. `migrate.outDir` in `.metaobjects/config.json`.
168+
3. `migrations_dir` from the captured `wrangler.toml` binding.
169+
4. `migrations/` (Wrangler default).
170+
171+
### Down sidecar
172+
173+
Down SQL lands at `migrations/.down/<seq>_<slug>.sql` with the same filename as the up file. Wrangler doesn't recurse, so it's invisible to `wrangler d1 migrations apply`. We do not touch `.gitignore` automatically — down files are useful artifacts for review, and projects that want them ignored can add `migrations/.down/` themselves.
174+
175+
`writeMigrationD1(emitResult, opts)` lives next to `writeMigration` in `write-migration.ts`. Existing `writeMigration` is unchanged.
176+
177+
## Apply hook
178+
179+
Opt-in via `--apply` flag or `migrate.d1.autoApply: true`.
180+
181+
After files are written successfully, shell:
182+
183+
```
184+
wrangler d1 migrations apply <binding> [--local|--remote] --config <wrangler-config-path>
185+
```
186+
187+
Output streams through `log`. The local/remote target is whatever drove introspection in the same run — diff target and apply target are the same environment by design. No flag combinations for "diff local, apply remote" in v1.
188+
189+
### Production confirmation
190+
191+
`--apply --remote` (without `--dry-run`) prints a confirmation banner naming the database (`Applying to remote D1 'myapp-prod' (database_id=abc...)`) and pauses 2 seconds before invoking wrangler, unless `--yes` is also passed. Mirrors wrangler's own production guard.
192+
193+
### Exit codes
194+
195+
- `0` — wrote files and (if `--apply`) applied cleanly.
196+
- `1` — diff produced blocked/ambiguous changes, or apply failed. Files are still written; rollback is the user's call.
197+
- `2` — config/auth/wrangler-not-installed error before any work happens.
198+
199+
## CLI + config surface
200+
201+
### CLI flags added
202+
203+
```
204+
--dialect d1 selects this pipeline
205+
--d1 <binding> explicit binding from wrangler.toml
206+
--remote target remote D1 (default: local)
207+
--apply run `wrangler d1 migrations apply` after write
208+
--yes skip the --remote --apply confirmation pause
209+
```
210+
211+
Existing flags (`--slug`, `--dry-run`, `--allow`, `--out-dir`, `--on-ambiguous`) work as today.
212+
213+
### Config block
214+
215+
`.metaobjects/config.json`:
216+
217+
```jsonc
218+
{
219+
"migrate": {
220+
"dialect": "d1",
221+
"d1": {
222+
"binding": "DB", // optional; overrides auto-detect
223+
"remote": false, // default target
224+
"autoApply": false, // default off
225+
"wranglerConfigPath": "wrangler.toml"
226+
},
227+
"outDir": "migrations" // optional; otherwise from wrangler.toml
228+
}
229+
}
230+
```
231+
232+
### `meta init --d1`
233+
234+
Adds the `dialect: "d1"` block and prompts for the binding (auto-fills from `wrangler.toml` if present). Existing `meta init` flow is unchanged.
235+
236+
## Testing
237+
238+
| Suite | Path | Covers |
239+
|---|---|---|
240+
| Safety pass | `migrate-ts/test/emit/d1.test.ts` | BEGIN strip, ATTACH reject, PRAGMA preserve, size warn |
241+
| Introspect | `migrate-ts/test/introspect/d1.test.ts` | wrangler invoked with right args; JSON → SchemaSnapshot mapping |
242+
| Writer | `migrate-ts/test/write-migration-d1.test.ts` | sequence numbering, `.down/` sidecar, dir resolution |
243+
| CLI | `cli/test/commands/migrate-d1.test.ts` | wrangler.toml parsing, binding resolution errors, `--remote` + `--apply` interactions, confirmation banner |
244+
245+
`execFile` is mocked in introspect tests — no live wrangler dependency in default CI. An opt-in `bun test:d1-live` script targets a real account when `CLOUDFLARE_API_TOKEN` is set, but isn't part of the default suite.
246+
247+
## Out of scope (v1)
248+
249+
- Direct Cloudflare HTTP API path (wrangler CLI only; add later if dependency becomes painful).
250+
- Reading `d1_migrations` table to skip already-applied migrations (diff is schema-state-based; wrangler tracks history itself).
251+
- Multi-environment diff/apply in one invocation (single target per run, as today).
252+
- Data migrations (schema only, as today).
253+
- Wrangler version pinning / probing (assume `>=3`; document the floor).
254+
- Batching introspection queries via `wrangler d1 execute --file` (perf optimization deferred).
255+
256+
## File-level change summary
257+
258+
New files:
259+
260+
- `server/typescript/packages/migrate-ts/src/introspect/d1.ts`
261+
- `server/typescript/packages/migrate-ts/src/emit/d1.ts`
262+
- `server/typescript/packages/migrate-ts/src/emit/d1-safety-pass.ts`
263+
- `server/typescript/packages/migrate-ts/src/write-migration-d1.ts`
264+
- `server/typescript/packages/migrate-ts/src/wrangler-config.ts` (wrangler.toml/jsonc parser)
265+
- `server/typescript/packages/cli/src/lib/wrangler.ts` (CLI invocation helpers, mocked in tests)
266+
- Test files mirroring each of the above.
267+
268+
Touched files:
269+
270+
- `server/typescript/packages/migrate-ts/src/types.ts` — extend `Dialect` to include `"d1"`.
271+
- `server/typescript/packages/migrate-ts/src/expected-schema.ts` — map `"d1"` → SQLite expected-schema path.
272+
- `server/typescript/packages/migrate-ts/src/emit/index.ts` — dispatch `case "d1"`.
273+
- `server/typescript/packages/migrate-ts/src/introspect/index.ts` — dispatch `case "d1"`.
274+
- `server/typescript/packages/migrate-ts/src/index.ts` — re-export D1 surface.
275+
- `server/typescript/packages/cli/src/commands/migrate.ts` — branch on `dialect === "d1"` for wrangler-based connection (skip `buildKyselyFromUrl`).
276+
- `server/typescript/packages/cli/src/lib/args.ts` — add `--d1`, `--remote`, `--apply`, `--yes`.
277+
- `server/typescript/packages/cli/src/lib/config.ts` — add `migrate.d1` block.
278+
- `server/typescript/packages/cli/src/commands/init.ts` — add `--d1` init path.
279+
- `docs/RELEASING.md` / `README.md` — document the new dialect.
280+
281+
## Open questions
282+
283+
None — all design decisions resolved during brainstorming.

0 commit comments

Comments
 (0)