-
Notifications
You must be signed in to change notification settings - Fork 1
feat: adopt the upstream alchemy/Prisma provider #197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
wmadden-electric
wants to merge
25
commits into
main
Choose a base branch
from
claude/alchemy-prisma-composer-eval-93897e
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
150229a
docs(drive): open alchemy-provider-adoption project — spec, plan, des…
wmadden-electric adab2f8
feat(lowering): alchemy beta.67 foundation — bump, PrismaComposer typ…
wmadden-electric 6f3673f
feat(lowering): adopt upstream alchemy/Prisma postgres resources
wmadden-electric d3e2579
docs(drive): draft upstream alchemy PR body
wmadden-electric 2f1b140
docs(drive): rewrite upstream PR body — grounding example, narrative,…
wmadden-electric 97e33fe
docs(drive): unwrap PR body — GitHub renders single newlines as breaks
wmadden-electric e789453
docs(drive): add embedder DX sample to upstream PR body
wmadden-electric 7d61d45
feat(lowering): adopt upstream alchemy/Prisma compute resources
wmadden-electric e006eb0
docs(design): ADR-0043 — Prisma Cloud resources come from the upstrea…
wmadden-electric c107f7f
fix(lowering): report migrated poison rows as retained, correct migra…
wmadden-electric 39a727a
feat(lowering): env changes always reach the running app again
wmadden-electric f781a00
docs(drive): Composer PR body
wmadden-electric d54e2b2
docs(design): rewrite ADR-0043 — grounding example, durable statement…
wmadden-electric c16fc70
Merge remote-tracking branch 'origin/main' into claude/alchemy-prisma…
wmadden-electric e69ce48
docs(design): DATABASE_URL is self-healed on first deploy, not absent
wmadden-electric 09d46a0
feat(lowering): claim DATABASE_URL with poison values at provision
wmadden-electric 7cffe00
docs: DATABASE_URL is claimed with a poison value at provision
wmadden-electric ce6c2cd
docs: address PR 197 review — stale names, propagation claims, --fres…
wmadden-electric 5b0e934
feat(lowering): deployment replaced when its environment changes, reu…
wmadden-electric 6c09b91
Merge remote-tracking branch 'origin/main' into claude/alchemy-prisma…
wmadden-electric 3949034
refactor(local-target): one implementation of the shared upstream-att…
wmadden-electric 6724667
fix(lowering): retire only LEGACY poison rows, decided by the props s…
wmadden-electric 377cb85
fix(lowering): migrate a replaced poison row's old generation before …
wmadden-electric e28acca
Merge branch 'main' into claude/alchemy-prisma-composer-eval-93897e
wmadden-electric c434e9d
fix(deps): pin alchemy's floating effect companions to the beta.100 c…
wmadden-electric File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
44 changes: 44 additions & 0 deletions
44
.drive/projects/alchemy-provider-adoption/assets/composer-pr-body.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| Composer's six hand-written Alchemy resources for Prisma Cloud are gone. Deploys and local dev now run on the official `alchemy/Prisma` provider (alchemy 2.0.0-beta.67): | ||
|
|
||
| ```ts | ||
| // lowering/src/providers.ts — the whole live wiring is now composition, not implementation | ||
| Layer.mergeAll( | ||
| Prisma.ProjectProvider(), | ||
| Prisma.DatabaseProvider(), | ||
| Prisma.ConnectionProvider(), | ||
| Prisma.AppProvider(), | ||
| Prisma.DeploymentProvider(), | ||
| Prisma.EnvironmentVariableProvider(), | ||
| // still ours until upstream PR alchemy-run/alchemy#1061 releases: | ||
| BucketProvider(), BucketKeyProvider(), | ||
| ) | ||
| ``` | ||
|
|
||
| Why: upstream tracks the Management API so we don't, and its deploy lifecycle is better than ours was — cleanup of failed deployments, terminal-status fast-fail, post-promote endpoint observation. Decision record: ADR-0043. | ||
|
|
||
| ## What changed | ||
|
|
||
| - **Foundation** — alchemy beta.59 → beta.67 (plus the forced effect beta.100 train). Our provider collection tag and remaining resource type-ids renamed to `PrismaComposer.*`; the old ids are aliases so existing state rows resolve. | ||
| - **Postgres family** — upstream `Project`/`Database`/`Connection` classes, driven by our own auth layer (`PrismaEnvironment` from `PRISMA_SERVICE_TOKEN`, no interactive profile store; one base-URL resolver shared with our SDK client). Branch stages create their database attached with a generated physical name (upstream correctly refuses explicit-name-plus-branch; verified against PDP source). `directConnectionString` is bound explicitly — upstream's `databaseUrl` is pooled-first. | ||
| - **Compute family** — upstream's low-level `App`/`Deployment`/`EnvironmentVariable`, not composite `Compute`: the `COMPOSER_*_ORIGIN` self-edge needs the App to exist before env rows, and `Deployment` has no build path at all (ADR-0005 by structure). The env→deployment ordering edge rides the deployment's `app` prop as an Output (`deployment-edge.ts`) — riding `artifactPath` would silently skip code deploys when a new env row lands in the same deploy (proven with tests against alchemy's real Output machinery, and re-proven live). | ||
| - **Env changes always ship** (`always-redeploy.ts`) — the deploy hook hard-links the artifact into a per-deploy-generation path so every deploy replaces the deployment, restoring the pre-existing guarantee that a rotated value reaches the running app. Cost: one deployment replacement per service per deploy, same as before this PR's base. Removed at a marked seam when upstream's `Deployment.redeployOn` (in #1061) releases. | ||
| - **Legacy state migrates on read** (`state/legacy-resources.ts`) — old type-ids and attribute shapes rewrite in the hosted store; the retired poison `DATABASE_URL` rows are reported `retained` (state row dropped, platform variable untouched). The platform's seeded `DATABASE_URL` is no longer overwritten; the authoring-side name ban remains. | ||
| - **Local dev unchanged in shape** — the local target binds upstream's resource classes to our emulators at the same seam (ADR-0041). | ||
|
|
||
| ## Verified | ||
|
|
||
| - Full suites green at every commit (build 36/36, typecheck 74/74, tests 62/63 with the one known dev-emulators flake; cast delta −8). | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| - Deployed smoke against real Prisma Cloud: fresh deploy + 2/2 smoke, idempotent redeploy (zero churn), the code-plus-new-var scenario that motivated the edge design (deployment replaced, new code live), clean destroy. | ||
| - A genuinely legacy stage (deployed from this PR's merge-base, redeployed from this branch): adopted in place — same database id, app ids, URLs — one-time deployment reship, then steady state byte-identical to a fresh stack. Clean destroy. | ||
|
|
||
| ## Operator notes (also in docs/guides/deploying.md) | ||
|
|
||
| - First deploy after upgrading replaces each service's deployment once. | ||
| - Branch-stage databases migrate to generated physical names; the database's *default* connection credentials rotate once (the app's own connection is unaffected). | ||
| - Migrated stages keep the legacy `"-"` placeholder rows on the platform until manually removed; the guide has the exact calls. | ||
|
|
||
| ## Follow-up (tracked in TML-3156) | ||
|
|
||
| When upstream PR alchemy-run/alchemy#1061 merges and releases: bump alchemy, delete our bucket resources, drop the alchemy pnpm patch, and swap `always-redeploy.ts` for `Deployment.redeployOn`. | ||
|
|
||
| 🤖 Generated with [Claude Code](https://claude.com/claude-code) | ||
122 changes: 122 additions & 0 deletions
122
.drive/projects/alchemy-provider-adoption/assets/upstream-pr-body.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| This PR completes three gaps in the Prisma provider that show up the moment a real framework embeds it. After it, this works: | ||
|
|
||
| ```ts | ||
| import * as Alchemy from "alchemy"; | ||
| import * as Prisma from "alchemy/Prisma"; | ||
| import { postgresState } from "alchemy/State/PostgresState"; | ||
| import * as Effect from "effect/Effect"; | ||
|
|
||
| export default Alchemy.Stack( | ||
| "Media", | ||
| { | ||
| // Bring your own local emulation in `alchemy dev`; live stays live. | ||
| providers: Prisma.providers({ dev: myEmulatorProviders() }), | ||
| // Durable, *locked* state — what Compute's own docs ask for. | ||
| state: postgresState({ dsn: process.env.STATE_DSN! }), | ||
| }, | ||
| Effect.gen(function* () { | ||
| const project = yield* Prisma.Project("app", {}); | ||
|
|
||
| // Object storage: the last deferred Management API surface. | ||
| const bucket = yield* Prisma.Bucket("media", { project }); | ||
| const key = yield* Prisma.BucketKey("media-rw", { | ||
| bucket, | ||
| role: "read_write", | ||
| }); | ||
|
|
||
| return { endpoint: key.endpoint, bucket: key.bucketName }; | ||
| }), | ||
| ); | ||
| ``` | ||
|
|
||
| The three ship together because they're the set a production embedder needs at once — we (Prisma Composer, the framework layer over Prisma Cloud) are deleting our own resource implementations in favor of this provider, and these were the three things we couldn't do with it. Each is an independent commit-sized concern; happy to split into separate PRs if you'd rather review them that way. | ||
|
|
||
| ## Object storage: `Prisma.Bucket` / `Prisma.BucketKey` | ||
|
|
||
| The 7 `/v1/buckets` routes were the provider's only deferred Management API surface. This adds them to the client/operations layer and puts two resources on top, shaped like their siblings (`Database`/`Connection` conventions throughout: `Refs` string-or-resource references, replace-on-identity-change diff, identity-verified delete, real `list` for nuke). | ||
|
|
||
| The interesting part is the key secret. The API returns `secretAccessKey` exactly once, at create — it can never be re-read. So persisted state is authoritative for the secret (the `Connection` pattern), which raises the crash-window question: what if the process dies after `POST …/keys` succeeds but before state is written? A naive retry would mint a second, working, never-expiring credential that no state row references and `list` can't surface. To close that, keys are created under a deterministic `instanceId`-derived physical name and looked up by name before create; a hit on retry means "create succeeded, response lost" — that orphan's secret is gone for good, so it is revoked and one fresh key is minted. Existence is re-verified on read/reconcile (secrets always from state), so a key revoked in the Console reads as gone instead of haunting the stack. | ||
|
|
||
| Docs: `prisma/data/buckets` page + sidebar entry, mirroring connections. | ||
|
|
||
| ## An embedder seam for dev mode: `providers({ dev })` + `liveProviders()` | ||
|
|
||
| `Prisma.providers()` picks dev-vs-live internally. That's the right default, but it's closed: an embedder with richer local emulation (we run a compute emulator that supervises the real artifact, a local S3, and a persistent dev Postgres) can't swap the dev half without rebuilding the client/auth wiring by hand. Two additive changes, defaults untouched: | ||
|
|
||
| - `providers({ dev })` — the supplied layer replaces the built-in dev providers during `alchemy dev` only. `PrismaLocalProviders` is a structural type (the union of the twelve resource providers), and the test proves an embedder layer built from `Provider.succeed` typechecks with **no casts** — that's the seam's contract. | ||
| - `liveProviders()` — the live layer exported for frameworks doing their own mode selection. | ||
|
|
||
| What implementing the dev side actually looks like — a provider per resource you emulate, plain `Provider.succeed`, no casts: | ||
|
|
||
| ```ts | ||
| const devDatabase = Provider.succeed(Prisma.Database, { | ||
| stables: ["databaseId"], | ||
| list: () => Effect.succeed([]), | ||
| diff: Effect.fn(function* () { | ||
| return { action: "update" } as const; | ||
| }), | ||
| read: Effect.fn(function* ({ output }) { | ||
| return output; | ||
| }), | ||
| reconcile: Effect.fn(function* ({ id, news }) { | ||
| // Start (or adopt) a local Postgres for this database and | ||
| // hand back the same attribute shape the live provider emits. | ||
| const server = yield* myDevPostgres.ensure(id); | ||
| return { | ||
| databaseId: server.instanceName, | ||
| databaseName: resolveId(news.project), | ||
| directConnectionString: Redacted.make(server.url), | ||
| databaseUrl: Redacted.make(server.url), | ||
| status: "ready", | ||
| // pooled/accelerate/origin fields are optional — omit what | ||
| // your emulator doesn't have. | ||
| }; | ||
| }), | ||
| delete: Effect.fn(function* ({ output }) { | ||
| yield* myDevPostgres.destroy(output.databaseId); | ||
| }), | ||
| }); | ||
|
|
||
| const myEmulatorProviders = (): Prisma.PrismaLocalProviders => | ||
| Layer.mergeAll(devDatabase, devConnection, devCompute /* … */); | ||
|
|
||
| // Deploy is untouched; `alchemy dev` runs on your emulators: | ||
| Prisma.providers({ dev: myEmulatorProviders() }); | ||
| ``` | ||
|
|
||
| And a framework that owns mode selection entirely skips `providers()` and composes the exported live layer with its own local one: | ||
|
|
||
| ```ts | ||
| const providers = isDev ? myEmulatorProviders() : Prisma.liveProviders(); | ||
| ``` | ||
|
|
||
| ## A locked state backend: `State/PostgresState` | ||
|
|
||
| `Compute`'s recovery docs tell users to "use a durable, locked state backend" — and no in-tree backend has locking. This adds one on the dependency the repo already carries (`pg`): | ||
|
|
||
| - per-`(stack, stage)` **session advisory lock** (`pg_try_advisory_lock(hashtextextended(key, 0))`) held on a reserved connection, with the holder re-verified against `pg_locks` from a *different* pool connection — so a silently dropped lock connection is detected, not trusted; | ||
| - a TTL-amortized lease check wrapping every operation; stage-less `deleteStack` locks each stage before touching it; | ||
| - schema migration under a transaction-scoped advisory lock, because concurrent `create table if not exists` genuinely fails on Postgres (duplicate `pg_type` errors — reproduced on PG 15) and first-boot races between two stacks are exactly the case a state store must survive. | ||
|
|
||
| It is deliberately **not** re-exported from the `State` barrel: the barrel is imported by engine files that get bundled for workers, and `pg` must stay off that graph. Deep import: `alchemy/State/PostgresState` (a comment in the barrel says why). | ||
|
|
||
| Tests are hermetic stubs per the existing backend convention (`HttpStateStore`). Two real-Postgres behaviors were verified against live PostgreSQL 15 during development and documented in comments rather than CI-tested: the 64-bit lock-key reconstruction from `pg_locks.classid/objid` (intentional bigint wraparound), and the concurrent-DDL failure motivating the migration lock. | ||
|
|
||
| ## One-line core fix: `Aliases` typing | ||
|
|
||
| `ResourceClass.Aliases` is `readonly string[] | undefined`, but `ResourceClassLike.Aliases?: readonly string[]`. Under a consumer tsconfig with `exactOptionalPropertyTypes: true`, every `Provider.effect(cls, …)` call fails to typecheck (we currently carry a pnpm patch for this). The fix widens the optional to `| undefined`. | ||
|
|
||
| ## Verification | ||
|
|
||
| - `bun run format:check` clean; `bun tsc -b` (monorepo) clean. | ||
| - `bun alchemy-test --fast test/Prisma test/State`: 391 passed / 0 failed. | ||
| - Core engine suites (exercising the `Resource.ts` change): 549 passed / 0 failed. | ||
| - `generate-api-reference`: no new category; Prisma gains the two bucket pages; `docs:check` builds clean. Contract fixture updated (`deferredRoutes` now empty; route coverage 71 → 78). | ||
|
|
||
| ## Alternatives considered | ||
|
|
||
| - **Three separate PRs.** Kept together because they're one consumer's complete need and the review context overlaps (BucketKey's crash-window design references Connection's; the dev seam is what makes the state backend's locking story testable end-to-end for us). Say the word and we'll split. | ||
| - **`ReturnType<typeof devProviderLayer>` for `PrismaLocalProviders`.** It bakes the built-in stubs' literal `stables` tuples into the public type — no real embedder layer can satisfy it without `as never`. Structural union instead. | ||
| - **Exporting `postgresState` from the `State` barrel.** Poisons worker bundles with `pg`. Deep import + explanatory comment instead. | ||
| - **Letting embedders rebuild the live wiring themselves** (no `liveProviders()` export). Works today but couples every embedder to the private composition of client/auth/upload layers — each upstream refactor breaks them silently. | ||
| - **A migration-free schema bootstrap** (plain `create table if not exists`). Fails under concurrency on real Postgres; see above. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.