From 4f00129f819068e9536185677fd3a7a9598ae479 Mon Sep 17 00:00:00 2001 From: Dario Lencina Date: Sun, 2 Aug 2026 00:36:32 -0700 Subject: [PATCH] Record how every screenshot scene was made Changing a scene was not possible without reverse-engineering it. generate.mjs stored the 12 base scene prompts, but everything made with `generate.mjs edit` -- which is every "what the camera sees" preview, plus the edits that fixed a scene's staging -- was passed as an argv string and never recorded. 9 of the 16 images manifest.js renders had no recipe. docs/marketing/prompt-pack.md, which the header points at, holds scene concepts, not these. Three additions: CHOSEN records which candidate each base scene kept, since `generate.mjs N` writes _c1..cN and the manifest references one of them. DERIVED records the edits and previews: source images, aspect, prompt. A new `generate.mjs derive ` re-runs one. `--as ` writes elsewhere so a regeneration can be compared before it replaces the committed asset -- the model is not deterministic, so you always want to look first. `generate.mjs derive` with no argument lists everything and checks that every manifest reference has a recipe, exiting non-zero if one does not. That is the part that keeps this from rotting: the gap was invisible until someone tried to change a scene, and now it fails loudly instead. Also adds the slot3_ipad prompt, which was missing entirely -- the iPad cardinal scene could not be regenerated at all. Honesty about what these are: the DERIVED prompts are RECONSTRUCTED from the committed images, not the original text. They are labelled as such in the file. I validated the mechanism by running slot3_preview end to end: it returned the same subject at the same aspect, framed wider than the committed shot, so I tightened that prompt and mac3_preview's with an explicit framing instruction. An equivalent asset, not the same pixels -- the recipe, not the receipt. Crop boxes are exact, not reconstructed: recovered by matching each committed crop against its parent (mac2_cook_c1_crop is 0,440,1536,2210, mean abs diff 0.34 -- JPEG noise). No scene asset is modified by this commit. Co-Authored-By: Claude Opus 5 (1M context) --- store_assets/screenshot-pipeline/README.md | 53 ++++- store_assets/screenshot-pipeline/generate.mjs | 209 +++++++++++++++++- 2 files changed, 254 insertions(+), 8 deletions(-) diff --git a/store_assets/screenshot-pipeline/README.md b/store_assets/screenshot-pipeline/README.md index c5d2ad79..b4251369 100644 --- a/store_assets/screenshot-pipeline/README.md +++ b/store_assets/screenshot-pipeline/README.md @@ -110,6 +110,45 @@ English callout text (CAMERA, REMOTE, …) to the localized pill text. whatever `AVCaptureDevice.localizedName` returns, so a capture made against a USB webcam has to have that chip blanked. +## Changing a scene + +Every image `manifest.js` renders traces back to a recorded recipe in +`generate.mjs`, so a scene can be changed rather than reverse-engineered: + +```bash +node generate.mjs derive # list every recipe + coverage check +node generate.mjs derive slot3_preview.jpg --as try.jpg # remake, don't clobber +``` + +Three kinds of recipe: + +- **`PROMPTS`** — the base scenes. `CHOSEN` records which candidate the manifest + kept, since `node generate.mjs N` writes `_c1..cN`. +- **`DERIVED`** — the "what the camera sees" previews and the edits that fixed a + scene's staging (a phone rotated to portrait, a feeder moved outside a window). +- **crops** — exact boxes, recovered by matching each committed crop against its + parent. + +`node generate.mjs derive` with no argument also checks that every manifest +reference has a recipe, and exits non-zero if one doesn't. Add a scene, add its +recipe — otherwise the gap stays invisible until someone tries to change it. + +**The `DERIVED` prompts are reconstructions.** The originals were typed as argv +strings and never recorded; these were rebuilt by reading the committed images. +They produce an equivalent asset, not the same pixels — the model is not +deterministic. Always `--as` and compare before overwriting. + +Worked example — swapping the cardinal for another animal: + +1. Edit the `slot3_ots`, `slot3_ipad` and `mac3_direct` prompts in `PROMPTS`. +2. `node generate.mjs slot3_ots 3`, pick a candidate, update `CHOSEN`. +3. Re-run the staging edits that sit on top (`slot3_ots_c1p`, `mac3_direct_e1`). +4. Re-measure the quads — the phone lands somewhere new. `tools.py detect`, then + ALWAYS `tools.py overlay` to check. +5. Re-run the previews (`slot3_preview`, `mac3_preview`, `mac3_preview_port`) + so the remote screens show the same animal the in-scene camera is pointed at. +6. `./ship-locales.sh`. + ## Adding a new screenshot (the Claude workflow) This pipeline was built with Claude Code and is easiest to extend the same way. @@ -127,13 +166,15 @@ Example prompts that work well: What Claude does under the hood (or do it manually): 1. **Generate the scene** — add a prompt to `generate.mjs` (`PROMPTS`), run - `node generate.mjs 2`. Scenes must show device screens *black/off*, - subjects in the upper two thirds, no logos (reject candidates with Apple logos). - Requires a Google AI Studio key in `AI_STUDIO`. + `node generate.mjs 2`, then record the candidate you keep in + `CHOSEN`. Scenes must show device screens *black/off*, subjects in the upper + two thirds, no logos (reject candidates with Apple logos). Requires a Google + AI Studio key in `AI_STUDIO`. 2. **Derive the preview** — what the camera device sees, seeded from the scene - itself for consistency: `node generate.mjs edit "," out.jpg ""`. - Crop the subject with `python3 tools.py crop` and pass it as the second - reference so pose/orientation match exactly. + itself for consistency. Add a `DERIVED` entry and run + `node generate.mjs derive `; use `edit` only while you are still + iterating on the wording. Crop the subject with `python3 tools.py crop` and + pass it as a second reference so pose/orientation match exactly. 3. **Find the screen quads** — `python3 tools.py detect X0 Y0 X1 Y1`, then ALWAYS verify with `python3 tools.py overlay TLx TLy ... --out q.png`. AI-image device edges bow slightly: trust locally-measured corners over global diff --git a/store_assets/screenshot-pipeline/generate.mjs b/store_assets/screenshot-pipeline/generate.mjs index a3881a6c..f2296d28 100644 --- a/store_assets/screenshot-pipeline/generate.mjs +++ b/store_assets/screenshot-pipeline/generate.mjs @@ -5,7 +5,14 @@ // Usage: // node generate.mjs [count] e.g. node generate.mjs slot0_v1 2 // node generate.mjs edit [aspect] -// image-to-image, e.g. deriving "what the camera phone sees" from a scene +// image-to-image, ad hoc. Anything the manifest ends up using should be +// recorded in DERIVED instead, so it can be remade. +// node generate.mjs derive [outFile] +// re-run a recorded derivation; no argument lists them all +// +// Every file manifest.js references traces back to one of three things: a +// PROMPTS entry (with CHOSEN recording which candidate was kept), a DERIVED +// entry, or a crop. Adding a scene means adding it here too — see README. import { readFileSync, writeFileSync, mkdirSync } from "node:fs"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; @@ -118,6 +125,17 @@ rustic wooden bird feeder with a bright red cardinal perched on its ledge, and a second unbranded smartphone mounted on a small flexible tripod clamped right next to the feeder, its screen dark — both clearly recognizable, gently soft from distance. Morning light, dewy greens. ${STYLE}`, + slot3_ipad: `Over-the-shoulder shot from directly behind a woman standing on a +cozy wooden porch on a bright morning. Her shoulder and hair are softly out of +focus at the right edge of the near foreground. She holds an unbranded modern +tablet upright in portrait orientation in one hand, screen facing the viewer, +completely black and switched off, no reflections, shown large in the +center-left of the frame. Beyond the porch railing, across the garden a few +steps away, a rustic wooden bird feeder with a bright red cardinal perched on +its ledge, and an unbranded smartphone mounted in PORTRAIT orientation on a +small flexible tripod clamped right next to the feeder, its screen dark — both +clearly recognizable, gently soft from distance. Morning light, dewy greens. +${STYLE}`, slot4_watch: `Night scene, close-up of a raised wrist wearing an unbranded modern smartwatch with a rectangular rounded screen, completely black and off, facing the viewer, in the upper half of the frame. In the softly blurred background, an @@ -138,6 +156,151 @@ the other hand. Both device screens dark and off. Airy daylight, white brick and wood floor. ${STYLE}`, }; +// Which candidate each base scene the manifest uses came from. `node +// generate.mjs N` writes _c1..cN; this records which one was kept, so +// a regenerated set can be compared against the same slot. +const CHOSEN = { + slot0_ots: "slot0_ots_c1.jpg", + slot1_group: "slot1_group_c2.jpg", + slot3_ipad: "slot3_ipad_c1.jpg", + slot3_ots: "slot3_ots_c1.jpg", + slot4_watch: "slot4_watch_c1.jpg", + mac0_studio: "mac0_studio_c2.jpg", + mac2_cook: "mac2_cook_c1.jpg", + mac3_direct: "mac3_direct_c1.jpg", +}; + +// Everything the manifest uses that is NOT a raw candidate: the "what the +// camera sees" previews, and the edits that fixed a scene's staging. +// +// IMPORTANT — these prompts are RECONSTRUCTED by reading the committed images, +// not the verbatim text originally typed. That was passed as an argv string and +// never recorded anywhere. They describe the same transformation and produce an +// equivalent asset; they will NOT reproduce the committed file pixel for pixel, +// because the model is not deterministic. Treat them as the recipe, not the +// receipt. Crop entries, by contrast, are exact: their boxes were recovered by +// matching the committed crop against its parent. +const DERIVED = { + // ---- Scene edits ---- + "slot3_ots_c1p.jpg": { + from: ["slot3_ots_c1.jpg"], + aspect: "9:16", + prompt: `Keep this photograph exactly as it is, with one change: rotate the +small smartphone clamped to the flexible tripod beside the bird feeder so it +stands in PORTRAIT orientation, taller than it is wide, its screen still facing +the viewer and completely black and switched off. Do not change the woman, the +phone in her hand, the feeder, the cardinal, the railing, the light, or the +background.`, + }, + "mac3_direct_e1.jpg": { + from: ["mac3_direct_c1.jpg"], + aspect: "9:16", + prompt: `Keep this room, window, laptop, desk and the woman exactly as they +are, with two changes: move the wooden bird feeder and the cardinal perched on +it OUTSIDE the window, into the garden behind the glass, so the window frame +clearly separates them from the room; and rotate the smartphone clamped to the +flexible tripod beside the feeder into PORTRAIT orientation. Every screen stays +completely black and switched off.`, + }, + "mac0_studio_e2.jpg": { + from: ["mac0_studio_c2.jpg"], + aspect: "9:16", + prompt: `Keep this studio scene exactly as it is and add the subject being +photographed: on the white product table in front of the professional camera, a +luxury wristwatch with a dark leather strap, being adjusted by a hand in a white +cotton glove reaching in from the right. The camera on the tripod points at the +watch. Every device screen stays completely black and switched off.`, + }, + + // ---- "What the camera sees" previews ---- + // Each renders the scene's subject alone, from the camera device's position, + // at the aspect that device would actually deliver. Crop the subject out of + // the scene with `tools.py crop` and pass it as a second reference when the + // pose needs to match exactly. + "slot0_ots_preview.jpg": { + from: ["slot0_ots_c1.jpg"], + aspect: "9:16", + prompt: `Render only what the tripod-mounted camera phone in this scene is +pointed at, as a clean full-frame product photograph: the luxury wristwatch +standing on its small white plinth against the seamless white studio backdrop, +softbox-lit. No devices, no people, no tripod in frame.`, + }, + "slot1_preview.jpg": { + from: ["slot1_group_c2.jpg"], + aspect: "16:9", + prompt: `Render only what the tripod-mounted camera phone in this scene is +pointed at, as a clean full-frame photograph: the whole multi-generation family +sitting together on the picnic blanket, everyone looking at the camera, the +picnic basket in front of them. Same warm golden light and string lights and +greenery behind. No devices, no tripod in frame.`, + }, + "slot3_preview.jpg": { + from: ["slot3_ots_c1p.jpg"], + aspect: "9:16", + prompt: `Render only what the tripod-mounted camera phone in this scene is +pointed at, as a clean full-frame photograph shot on a telephoto lens: the +rustic wooden bird feeder with the bright red cardinal perched on its ledge, +framed TIGHT so the bird and the feeder ledge fill most of the frame and the +bird is large enough to read at a glance. Same morning light, dewy greens, +softly blurred garden behind. No devices, no people, no porch railing in +frame.`, + }, + "slot4_preview.jpg": { + from: ["slot4_watch_c1.jpg"], + aspect: "9:16", + prompt: `Render only what the phone clamped to the telescope eyepiece in this +scene is pointed at, as a clean full-frame photograph: the bright full moon, +sharp and detailed, filling much of the frame against a black night sky. No +devices, no people, no telescope in frame.`, + }, + "mac0_preview.jpg": { + from: ["mac0_studio_e2.jpg"], + aspect: "16:9", + prompt: `Render only what the professional camera on the tripod in this scene +is pointed at, as a clean full-frame product photograph: the luxury wristwatch +with the dark leather strap lying on the white product table, a hand in a white +cotton glove adjusting it. Softbox-lit, bright and premium. No devices, no +tripod in frame.`, + }, + "mac2_preview.jpg": { + from: ["mac2_cook_c1.jpg"], + aspect: "16:9", + prompt: `Render only what the overhead phone on the under-cabinet arm in this +scene is pointed at, as a clean full-frame overhead photograph shot straight +down: the round wooden charcuterie board on the kitchen counter, loaded with +cured meats, cheeses, figs, grapes, nuts, olives and edible flowers, with a hand +reaching in to place a sprig of rosemary. No devices, no arm in frame.`, + }, + "mac3_preview.jpg": { + from: ["mac3_direct_e1.jpg"], + aspect: "16:9", + prompt: `Render only what the phone clamped beside the feeder in this scene is +pointed at, as a clean full-frame photograph: the rustic wooden bird feeder with +the bright red cardinal perched on its ledge, framed TIGHT so the feeder and the +bird fill most of the frame, softly blurred garden foliage behind. No devices, +no window frame, no people in frame.`, + }, + "mac3_preview_port.jpg": { + from: ["mac3_direct_e1.jpg"], + aspect: "9:16", + // Committed at 1295x2752 (aspect 0.47) rather than the 1536x2752 the model + // returns: that near-phone aspect is what makes the frame reach the top and + // bottom edges of a portrait screen in slots 2 and 2i. The trim was not + // recorded; a centred crop reproduces the committed framing. + crop: { from: "mac3_preview_port.jpg", box: [120, 0, 1415, 2752] }, + prompt: `Render a tight close-up of what the phone clamped beside the feeder +in this scene is pointed at, as a clean full-frame photograph: the bright red +cardinal filling most of the frame, perched on the weathered wooden ledge with +seed scattered around it, softly blurred green foliage and a wooden post behind. +No devices, no window frame, no people in frame.`, + }, + + // ---- Deterministic crops (exact; boxes recovered from the committed files) ---- + "mac2_cook_c1_crop.jpg": { + crop: { from: "mac2_cook_c1.jpg", box: [0, 440, 1536, 2210] }, + }, +}; + async function callModel(parts, aspectRatio) { const res = await fetch( `https://generativelanguage.googleapis.com/v1beta/models/${MODEL}:generateContent`, @@ -167,7 +330,49 @@ function save(file, inline) { mkdirSync(OUT_DIR, { recursive: true }); -if (process.argv[2] === "edit") { +if (process.argv[2] === "derive") { + const want = process.argv[3]; + if (!want) { + console.log("recorded derivations:\n"); + for (const [out, d] of Object.entries(DERIVED)) { + const how = d.prompt ? `edit of ${d.from.join(", ")} @ ${d.aspect}` : "crop"; + console.log(` ${out.padEnd(26)} ${how}${d.crop && d.prompt ? " + crop" : ""}`); + } + console.log("\nbase scenes kept from PROMPTS:\n"); + for (const [id, file] of Object.entries(CHOSEN)) console.log(` ${file.padEnd(26)} node generate.mjs ${id}`); + + // Coverage: every scene the manifest renders must be remakeable. Without + // this the gap is invisible until someone tries to change a scene and finds + // the recipe was never written down. + const manifest = readFileSync(join(here, "manifest.js"), "utf8"); + const used = [...new Set(manifest.match(/\.\.\/ai-scenes\/[\w.-]+/g) || [])] + .map((p) => p.replace("../ai-scenes/", "")); + const known = new Set([...Object.keys(DERIVED), ...Object.values(CHOSEN)]); + const orphans = used.filter((f) => !known.has(f)); + console.log(`\ncoverage: ${used.length - orphans.length}/${used.length} manifest scenes have a recipe`); + for (const f of orphans) console.log(` NO RECIPE: ${f}`); + process.exit(orphans.length ? 1 : 0); + } + const d = DERIVED[want]; + if (!d) { console.error(`no recorded derivation for ${want}; have: ${Object.keys(DERIVED).join(", ")}`); process.exit(1); } + // `--as ` writes somewhere else, so a regeneration can be compared + // against the committed asset before it replaces it. The model is not + // deterministic; you always want to look before you overwrite. + const asIdx = process.argv.indexOf("--as"); + const outName = asIdx > 0 ? process.argv[asIdx + 1] : want; + if (d.prompt) { + const parts = d.from.map((f) => ({ + inlineData: { mimeType: "image/jpeg", data: readFileSync(join(OUT_DIR, f)).toString("base64") }, + })); + parts.push({ text: d.prompt }); + save(join(OUT_DIR, outName), await callModel(parts, d.aspect)); + } + if (d.crop) { + const [x0, y0, x1, y1] = d.crop.box; + // tools.py owns image manipulation; this script stays dependency-free. + console.log(`then crop:\n python3 tools.py crop ../ai-scenes/${d.crop.from} ${x0} ${y0} ${x1} ${y1} ../ai-scenes/${want}`); + } +} else if (process.argv[2] === "edit") { // input may be several comma-separated reference image paths const [input, outFile, prompt, aspect = "9:16"] = process.argv.slice(3); const parts = input.split(",").map((p) => ({