Add media usage index write-side foundation by khoinguyenpham04 · Pull Request #1746 · emdash-cms/emdash

khoinguyenpham04 · 2026-07-03T14:55:06Z

What does this PR do?

Adds the write-side foundation for the Media Usage index. This includes the database tables/status metadata, content media reference extraction, content source snapshots for columns and draft_overlay, repository APIs, runtime refresh wiring for normal content lifecycle writes, stale marking for known bypass paths, and generation-guarded source replacement/deletion so refresh races preserve current usage and mark coverage stale instead of failing content writes.

This PR intentionally stops before repair/rebuild, the usage read API, admin Used In UI, safe delete, replace-in-place behavior, or media delete behavior changes.

Related discussion: #1503

Type of change

Checklist

I have read CONTRIBUTING.md
pnpm typecheck passes
pnpm lint passes
pnpm test passes (or targeted tests for my change)
pnpm format has been run
I have added/updated tests for my changes (if applicable)
User-visible strings in the admin UI are wrapped for translation (if applicable). Do not include messages.po changes except in translation PRs — a workflow extracts catalogs on merge to main. N/A: no admin UI strings in this PR.
I have added a changeset (if this PR changes a published package)
New features link to an approved Discussion: Media usage index — power image sitemaps, "Used in", and safe deletes #1503

AI-generated code disclosure

This PR includes AI-generated code — model/tool: OpenCode with GPT-5.5 (openai/gpt-5.5)

Screenshots / test output

Non-visual backend/indexing change.

Verified:

pnpm lint
pnpm typecheck
pnpm format
pnpm --filter emdash exec vitest run tests/integration/database/media-usage-status-migration.test.ts tests/integration/database/media-usage-content-fields.test.ts tests/integration/database/media-usage-repository.test.ts tests/integration/database/media-usage-migration.test.ts tests/integration/database/media-usage-content-snapshots.test.ts tests/integration/database/media-usage-content-refresh.test.ts tests/integration/database/media-usage-runtime-refresh.test.ts tests/integration/database/media-usage-stale-bypass.test.ts tests/unit/media/usage-extractor.test.ts tests/unit/media/usage-source-key.test.ts (10 files, 98 tests)
git diff --check

Try this PR

Open a fresh playground →

A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.

_{Tracks feature/media-usage-index-core. Updated automatically when the playground redeploys.}

…ures

…tests

…ated tests

…tests

…n handling and fingerprinting

…eFingerprint

…with generation checks

changeset-bot · 2026-07-03T14:55:11Z

🦋 Changeset detected

Latest commit: 042c0e5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
emdash	Patch
@emdash-cms/cloudflare	Patch
@emdash-cms/sandbox-workerd	Patch
@emdash-cms/fixture-perf-site	Patch
@emdash-cms/perf-demo-site	Patch
@emdash-cms/cache-demo-site	Patch
@emdash-cms/do-demo-site	Patch
@emdash-cms/do-solo-demo-site	Patch
@emdash-cms/admin	Patch
@emdash-cms/auth	Patch
@emdash-cms/blocks	Patch
@emdash-cms/gutenberg-to-portable-text	Patch
@emdash-cms/x402	Patch
create-emdash	Patch
@emdash-cms/auth-atproto	Patch
@emdash-cms/plugin-embeds	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

cloudflare-workers-and-pages · 2026-07-03T14:55:14Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	emdash-demo-cache	`042c0e5`	Jul 03 2026, 05:38 PM

cloudflare-workers-and-pages · 2026-07-03T14:55:15Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	emdash-playground	`042c0e5`	Jul 03 2026, 05:38 PM

pkg-pr-new · 2026-07-03T15:10:15Z

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@1746

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@1746

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@1746

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@1746

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@1746

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@1746

emdash

npm i https://pkg.pr.new/emdash@1746

create-emdash

npm i https://pkg.pr.new/create-emdash@1746

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@1746

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@1746

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@1746

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@1746

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@1746

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@1746

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@1746

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@1746

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@1746

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@1746

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@1746

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@1746

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@1746

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@1746

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@1746

commit: 042c0e5

…uccess signal

ascorbic

Great stuff. Just a couple of notes. However this is a very large PR. For future ones, can you keep them smaller in scope so they're easier to review.

…ation conflicts

github-actions · 2026-07-03T18:36:51Z

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

feat(admin): add Taxonomy Types management page #213 (5 shared files)
feat(core): add group and sortOrder to collections for admin UI organization #292 (4 shared files)

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

…ation) into mj/prod for live trial

masonjames · 2026-07-03T21:47:28Z

Tested this on a real production site (masonjames.com — Ghost→EmDash migration, Node + SQLite, downstream fork with local media patches). Results:

Merge + tests: merged cleanly, 4364/4364 core tests passing including the new media-usage suites.
Migration on live data: 050_media_usage_index_status applied cleanly at boot on the production SQLite DB — all three tables present, ledger correct, no boot noise.
Write-side, end-to-end: inserting a media-library image into a post and publishing produced exactly the expected _emdash_media_usage row; lifecycle writes create content:{collection}:{id}:columns source rows as designed. Alt/caption edits and unreferenced uploads correctly created no rows. All collections initialized as stale / CONTENT_USAGE_STALE — the intended "incomplete until rebuild" state. No route errors, no interference with our local media patches.
Field observation for later phases: pre-existing content yields zero usage rows here — migrated bodies embed media as absolute URLs on a custom media domain, so there are no managed references to extract. Expected given this PR's scope, but imported sites will depend on repair/rebuild to populate the index — and it may be worth attributing absolute URLs that match a configured media/public URL, since WordPress and Ghost migrations commonly end up in this shape.

Happy to re-test when the read API / rebuild phases land.

khoinguyenpham04 · 2026-07-03T23:31:16Z

@masonjames Thank you so much for testing this out! This is actually a really helpful validation :)

Also, I agree URL attribution for configured media/public domains is worth considering in the rebuild/import follow-up.

masonjames · 2026-07-04T01:33:41Z

@khoinguyenpham04 awesome! I'm really excited about the media components in emdash. Feel free to tag me on PRs whenever you'd like additional testing. Always happy to do so. 🙌

ascorbic

Thanks! And thanks @masonjames for the review

* feat(media): add usage index tables * feat(media): add usage extractor * feat(media): add usage repository * chore(media): add usage foundation changeset * ci: update query-count snapshots * fix(media): skip path-like local usage refs * feat(media): add usage index status metadata * feat: enhance media usage index with new tracking and management features * feat(media): Rename Repository Fixtures To Storage Variants * feat(media): Add Source-Key Contract Tests * feat(media): refine extractor to focus on image subfields and update tests * feat(media): tightened the source-variant write/input type seam. * feat(media): implement content media usage field discovery and associated tests * feat(media): add content media usage snapshots loader and associated tests * feat(media): enhance content media usage snapshots with draft revision handling and fingerprinting * feat(media): update buildContentSource to conditionally include sourceFingerprint * feat(media): Add Content Usage Refresh Service * feat(media): Wire Runtime Create, Update, Duplicate, Revision Restore * Wire Publish, Unpublish, Schedule, Unschedule, Discard Draft * Wire Trash And Permanent Delete * Refresh I18n Non-Translatable Siblings * Mark Bypass And Schema Paths Stale * Add Refresh Failure Path Coverage * Implement guarded replace and delete methods in MediaUsageRepository with generation checks * Fix media usage lint issues * Fix media usage display title extraction * update guarded replace result handling and add test for replacement success signal * Implement retry logic for content media usage refresh to handle generation conflicts * Harden media usage failure handling --------- Co-authored-by: emdashbot[bot] <emdashbot[bot]@users.noreply.github.com>

khoinguyenpham04 and others added 25 commits July 1, 2026 00:40

feat(media): add usage index tables

d1bc115

feat(media): add usage extractor

72b3253

feat(media): add usage repository

8ef7d0e

chore(media): add usage foundation changeset

fa9c6b1

ci: update query-count snapshots

a97e17e

fix(media): skip path-like local usage refs

0b99ad1

feat(media): add usage index status metadata

f477b81

feat: enhance media usage index with new tracking and management feat…

6b6c011

…ures

feat(media): Rename Repository Fixtures To Storage Variants

0066d19

feat(media): Add Source-Key Contract Tests

81158b2

feat(media): refine extractor to focus on image subfields and update …

c66ca0f

…tests

feat(media): tightened the source-variant write/input type seam.

f7590e1

feat(media): implement content media usage field discovery and associ…

918ac41

…ated tests

feat(media): add content media usage snapshots loader and associated …

5a662f5

…tests

feat(media): enhance content media usage snapshots with draft revisio…

35c90cf

…n handling and fingerprinting

feat(media): update buildContentSource to conditionally include sourc…

647361d

…eFingerprint

feat(media): Add Content Usage Refresh Service

7466f64

feat(media): Wire Runtime Create, Update, Duplicate, Revision Restore

0705860

Wire Publish, Unpublish, Schedule, Unschedule, Discard Draft

cf64e8a

Wire Trash And Permanent Delete

1ecf9f5

Refresh I18n Non-Translatable Siblings

24e5333

Mark Bypass And Schema Paths Stale

7b6dbb9

Add Refresh Failure Path Coverage

18cdf12

Implement guarded replace and delete methods in MediaUsageRepository …

770f8a2

…with generation checks

Fix media usage lint issues

f30f1de

Copilot AI review requested due to automatic review settings July 3, 2026 14:55

github-actions Bot added the review/needs-review No maintainer or bot review yet label Jul 3, 2026

github-actions Bot added area/docs and removed needs-rebase labels Jul 3, 2026

khoinguyenpham04 self-assigned this Jul 3, 2026

khoinguyenpham04 requested a review from ascorbic July 3, 2026 15:14

Fix media usage display title extraction

1446a65

github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/awaiting-author Reviewed; waiting on the author to respond labels Jul 3, 2026

update guarded replace result handling and add test for replacement s…

fc980c3

…uccess signal

ascorbic reviewed Jul 3, 2026

View reviewed changes

Comment thread .changeset/media-usage-index-hardening.md

Comment thread packages/core/src/media/usage/content-refresh.ts Outdated

Comment thread packages/core/src/database/repositories/media-usage.ts Outdated

github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-rereview Author pushed changes since the last review labels Jul 3, 2026

Implement retry logic for content media usage refresh to handle gener…

6f496fb

…ation conflicts

khoinguyenpham04 added this to the 1.0 milestone Jul 3, 2026

Harden media usage failure handling

042c0e5

github-actions Bot added review/needs-rereview Author pushed changes since the last review overlap and removed review/awaiting-author Reviewed; waiting on the author to respond labels Jul 3, 2026

masonjames added a commit to masonjames/emdash that referenced this pull request Jul 3, 2026

Merge upstream PR emdash-cms#1746 (media usage index write-side found…

fcf3ecd

…ation) into mj/prod for live trial

ascorbic approved these changes Jul 6, 2026

View reviewed changes

ascorbic merged commit a9e9dde into main Jul 6, 2026
59 checks passed

ascorbic deleted the feature/media-usage-index-core branch July 6, 2026 09:27

github-actions Bot added review/approved Approved; no new commits since and removed review/needs-rereview Author pushed changes since the last review labels Jul 6, 2026

emdashbot Bot mentioned this pull request Jul 6, 2026

ci: release #1734

Merged

Navigation Menu

Uh oh!

Add media usage index write-side foundation - #1746

Add media usage index write-side foundation#1746
ascorbic merged 30 commits into
mainfrom
feature/media-usage-index-core

khoinguyenpham04 commented Jul 3, 2026 •

edited by github-actions Bot

Loading

Uh oh!

changeset-bot Bot commented Jul 3, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 3, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 3, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Jul 3, 2026 •

edited

Loading

Uh oh!

ascorbic left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

masonjames commented Jul 3, 2026

Uh oh!

khoinguyenpham04 commented Jul 3, 2026

Uh oh!

masonjames commented Jul 4, 2026

Uh oh!

ascorbic left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

khoinguyenpham04 commented Jul 3, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Type of change

Checklist

AI-generated code disclosure

Screenshots / test output

Try this PR

Uh oh!

changeset-bot Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

pkg-pr-new Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ascorbic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

Overlapping PRs

Uh oh!

masonjames commented Jul 3, 2026

Uh oh!

khoinguyenpham04 commented Jul 3, 2026

Uh oh!

masonjames commented Jul 4, 2026

Uh oh!

ascorbic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

khoinguyenpham04 commented Jul 3, 2026 •

edited by github-actions Bot

Loading

changeset-bot Bot commented Jul 3, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Jul 3, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Jul 3, 2026 •

edited

Loading

pkg-pr-new Bot commented Jul 3, 2026 •

edited

Loading