Skip to content

feat(pipeline): add IDML/PDF ingest stage to the build pipeline#152

Merged
PAMulligan merged 2 commits into
mainfrom
117-redence-verify-pipelines-dataetl-add-an-idmlpdf-ingest-stage-to-the-build-pipeline
Jun 24, 2026
Merged

feat(pipeline): add IDML/PDF ingest stage to the build pipeline#152
PAMulligan merged 2 commits into
mainfrom
117-redence-verify-pipelines-dataetl-add-an-idmlpdf-ingest-stage-to-the-build-pipeline

Conversation

@PAMulligan

Copy link
Copy Markdown
Collaborator

Summary

Adds a new ingest ETL stage to the InDesign pipeline that extracts a source document (.idml/.pdf) into a generator-ready bundle on disk:

  • ir.json — the intermediate IR the theme generator consumes
  • assets/ — image bytes pulled from the source, named to match the generator's planAssets() output
  • assets.manifest.json — per-frame provenance + unresolved-asset warnings

Why

Embedded IDML image bytes previously had no automatic path into a generated theme — generate-theme could only stage images from a loose --asset-dir, and nothing extracted the bytes the IDML zip carries. This stage resolves them once and stages them under the exact filenames generateTheme() references, so the bundle feeds straight into generation:

flavian pipeline ingest brochure.idml --out-dir build/brochure
flavian pipeline indesign build/brochure/ir.json --asset-dir build/brochure/assets

What's in the stage

  • Extract — parse to IR; pull image bytes (IDML zip via extractIdmlAssets, PDF via the parse-pdf image cache).
  • TransformresolveAssets / buildIngestBundle join IR image frames to an href → bytes resolver (pure, source-agnostic), computing staged paths via the generator's own planAssets so they line up byte-for-byte downstream.
  • LoadwriteIngestBundle writes ir.json + assets/ + assets.manifest.json.

Runnable via pnpm pipeline:ingest, flavian pipeline ingest, and the flavian-ingest bin. Adopts the in-progress idml-assets.js helper (and its unit test).

Test plan

  • New ingest.test.mjs — transform, manifest shape, and an end-to-end IDML run whose staged paths line up with generateTheme() (3/3)
  • assets-idml.test.mjs — embedded-asset extraction (2/2)
  • InDesign smoke test scripts/indesign-fse/smoke-test.mjs
  • InDesign fixtures test tests/fixtures/indesign/fixtures.test.mjs (3/3)
  • End-to-end CLI: embedded IDML image → ingest → generate → image bytes land in theme/assets/ automatically
  • commitlint passes on the commit

The full @flavian/pipeline suite is green except two pre-existing snapshot tests that only fail on a Windows (CRLF) checkout — .gitattributes stores them as LF (git ls-files --eoli/lf), so they pass on the Ubuntu CI runner.

Closes #117

🤖 Generated with Claude Code

Add an `ingest` ETL stage that extracts an InDesign source (.idml or .pdf)
into a generator-ready bundle on disk: ir.json (the intermediate IR the theme
generator consumes), assets/ (image bytes pulled from the source, named to
match the generator's planAssets output), and assets.manifest.json (per-frame
provenance plus unresolved-asset warnings).

This closes a real gap: embedded IDML image bytes previously had no automatic
path into a generated theme. The stage resolves them once and stages them
under the exact filenames generateTheme references, so the bundle feeds
straight into generation:

    flavian pipeline ingest brochure.idml --out-dir build/brochure
    flavian pipeline indesign build/brochure/ir.json --asset-dir build/brochure/assets

Details:
- Pure, source-agnostic transform (resolveAssets/buildIngestBundle) joins IR
  image frames to an href->bytes resolver; runIngest wires in IDML zip
  extraction and the PDF image cache.
- Runnable via `pnpm pipeline:ingest`, `flavian pipeline ingest`, and the
  `flavian-ingest` bin.
- Tested on an in-memory IDML fixture: the transform, the manifest shape, and
  an end-to-end run whose staged paths line up with generateTheme.

Redence proof token (Pipelines / Data-ETL): RDNC-NWYQDP

Closes #117

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@PAMulligan PAMulligan merged commit 9a9ed9a into main Jun 24, 2026
20 checks passed
@PAMulligan PAMulligan deleted the 117-redence-verify-pipelines-dataetl-add-an-idmlpdf-ingest-stage-to-the-build-pipeline branch June 24, 2026 20:38
PAMulligan added a commit that referenced this pull request Jun 24, 2026
feat(pipeline): add asset-staging stage and repair #152 merge fallout
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Redence verify] Pipelines (Data/ETL) — add an IDML/PDF ingest stage to the build pipeline

1 participant