feat(pipeline): add format-detecting ingest stage with content extraction#146
Merged
Conversation
…tion
Add a front-of-pipeline ingest stage for the InDesign->FSE pipeline. Where the
per-format parse bins dispatch on file extension and only emit design/geometry,
this stage sniffs the source bytes (Zip + designmap.xml = IDML, %PDF- = PDF),
parses to the canonical IR, re-validates it against the schema, and derives a
reading-order content model: each editorial spread becomes a section of blocks
(headings with font-size-inferred levels h1-h6, paragraphs, figures), plus a
flat outline and stats.
The content model is attached under an additive `content` key the mapper and
generator ignore, so the artifact remains a valid Document and the shared IR
schema is not forked. Exposed as `flavian-ingest` and `pnpm pipeline:ingest`.
- src/indesign/ingest/{detect,content,index}.js + bin/ingest.mjs
- exports + bin wiring + root pnpm script
- tests: byte detection, content transform, end-to-end ingest (IDML + PDF),
determinism, and generator-consumability
- docs/pipeline/ingest-stage.md
Redence-Proof-Token: RDNC-8EU6UU
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4 tasks
11 tasks
PAMulligan
pushed a commit
that referenced
this pull request
Jun 22, 2026
…p-tokens, and flavian indesign
The positional-capture guard `!arg.startsWith('-')` incorrectly rejected
a bare `-` because `'-'.startsWith('-')` is true. Add `arg === '-'` as
an explicit early condition so the stdin sentinel is captured before the
dash check, matching the pattern already used in flavian-ingest (ref #146).
Affected: packages/pipeline/bin/generate-theme.mjs:68
packages/pipeline/bin/map-tokens.mjs:60
bin/flavian.mjs:80
Fixes #148
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new ingest stage at the front of the InDesign→FSE pipeline
(
ingest → map → generate). Unlike the per-formatparse-idml/parse-pdfbins — which dispatch on file extension and emit only design/geometry — this
stage sniffs the source bytes and extracts semantic content that the IR
previously didn't capture.
detect.js— format detection by magic bytes:PK+ the literaldesignmap.xmlmanifest name ⇒ IDML (zip filenames are stored uncompressed,so this works even though entry data is deflated);
%PDF-⇒ PDF. Works onextensionless files and streams.
content.js— the ETL transform: walks spreads → frames → stories →styles to build a reading-order content model (headings with font-size-
inferred levels h1–h6, paragraphs, figures), a flat outline, and stats. Pure
and deterministic.
ingest/index.js—ingestSource/ingestBuffer(detect → parse →Document.parsere-validate → enrich) andtoArtifact.bin/ingest.mjs—flavian-ingestCLI (file or stdin,--out,--format,--dpi,--no-content). Runnable viapnpm pipeline:ingest.The content model is attached under an additive
contentkey the mapper andgenerator ignore, so the artifact stays a valid
Documentand the shared IRschema is not forked (project invariant). The enriched artifact pipes straight
into
generate-theme.Test plan
New tests (fixtures generated in code — no binary blobs, per pipeline convention):
ingest-detect.test.mjs— IDML/PDF/plain-zip/garbage detection.ingest-content.test.mjs— reading-order sort, multi-level heading inference,figures, run grouping, dangling-ref accounting.
ingest.test.mjs— end-to-end IDML + PDF ingest, determinism, and proof theartifact is generator-consumable.
Verified locally:
pnpm --filter @flavian/pipeline test→ 118 pass / 0 fail (18 new).tests/fixtures/indesign/fixtures.test.mjs→ pass (no regression).from an extensionless file; unrecognized input exits 1.
commitlinton the commit message → pass.🤖 Generated with Claude Code