v0.4.0 — PDF support + spec v0.2 (kind: document) - #3
Merged
Conversation
PDF support. The same wire format now applies to documents — convertPdf()
produces a kind: 'document' AgentMark snapshot from PDF bytes.
Spec v0.2 extension
- Adds kind: 'webpage' | 'document' | 'form' discriminator (v0.2+)
- Adds optional document metadata block: pages, author, created_at,
modified_at, format, format_version, ocr_used
- Adds [PAGE:p_n] body tag for page-boundary markers in documents
- Fully backwards-compatible: v0.1 snapshots without kind still validate
- New schema/agentmark-v0.2.json; validator picks v0.1 or v0.2 based on
declared agentmark version
- AGENTMARK_VERSION constant bumped from '0.1' to '0.2'
PDF converter
- convertPdf({ data, sourceUrl, ... }): main entry. Returns the same
ConversionResult shape as convertPage() so downstream LLM pipelines
are uniform regardless of source surface.
- extractPdf(): lower-level extraction returning structured PdfDocument
(positioned text items + metadata) for callers wanting custom
structural inference.
- buildBodyFromPdf(): body-segment builder consumed by convertPdf,
exposed for callers wanting a different envelope.
- Heading detection via font-size outliers (configurable threshold).
- Bullet + ordered list detection via leading-glyph patterns.
- Paragraph reflow with vertical-gap-based break detection.
- PDF metadata parser handles non-ISO PDF date format
(D:YYYYMMDDHHMMSS+HH'mm' → ISO 8601).
Dependencies
- pdfjs-dist@^4 added as optional peer dependency (web-only callers
pay no install cost). Lazy-imported via dynamic import; throws clean
SnapshotError with install instructions if missing.
- pdf-lib added as devDependency for test-fixture generation
(PDFs constructed in-process, not committed as binaries).
Tests (166 unit + 10 real-Chromium = 176 total, all passing)
- 13 new spec-v0.2 tests covering kind discriminator, document metadata,
PAGE markers, version negotiation, backwards compat with v0.1.
- 12 new PDF converter tests covering metadata extraction, page counts,
heading promotion, bullet detection, PAGE markers, title fallback,
vendor extensions, logger event flow, error handling on garbage input.
Not yet shipped (deferred to v0.5)
- OCR for scanned PDFs (interface designed via document.ocr_used flag)
- Table detection
- AcroForm support (M3)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ening) Builds a diagnostic CLI for evaluating PDF→AgentMark quality on real-world documents (county forms, etc.) before extending PDF features. Also fixes two real bugs surfaced by running the diagnostic on the first sample. Bug fixes - pdf-extractor: defensively copy input bytes before passing to pdfjs-dist. pdfjs-dist (a) does a strict prototype check that rejects Node's Buffer even though it extends Uint8Array, and (b) transfers ownership of the underlying ArrayBuffer during parse, so calling extractPdf twice on the same data fails with "Cannot perform Construct on a detached ArrayBuffer". Both regressions now have unit tests. Diagnostic tool — examples/diagnose-pdf.ts - Per-page diagnostics: text-item count, font size distribution, median + outlier detection, suspected-scan flag (zero text items), suspected-multi-column flag (X-coordinate clustering) - Body-builder analysis: heading/paragraph/list counts, page-marker count - AgentMark size + estimated token cost - Quality score (0-100, heuristic) - Aggregated flag counts across a corpus - Suggestions tied to specific failure modes (OCR, multi-column, etc.) - Outputs Markdown report; --out flag writes to file - Accepts a single PDF or a directory of PDFs Use: npx tsx examples/diagnose-pdf.ts <pdf-or-dir> [--out report.md] Tests - 168 total now (was 166), 14 PDF tests including 2 new regression tests for the Buffer + ArrayBuffer-detachment fixes devDeps - tsx@^4 added so examples can be run with npx tsx without external installation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rance corpus)
Validation findings from a 12-doc real-world insurance corpus drive two
hardening improvements.
Improvements
- Bold-font heading detection in body-builder. Detects headings encoded
via font *weight* (e.g. "Helvetica-Bold") at body-sized point sizes,
not just outlier sizes. Working on PDFs with proper bold encoding (e.g.
Eventbrite tickets gained 6 headings; previously 0). Conservative
guards: max 80 chars, ≤20% size delta from median, all items must use
bold font names.
- Diagnostic CLI now classifies source mode into:
- real_text: text streams present — extraction works
- print_to_pdf_vector: glyphs rendered as filled paths (Microsoft
Print To PDF / similar — needs OCR or original source)
- scan: image-only pages (scanner output — needs OCR)
- mixed: some text + some image pages
- empty / unknown
Classification uses producer metadata + operator histograms
(showText vs paintImageXObject vs constructPath/fill).
- Diagnostic prints source-mode breakdown table + per-mode suggestions
so v0.5 priorities are obvious from the report alone.
New investigation tools
- examples/probe-pdf.ts: dump operator histogram, metadata, font count
for a single problem PDF.
- examples/dump-fonts.ts: enumerate distinct fonts + sample text per
font to debug heading-detection failures.
Insurance corpus results (12 docs)
- real_text: 6 (50%) — extraction works (FB renewals, tickets, CORP
Articles, PRINTHEAD AGREEMENT)
- print_to_pdf_vector: 4 (33%) — Erie auto/home quotes printed via
"Microsoft: Print To PDF"
- scan: 2 (17%) — Flood Map screenshot, NC reseller cert (Epson
ScanSmart)
- 0 outright failures
- All 6 failing docs need OCR — that's the v0.5 priority
Limitations surfaced (deferred to v0.5)
- Form-style PDFs (insurance renewals) use anonymized embedded fonts
and have field labels at SMALLER font sizes than body text, so neither
size-based nor weight-based heading inference applies. Form-structure
detection (label/value pairs) is a v0.5 feature paired with M3 AcroForm
support.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 10, 2026
This was referenced May 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PDF support — the same wire format now applies to documents. `convertPdf()` produces a `kind: 'document'` AgentMark snapshot from PDF bytes, mirroring `convertPage()` for web pages.
Stacked on top of #2 (M1 / v0.3.0). Merge that one first.
Spec v0.2
Backwards-compatible extension — v0.1 snapshots without `kind` still validate.
PDF converter
Dependencies
Tests
176 total, all passing:
Not in this PR (v0.5 / M3)
Test plan
🤖 Generated with Claude Code