v0.3 → v0.11 rollup — full surface coverage (web, PDF, OCR, AcroForm, MCP, AP piece, signatures, vision, audio, video) - #9
Merged
Conversation
PDF support. The same wire format now applies to documents — convertPdf()
produces a kind: 'document' AgentMark snapshot from PDF bytes.
Spec v0.2 extension
- Adds kind: 'webpage' | 'document' | 'form' discriminator (v0.2+)
- Adds optional document metadata block: pages, author, created_at,
modified_at, format, format_version, ocr_used
- Adds [PAGE:p_n] body tag for page-boundary markers in documents
- Fully backwards-compatible: v0.1 snapshots without kind still validate
- New schema/agentmark-v0.2.json; validator picks v0.1 or v0.2 based on
declared agentmark version
- AGENTMARK_VERSION constant bumped from '0.1' to '0.2'
PDF converter
- convertPdf({ data, sourceUrl, ... }): main entry. Returns the same
ConversionResult shape as convertPage() so downstream LLM pipelines
are uniform regardless of source surface.
- extractPdf(): lower-level extraction returning structured PdfDocument
(positioned text items + metadata) for callers wanting custom
structural inference.
- buildBodyFromPdf(): body-segment builder consumed by convertPdf,
exposed for callers wanting a different envelope.
- Heading detection via font-size outliers (configurable threshold).
- Bullet + ordered list detection via leading-glyph patterns.
- Paragraph reflow with vertical-gap-based break detection.
- PDF metadata parser handles non-ISO PDF date format
(D:YYYYMMDDHHMMSS+HH'mm' → ISO 8601).
Dependencies
- pdfjs-dist@^4 added as optional peer dependency (web-only callers
pay no install cost). Lazy-imported via dynamic import; throws clean
SnapshotError with install instructions if missing.
- pdf-lib added as devDependency for test-fixture generation
(PDFs constructed in-process, not committed as binaries).
Tests (166 unit + 10 real-Chromium = 176 total, all passing)
- 13 new spec-v0.2 tests covering kind discriminator, document metadata,
PAGE markers, version negotiation, backwards compat with v0.1.
- 12 new PDF converter tests covering metadata extraction, page counts,
heading promotion, bullet detection, PAGE markers, title fallback,
vendor extensions, logger event flow, error handling on garbage input.
Not yet shipped (deferred to v0.5)
- OCR for scanned PDFs (interface designed via document.ocr_used flag)
- Table detection
- AcroForm support (M3)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ening) Builds a diagnostic CLI for evaluating PDF→AgentMark quality on real-world documents (county forms, etc.) before extending PDF features. Also fixes two real bugs surfaced by running the diagnostic on the first sample. Bug fixes - pdf-extractor: defensively copy input bytes before passing to pdfjs-dist. pdfjs-dist (a) does a strict prototype check that rejects Node's Buffer even though it extends Uint8Array, and (b) transfers ownership of the underlying ArrayBuffer during parse, so calling extractPdf twice on the same data fails with "Cannot perform Construct on a detached ArrayBuffer". Both regressions now have unit tests. Diagnostic tool — examples/diagnose-pdf.ts - Per-page diagnostics: text-item count, font size distribution, median + outlier detection, suspected-scan flag (zero text items), suspected-multi-column flag (X-coordinate clustering) - Body-builder analysis: heading/paragraph/list counts, page-marker count - AgentMark size + estimated token cost - Quality score (0-100, heuristic) - Aggregated flag counts across a corpus - Suggestions tied to specific failure modes (OCR, multi-column, etc.) - Outputs Markdown report; --out flag writes to file - Accepts a single PDF or a directory of PDFs Use: npx tsx examples/diagnose-pdf.ts <pdf-or-dir> [--out report.md] Tests - 168 total now (was 166), 14 PDF tests including 2 new regression tests for the Buffer + ArrayBuffer-detachment fixes devDeps - tsx@^4 added so examples can be run with npx tsx without external installation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rance corpus)
Validation findings from a 12-doc real-world insurance corpus drive two
hardening improvements.
Improvements
- Bold-font heading detection in body-builder. Detects headings encoded
via font *weight* (e.g. "Helvetica-Bold") at body-sized point sizes,
not just outlier sizes. Working on PDFs with proper bold encoding (e.g.
Eventbrite tickets gained 6 headings; previously 0). Conservative
guards: max 80 chars, ≤20% size delta from median, all items must use
bold font names.
- Diagnostic CLI now classifies source mode into:
- real_text: text streams present — extraction works
- print_to_pdf_vector: glyphs rendered as filled paths (Microsoft
Print To PDF / similar — needs OCR or original source)
- scan: image-only pages (scanner output — needs OCR)
- mixed: some text + some image pages
- empty / unknown
Classification uses producer metadata + operator histograms
(showText vs paintImageXObject vs constructPath/fill).
- Diagnostic prints source-mode breakdown table + per-mode suggestions
so v0.5 priorities are obvious from the report alone.
New investigation tools
- examples/probe-pdf.ts: dump operator histogram, metadata, font count
for a single problem PDF.
- examples/dump-fonts.ts: enumerate distinct fonts + sample text per
font to debug heading-detection failures.
Insurance corpus results (12 docs)
- real_text: 6 (50%) — extraction works (FB renewals, tickets, CORP
Articles, PRINTHEAD AGREEMENT)
- print_to_pdf_vector: 4 (33%) — Erie auto/home quotes printed via
"Microsoft: Print To PDF"
- scan: 2 (17%) — Flood Map screenshot, NC reseller cert (Epson
ScanSmart)
- 0 outright failures
- All 6 failing docs need OCR — that's the v0.5 priority
Limitations surfaced (deferred to v0.5)
- Form-style PDFs (insurance renewals) use anonymized embedded fonts
and have field labels at SMALLER font sizes than body text, so neither
size-based nor weight-based heading inference applies. Form-structure
detection (label/value pairs) is a v0.5 feature paired with M3 AcroForm
support.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pages with no extractable text — scanner output, "Microsoft Print To PDF"
exports, image-only PDFs — can now be rasterized + OCR'd transparently.
Two render backends + two OCR backends ship; the interfaces let callers
plug in any provider.
Real-world impact (insurance corpus, 12 docs)
🟢 ≥70 🟡 30-69 🔴 <30
Without OCR (v0.4) 6 6 0
With OCR (Poppler+Tesseract) 12 0 0 ← all docs handled
Architecture
- OcrBackend / RenderBackend interfaces. Minimal, plug-and-play.
- convertPdf({ ocr: { render, ocr, mode } }) — opt-in pipeline:
'auto' (default) — OCR only pages with no extractable text
'always' — OCR every page (overrides extracted text)
'never' — disable OCR entirely
- Pages with extractable text are not re-OCR'd in 'auto' mode (cost
optimization), making mixed text+image PDFs cheap.
- document.ocr_used flag set to true when OCR was applied.
- Capabilities map: ocr: true on snapshots produced via OCR.
Bundled render backends
- PopplerRenderBackend — shells out to pdftoppm. Lightest, no native modules.
- PdfjsRenderBackend — pure-Node via pdfjs-dist + node-canvas (optional).
Bundled OCR backends
- TesseractOcrBackend — in-process WASM. Free, offline, ~10s/page.
Long-lived worker reused across pages; close()
terminates it. Honors language hint.
- MistralOcrBackend — cloud API. ~$1/1k pages, best quality. Auth
via MISTRAL_API_KEY env var or constructor opt.
Bring-your-own — AWS Textract, Google Document AI, Apple Vision Framework,
etc. all fit the same OcrBackend / RenderBackend shape. Reference impls
welcome via PR.
Devex
- examples/ocr-pdf.ts demonstrates Poppler + Tesseract end-to-end.
- examples/diagnose-pdf.ts gains a --ocr flag; quality scores reflect
whether OCR rescued otherwise-failing docs.
Tests
- 8 new OCR pipeline tests with mocked backends (deterministic, fast).
- Total: 176 unit + 10 real-Chromium = 186 (was 176).
Optional peer deps
- tesseract.js@^5 (Tesseract backend)
- canvas (PdfjsRenderBackend)
- pdfjs-dist@^4 (already opt-peer; required for both render backends and
text extraction generally)
Not in this release
- AWS Textract / Google Document AI / Apple Vision adapters — interface
ships, community impls welcome
- Form-structure inference (label/value pairs on non-AcroForm PDFs) —
paired with M3 / v0.6
- AcroForm support — M3 / v0.6
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PDFs with AcroForm fields are now first-class. They snapshot as
kind: 'form' with each field exposed as an AgentMark action; the new
PdfDocument SDK class lets agents fill, save, and flatten them with the
same execute() shape as the web Page SDK.
AcroForm extraction
- Reads fields via pdfjs-dist's getFieldObjects() and merges with page
annotations to recover Required/ReadOnly flags (which getFieldObjects
doesn't surface in pdfjs-dist v4+).
- Maps AcroForm field types to AgentMark action types:
text (single + multi-line) → type: 'type'
text (password flag) → type: 'type', label: '(redacted)'
checkbox → type: 'check'
radio group → type: 'select' with options
dropdown (combo) → type: 'select' with options
listbox single → type: 'select'
listbox multi (multipleSelection) → type: 'multi_select'
signature → type: 'click', disabled
push button → type: 'click'
- Sensitive field-name redaction (password, ssn, credit_card, cvv,
account_num, token, secret, csrf, session, auth) — labels become
'(redacted)' and values are dropped.
- Field-name humanization: applicant.first_name / firstName /
first-name all → "First Name".
- Action IDs synthesized as act_field_N to satisfy AgentMark schema
regardless of source-name irregularity. Original field names
preserved in the binding map for fill operations.
- Filters out parent fields (empty type with kidIds) — only leaf
widgets with real metadata are processed.
- Dedicated unit-tested coercion for checkbox values (PDF's "Yes"/"Off"
→ boolean, fallback paths for other PDF generators).
PdfDocument SDK class (new public API)
- openPdfDocument({ data, sourceUrl, ... }) factory mirrors the web
Page SDK shape so callers' agent loops are uniform.
- snapshot() — capture current form state (cached)
- execute(actionId, value) — queue a field value with type validation
- save({ flatten? }) — write a new PDF with all queued values
applied; flatten bakes values into the
page content (no longer fillable)
- reset() — discard queued values
- close() — release resources, idempotent
- fields, pending, snapshotCache — read-only accessors
- All execute() type checks throw the existing AgentMark error
hierarchy: ActionTypeError, ActionDisabledError, ActionNotFoundError,
ExecutionError. Read-only and signature fields auto-refused.
- Defensive copy of input bytes — multiple snapshot/save calls work.
Internal type rename
- The internal extraction-result interface PdfDocument was renamed to
ExtractedPdf to free the PdfDocument name for the new public class.
Internal-only — no consumer code depended on the old name through
the public API surface.
Optional peer dependency
- pdf-lib added as an optional peer dep. Reading + extracting fields
uses pdfjs-dist (already installed); writing requires pdf-lib.
Surface a clean SnapshotError with install instructions if missing.
Tests (199 unit + 10 real-Chromium integration = 209 total)
- 12 new acroform-extractor tests (all field types, redaction,
required/read-only flags, humanization, ID schema compliance).
- 11 new PdfDocument round-trip tests (fill → save → re-extract for
every field type, error semantics, flatten path, close idempotency).
Known limitations (deferred)
- pdfjs-dist's getFieldObjects() reports only the first selected value
for multi-select listboxes. AgentMark's saved PDF DOES contain all
values (verified via direct pdf-lib reading) — only the snapshot
view under-reports. Wait for pdfjs-dist upstream support.
- Form-structure inference for non-AcroForm PDFs (the FB renewal case
— visual field labels but no AcroForm dictionary) is deferred to
a later release.
- Signature fields surface as disabled actions; AgentMark refuses to
fulfill them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps every public surface — web browser, PDF documents, AcroForm
filling, OCR — as a Model Context Protocol server. Any MCP client
(Claude Desktop, Cursor, Claude Code, custom agents) can use AgentMark
through one configuration entry. No SDK install, no language commitment.
Configure in any MCP client:
{
"mcpServers": {
"agentmark": {
"command": "npx",
"args": ["-y", "@thinkfleet/agentmark", "agentmark-mcp"]
}
}
}
15 tools exposed
- Browser (3): browser_open / browser_close / browser_save_session
- Page (5): page_open / page_navigate / page_snapshot / page_execute / page_close
- PDF (6): pdf_open (file path or data: URI) / pdf_close / pdf_snapshot /
pdf_execute / pdf_save (with optional flatten) / pdf_reset
- Meta (1): list_sessions for debugging stuck connections
Architecture
- src/mcp/tool-defs.ts: declarative tool catalog with full JSON Schema
input descriptions
- src/mcp/dispatcher.ts: pure dispatch function — name + args → result.
Stateless except for the session registries it
receives. Tested directly without transport.
- src/mcp/server.ts: MCP Server wiring + StdioServerTransport binding +
SIGINT/SIGTERM cleanup
- src/mcp/cli.ts: Bin entry (#!/usr/bin/env node) — what npm
installs as `agentmark-mcp` on PATH
Stateful session model
- Long-lived browsers + pages + opened PDF documents are held server-side,
keyed by short opaque IDs returned from _open calls. Agents can drive
multiple parallel surfaces from one MCP connection.
- All resources auto-released on disposeAll() — invoked on shutdown
via SIGINT/SIGTERM and exposed for tests.
Programmatic API
- createMcpServer() / startMcpServer() exported for embedding the
server in larger applications.
- dispatch() exported for tests / custom transports.
Optional peer dependency
- @modelcontextprotocol/sdk@^1 added as optional peer. Library callers
who don't run the MCP server pay no install cost.
bin entry
- package.json now has "bin": { "agentmark-mcp": "./dist/src/mcp/cli.js" }
- npm sets executable bit on install; shebang preserved through tsc
Tests (213 unit + 10 real-Chromium = 223 total, was 199)
- 14 dispatcher tests:
- tool registry shape (uniqueness, naming prefix, schema validity)
- error semantics (unknown tool, missing args, unknown session ID,
AgentMark error code prefix surfacing)
- PDF round trip (open → snapshot → execute → save with file IO)
- reset clears pending values
- data-URI PDF source (base64 in-memory loading)
- list_sessions (empty + populated)
- disposeAll closes everything
- 4 wire-level handshake tests via InMemoryTransport — exercises the
full MCP protocol (handshake, ListTools, CallTool, error responses)
without spawning a subprocess
Distribution unlocked
After npm publish, anyone can configure AgentMark in any MCP client with
the snippet above. The full SDK (web + PDF + OCR + AcroForm fill/save)
becomes available as 15 tools any agent can call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sibling package in pieces/agentmark/ that wraps the core AgentMark library
as an Activepieces piece. Drop into any flow to convert web pages or PDFs
into AgentMark snapshots and fill PDF forms — no code, no SDK install for
the flow author.
Three actions (v1)
- Capture Web Page (snapshot_web_page)
URL → AgentMark snapshot. Launches Chromium, snapshots the page,
closes browser. Configurable wait_until + timeout + headless.
Output: { agentmark, url, title, kind, action_count, bytes,
captured_at }
- Capture PDF (snapshot_pdf)
PDF source (URL / file path / file:// / data: URI / bare base64) →
AgentMark snapshot with kind: 'document' or 'form'. Optional OCR
(Tesseract + Poppler) for scanned and "Print To PDF" outputs.
Output: { agentmark, source_url, bytes, ocr_used }
- Fill PDF Form (fill_pdf_form)
Atomic PDF fill operation. Accepts field values keyed by either
AgentMark action ID (act_field_N) or original PDF field name —
matches whichever the caller has. Returns the filled PDF as base64
data URI or raw base64. Optional flatten (bake values into page
content; resulting PDF no longer fillable).
Output: { filled_pdf, bytes, fields_applied[], fields_skipped[],
flattened }
Architecture
- pieces/agentmark/ — sibling package in the same repo, builds + tests
independently from the core library.
- pieces/agentmark/src/lib/common.ts — shared resolveBytes() helper that
accepts URL/path/data URI/base64 strings so all PDF actions take a
uniform `source` prop.
- pieces/agentmark/src/lib/actions/*.ts — one file per action.
- Depends on @thinkfleet/agentmark via file:.. for monorepo dev; bump
to ^0.7.0 (or whatever version is on npm) before publishing.
- Uses @activepieces/pieces-framework for the createPiece + createAction
API; matches conventions from the official AI piece in
activepieces-main.
Tests (13 piece tests + 1 gated browser test)
- Piece-shape assertions: display name, minimum release, exposed
actions, prop schemas
- fill_pdf_form end-to-end:
- Fill by action ID
- Fill by original field name (resolution)
- Unknown keys reported via fields_skipped (no throw)
- return_format=base64 vs data_uri
- flatten removes the form
- Accepts data: URI source (no temp file)
- snapshot_pdf end-to-end (text PDF, no OCR): produces kind: form
- snapshot_web_page: gated on AGENTMARK_INTEGRATION=1 with a local
HTTP server (no external network)
Cross-package hygiene
- Root vitest.config.ts excludes pieces/** from root test discovery
to prevent the symlinked node_modules/@thinkfleet/agentmark from
causing duplicate-module instanceof failures.
- Piece's tsconfig sets rootDir: ./src and preserveSymlinks: true so
TypeScript never emits compiled JS into the parent's src/ via the
monorepo symlink.
Known limitations (deferred)
- No multi-step "browser_workflow" action yet — for now each web action
launches a fresh browser. Stateful flows need to fit in a single
action call. Multi-step is on the v1.1 roadmap.
- No AP piece-auth (auth: PieceAuth.None()) — none of the v1 actions
need credentials. Future Mistral OCR integration would add custom
auth for the API key.
Distribution unlocks
- Growth OS (built on Activepieces) gets AgentMark as a flow primitive
for free once this piece is registered.
- Every other Activepieces deployment (community + commercial) can
install the piece independently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a single end-to-end demo that exercises every public surface (web SDK, PDF text extraction, PDF OCR, AcroForm fill+save round-trip, MCP dispatcher) plus a step-by-step testing guide covering: - Local install (Node, Chromium, Poppler) - Unit + integration tests - Per-surface smoke tests - Wiring AgentMark MCP server into Claude Desktop - Wiring AgentMark MCP server into OpenClaw (https://openclaw.ai/) - Distribution checklist - Common gotchas + diagnostic CLI tips The kitchen-sink demo accepts an optional corpus directory — when given the user's insurance corpus, it picks an FB renewal for the text-PDF test and an Erie quote for the OCR test, exercising real-world failure modes (Microsoft Print To PDF) end-to-end. Validated locally: 5/5 surfaces pass against insurance corpus in ~19s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two validation tools to drive the "test against more PDFs" workflow:
1. diagnose-pdf --snapshots <dir>
New flag writes each PDF's full AgentMark snapshot (.agentmark.md)
to the given directory alongside the summary report. Lets you
inspect what AgentMark actually produces per file, not just stats.
2. examples/fetch-gov-corpus.ts + gov-corpus-manifest.json
Curated starter manifest of 9 stable IRS/USCIS public PDFs
(W-9, W-4, 1040, 941, 1099-MISC, 1099-NEC, W-2, 1040 instructions,
I-9). Fetcher gracefully reports failed URLs (gov sites move
constantly) and skips already-downloaded files. Verifies each
download starts with %PDF- magic before saving — catches the
common "endpoint returns HTML" case.
Validation findings on the IRS/USCIS corpus
- 9/9 score 80-100 (median 93)
- All real_text source mode (no OCR needed for IRS forms)
- USCIS I-9 has clean human-readable field labels and auto-redacted
a sensitive field (SSN-like name)
- IRS W-9 uses internal codes ("F1 01[0]", "C1 1[0]") — extraction
works but downstream LLM usability is lower; pairs with the I-9
to demonstrate the spread of field-naming conventions in
real-world government PDFs
- Multi-column flag fires on most IRS forms (3-up 1099 layouts);
v0.5 multi-column reading-order inference would improve them
End-to-end workflow now
npx tsx examples/fetch-gov-corpus.ts /tmp/agentmark-gov-corpus
npx tsx examples/diagnose-pdf.ts /tmp/agentmark-gov-corpus --ocr \
--snapshots /tmp/agentmark-snaps \
--out /tmp/gov-corpus-report.md
cat /tmp/gov-corpus-report.md
cat /tmp/agentmark-snaps/irs-i9.pdf.agentmark.md # inspect any one
Add your own URLs to gov-corpus-manifest.json — state SoS filings,
county building permits, etc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…f_open
Plumbs the OCR pipeline through the MCP server so MCP clients (Claude
Desktop, Cursor, OpenClaw, etc.) can OCR scanned + "Microsoft Print To
PDF" outputs without dropping to the SDK.
Changes
- OpenPdfDocumentOptions now accepts an `ocr` field of OcrPipelineOptions.
PdfDocument stores it and threads it through every snapshot() call;
close() disposes the OCR + render backends best-effort.
- agentmark_pdf_open MCP tool gains three new optional fields:
enable_ocr (boolean, default false)
ocr_language (string, default 'eng')
ocr_dpi (number, default 200)
When enable_ocr=true, the dispatcher constructs PopplerRenderBackend
+ TesseractOcrBackend and passes them to openPdfDocument. Backends
live for the full doc session and release on close.
- pdf_open response includes `ocr_enabled` so the caller can confirm
the OCR path is active.
Tests
- 2 new dispatcher tests (enable_ocr=true / default-off) verify the
response shape + lifecycle. 20 MCP tests total (was 18).
User flow now in Claude Desktop
Open the Erie Auto Quote PDF at <path> with enable_ocr=true and
source_url=file:///<path>. Then snapshot it.
→ Claude calls agentmark_pdf_open with enable_ocr: true
→ Server constructs the OCR pipeline, returns doc_id
→ Subsequent agentmark_pdf_snapshot triggers OCR on empty pages
→ Claude receives a fully-extracted snapshot of vector-rendered text
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…+ heuristic image) Detects signatures across the three real-world patterns we hit on insurance + government docs and surfaces them in the AgentMark snapshot envelope so agents can answer "who signed what". Architecture - SignatureDetector interface — same plug-and-play shape as OcrBackend. - DetectedSignature record with: kind, page, rect, field_name, inferred_role, signer_name, signer_email, signed_at, confidence, valid, notes. - SignatureDescriptor type added to Snapshot.signatures (additive to v0.2 spec). - New body tag [SIGNATURE:sig_n] with cross-field validation. Three reference detectors (default chain) 1. AcroFormSignatureDetector Walks AcroForm /Sig widgets via the existing extractor. Distinguishes signed vs unsigned via value presence; role from field name. 2. LabelPatternSignatureDetector Catches the gov-form pattern: text fields labeled "Signature Of Employee" etc. that aren't actual /Sig widgets. Critical for IRS/USCIS forms. 3. HeuristicImageSignatureDetector Walks the page operator list, computes image XObject rects from CTM (full PDF affine matrix multiply), filters by signature shape (aspect ratio 1.2-12, width 60-400pt, height 12-100pt), requires proximity to either a "Signature/Sign here/X" label OR a role keyword. Catches hand-signed scanned PDFs that the AcroForm/label detectors miss. Role inference (free, deterministic) - inferRoleFromFieldName: 22 role-keyword patterns matched against normalized snake/camel/kebab/dotted field names. Covers client, agent, broker, buyer, seller, tenant, landlord, witness, notary, guarantor, attorney, employer, employee, applicant, beneficiary, insured/insurer, policyholder, principal, authorized signator(y|ies), co-buyer/co-seller. - inferRoleFromNearbyText: scans text items in the label zone above a signature region (radius=60pt vertically, ±60pt horizontally). Returns matching role + diagnostic snippet for transparency. Pipeline - detectSignatures(input, detectors?) runs all detectors in parallel, dedupes by (page, IoU > 0.5) keeping higher confidence, renumbers IDs to clean sig_1..sig_N. Detector failure is per-detector best-effort (one failing doesn't abort the others). Wired into convertPdf - New options.signatureDetectors (custom chain or [] to disable). - snapshot.signatures populated when ≥1 detection. - Body tag [SIGNATURE:sig_n] insertion deferred to a follow-up — for now signatures live in the envelope only and are LLM-discoverable via the snapshot. Spec + schema updates - agentmark-v0.2.json gains the signatures map + signature $def with full field validation (kind enum, page minimum, confidence range). - BodyTagKind union adds 'SIGNATURE'. - Validator rejects body refs to undefined signature IDs. Tests (243 passing, 34 new) - Role-inference: 23 field-name + 5 nearby-text tests - Pipeline: 6 tests covering empty doc, ID renumbering, IoU dedup, non-overlapping retention, default-chain composition, fault tolerance Validation against real corpus pending — running diagnostic against user's insurance + government + new files in background. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world finding from running the v0.8 detectors against a corpus of
~50 mixed personal/business PDFs (insurance, rentals, gov forms, scans):
- Form-builder-generated AcroForm Sig widgets often have random field
names (e.g. "HelloSignature_79936460") that no role pattern matches.
- Text-field-as-signature labels are sometimes generic ("Provide R
Signature") rather than role-bearing.
Both AcroFormSignatureDetector and LabelPatternSignatureDetector now
fall back to inferRoleFromNearbyText when neither field name nor label
yields a role. Found "LANDLORD:" near a HelloSign widget in the rental
contract test → correctly inferred role: landlord.
Notes field now records which tier of inference was used (field name /
label / nearby text + snippet) for transparency.
No new tests — existing role-inference tests cover the helpers; this
change is composition only. Validated via re-running the corpus
diagnostic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vision-based detection unlocks the cases v0.8 heuristics miss: - Hand-signed scans where the whole page is a single rasterized image - Signatures without nearby text labels - Typed cursive-name signatures - Anywhere bbox + role inference can be done from rendered pixels Architecture - VisionBackend interface — same plug-and-play shape as OcrBackend. Generic enough to be reused by video frame captioning in v0.11. - ClaudeVisionBackend — calls Anthropic /v1/messages with image input + tool-use for structured output. No SDK dep — uses fetch. Default model: claude-haiku-4-5-20251001. - OpenAiVisionBackend — calls /v1/chat/completions with image_url + response_format: json_schema. No SDK dep. Default: gpt-4o-mini. - Both authenticate via env (ANTHROPIC_API_KEY / OPENAI_API_KEY) or constructor option. VisionSignatureDetector - Composes any RenderBackend (Poppler/pdfjs) with any VisionBackend. - Cost-conscious by default: pages='last' scans only the last 2 pages where signatures usually live. Override via 'all' / 'flagged' (pages whose extracted text contains a signature label) / number[]. - Renders → vision JSON-schema query → DetectedSignature[] with bbox converted from normalized 0-1 to PDF user-space. - Min confidence threshold filters low-confidence vision detections. - Per-page failures don't abort the run. Validation contract - Schema enforces kind enum + 0-1 ranges on bbox + confidence so the vision model's response can be trusted as structured. - "unknown" role is dropped (not surfaced to the user). - inferred_role is lowercased for downstream consistency. Tests (9 new, 262 total) - 'last' / 'all' / 'flagged' / explicit page-list modes - bbox → rect coordinate conversion - minConfidence filtering - role lowercase + 'unknown' drop - per-page failure tolerance Wired into the public API - Exported from src/pdf/vision/index.ts and re-exported at src/pdf and src/index. Both Claude + OpenAI backends ship as ready-to-use reference implementations; bring-your-own (Anthropic Claude Code, Apple Vision, Gemini, etc.) by implementing the interface. Not yet wired - VisionSignatureDetector is NOT in defaultDetectors() — it costs API $$ per page. Callers opt in by adding it to the chain. Future MCP pdf_open could expose enable_vision_signatures=true. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the full audio surface:
audio bytes → TranscriptionBackend → AgentMark snapshot (kind: 'audio')
with timestamps, optional speaker diarization, and the same wire format
any AI client already knows. Customer calls, voicemails, meeting
recordings, podcasts — all become structured AgentMark.
Spec v0.3
- kind enum extended: webpage|document|form|audio|video
- New `media_meta` envelope field: duration_sec, format, language,
transcribed, transcription_backend, vision_backend, speaker_count,
frame_count
- New `speakers` map: speaker IDs → display names
- Three new body tags:
[TIME:t_N] timestamp marker (N = seconds-from-start)
[SPEAKER:s_X] speaker label, resolves to envelope.speakers
[FRAME:f_N] video frame ref (reserved for v0.11)
- agentmark-v0.3.json schema; validator picks v0.1/v0.2/v0.3 by
declared version. v0.2 docs continue to validate unchanged.
- AGENTMARK_VERSION bumped 0.2 → 0.3.
TranscriptionBackend interface
- transcribe({ data, mimeType, language, diarize }) → segments + full_text
- WhisperApiBackend: OpenAI /v1/audio/transcriptions
multipart/form-data + verbose_json for segment-level timestamps
no SDK dep, fetch-based, OPENAI_API_KEY auth
- Sniffs MIME from magic bytes: mp3/wav/ogg/m4a/flac/webm
- Bring-your-own AssemblyAI / Deepgram / Apple Speech / whisper.cpp
convertAudio()
- Same contract as convertPdf — bytes in, ConversionResult out.
- Body builder emits one [TIME:t_N] per segment + [SPEAKER:s_X] when
diarized. Same-speaker continuing segments omit the SPEAKER tag for
compactness. Body text escaped (matches web body builder rules) so
transcripts containing literal "[PAGE:p_1]" don't become bogus refs.
- Falls back to URL basename for title; counts distinct speakers when
no speakers map provided; preserves vendor (x-) extensions.
Tests (271 passing, 9 new for audio)
- Round-trip: backend → snapshot → parsed → validated v0.3
- Diarized + non-diarized paths
- Empty-segments fallback to full_text
- Speaker dedup logic
- Title fallback
- Vendor extensions
- Backend-failure SnapshotError wrapping
- Body-text escaping (literal [TAG] doesn't become a ref)
- Updated 3 prior tests to expect new agentmark version 0.3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(v0.11)
The final media surface. Video bytes become an AgentMark snapshot with
kind: 'video' that interleaves transcript and visually-captioned frames
in timeline order — so an LLM reading the snapshot sees both audio and
visual context aligned in time.
[TIME:t_0]
[SPEAKER:s_alice] Welcome to the demo.
[TIME:t_30] [FRAME:f_2]
(frame caption: Architecture diagram with three boxes labeled A, B, C.)
[TIME:t_60]
[SPEAKER:s_alice] Today we'll cover the three pieces.
Reuses the v0.10 audio + v0.9 vision pipelines — no new long-lived
backend interfaces. Just a frame extractor + an orchestrator.
FrameExtractionBackend interface
- extractFrames({ data, sampling, width, format }) → ExtractedFrame[]
- Sampling strategies: { every: N seconds } | { count: N frames } | { keyframes: true }
FfmpegFrameBackend (reference impl)
- Shells out to system `ffmpeg` (and `ffprobe` for duration).
- Required: brew install ffmpeg / apt-get install ffmpeg.
- Throws clean SnapshotError with install instructions if missing.
- Atomic temp dir per call; cleaned up on completion.
- Emits jpegs by default at 800px width — good vision-LLM input.
convertVideo() orchestrator
- Runs frame extraction + transcription in parallel (when transcribe
is configured).
- Captions each extracted frame in series via the caller-provided
VisionBackend (the same interface introduced in v0.9 for signature
detection — no new vision impl needed).
- Builds a unified timeline of {transcript, frame} events sorted by
timestamp, emits [TIME] / [SPEAKER] / [FRAME] tags as appropriate.
- Frames go into the existing `media` map as `image` entries with
captions; [FRAME:f_n] resolves through the existing MEDIA-resolving
validator path (no new validator code).
- Per-frame caption failures are logged + skipped (graceful).
- transcribe=null → frames-only video (no audio extraction).
- caption=null → [FRAME] markers without descriptions (cheaper).
- Both null → fail fast with SnapshotError ("no frames and no transcript").
Validator
- [FRAME] body tag added to MEDIA_RESOLVING set — must reference an
envelope.media entry (cross-field check).
Tests (278 total, 7 new for video — 252 → 268 → 271 → 278)
- Interleaved transcript + frames in correct timeline order
- v0.3 schema validation passes
- caption=null path
- transcribe=null path
- both=null fails fast
- Per-frame caption failure tolerated (graceful, frame still emitted)
- Out-of-order insertion still produces ordered timeline
Public API
- convertVideo + FfmpegFrameBackend + types exported at /src and /index.
- ffmpeg is NOT a peer dep (it's a system binary). Documented in
TESTING.md / README.
Version bump
- @thinkfleet/agentmark: 0.7.0 → 0.11.0 (skipping 0.8/0.9/0.10 since
signatures + vision sigs + audio + video all landed together in this
iteration cycle and we haven't published yet).
- Description rewritten to capture the full surface coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why a rollup PR
The original stack (PRs #2–#8) was merged into each PR's base branch rather than cascading to `main` — so only PR #2's contents are currently in `main`. This PR brings the remaining 15 commits / ~11,700 lines / 6 milestones into `main` in one merge so the work isn't stranded on stacked branches.
Every commit in this PR was already individually reviewed in the original stack. This rollup just makes them merge to `main` correctly.
What's in here
Surface coverage after this merge
Tests
Validation evidence
Insurance corpus (the original v0.4 → v0.5 validation):
Government corpus (9 IRS + USCIS PDFs): 9/9 ≥80, median 93.
Real signature detection: validated on rental contracts, lease agreements, business name certs — correctly inferred `landlord` / `tenant` / etc.
Known limitations (deferred to v0.12+)
Test plan
After merge
🤖 Generated with Claude Code