v0.8–v0.11 — signatures (heuristic + vision) + audio + video - #8
Merged
Merged
Conversation
…+ heuristic image) Detects signatures across the three real-world patterns we hit on insurance + government docs and surfaces them in the AgentMark snapshot envelope so agents can answer "who signed what". Architecture - SignatureDetector interface — same plug-and-play shape as OcrBackend. - DetectedSignature record with: kind, page, rect, field_name, inferred_role, signer_name, signer_email, signed_at, confidence, valid, notes. - SignatureDescriptor type added to Snapshot.signatures (additive to v0.2 spec). - New body tag [SIGNATURE:sig_n] with cross-field validation. Three reference detectors (default chain) 1. AcroFormSignatureDetector Walks AcroForm /Sig widgets via the existing extractor. Distinguishes signed vs unsigned via value presence; role from field name. 2. LabelPatternSignatureDetector Catches the gov-form pattern: text fields labeled "Signature Of Employee" etc. that aren't actual /Sig widgets. Critical for IRS/USCIS forms. 3. HeuristicImageSignatureDetector Walks the page operator list, computes image XObject rects from CTM (full PDF affine matrix multiply), filters by signature shape (aspect ratio 1.2-12, width 60-400pt, height 12-100pt), requires proximity to either a "Signature/Sign here/X" label OR a role keyword. Catches hand-signed scanned PDFs that the AcroForm/label detectors miss. Role inference (free, deterministic) - inferRoleFromFieldName: 22 role-keyword patterns matched against normalized snake/camel/kebab/dotted field names. Covers client, agent, broker, buyer, seller, tenant, landlord, witness, notary, guarantor, attorney, employer, employee, applicant, beneficiary, insured/insurer, policyholder, principal, authorized signator(y|ies), co-buyer/co-seller. - inferRoleFromNearbyText: scans text items in the label zone above a signature region (radius=60pt vertically, ±60pt horizontally). Returns matching role + diagnostic snippet for transparency. Pipeline - detectSignatures(input, detectors?) runs all detectors in parallel, dedupes by (page, IoU > 0.5) keeping higher confidence, renumbers IDs to clean sig_1..sig_N. Detector failure is per-detector best-effort (one failing doesn't abort the others). Wired into convertPdf - New options.signatureDetectors (custom chain or [] to disable). - snapshot.signatures populated when ≥1 detection. - Body tag [SIGNATURE:sig_n] insertion deferred to a follow-up — for now signatures live in the envelope only and are LLM-discoverable via the snapshot. Spec + schema updates - agentmark-v0.2.json gains the signatures map + signature $def with full field validation (kind enum, page minimum, confidence range). - BodyTagKind union adds 'SIGNATURE'. - Validator rejects body refs to undefined signature IDs. Tests (243 passing, 34 new) - Role-inference: 23 field-name + 5 nearby-text tests - Pipeline: 6 tests covering empty doc, ID renumbering, IoU dedup, non-overlapping retention, default-chain composition, fault tolerance Validation against real corpus pending — running diagnostic against user's insurance + government + new files in background. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world finding from running the v0.8 detectors against a corpus of
~50 mixed personal/business PDFs (insurance, rentals, gov forms, scans):
- Form-builder-generated AcroForm Sig widgets often have random field
names (e.g. "HelloSignature_79936460") that no role pattern matches.
- Text-field-as-signature labels are sometimes generic ("Provide R
Signature") rather than role-bearing.
Both AcroFormSignatureDetector and LabelPatternSignatureDetector now
fall back to inferRoleFromNearbyText when neither field name nor label
yields a role. Found "LANDLORD:" near a HelloSign widget in the rental
contract test → correctly inferred role: landlord.
Notes field now records which tier of inference was used (field name /
label / nearby text + snippet) for transparency.
No new tests — existing role-inference tests cover the helpers; this
change is composition only. Validated via re-running the corpus
diagnostic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vision-based detection unlocks the cases v0.8 heuristics miss: - Hand-signed scans where the whole page is a single rasterized image - Signatures without nearby text labels - Typed cursive-name signatures - Anywhere bbox + role inference can be done from rendered pixels Architecture - VisionBackend interface — same plug-and-play shape as OcrBackend. Generic enough to be reused by video frame captioning in v0.11. - ClaudeVisionBackend — calls Anthropic /v1/messages with image input + tool-use for structured output. No SDK dep — uses fetch. Default model: claude-haiku-4-5-20251001. - OpenAiVisionBackend — calls /v1/chat/completions with image_url + response_format: json_schema. No SDK dep. Default: gpt-4o-mini. - Both authenticate via env (ANTHROPIC_API_KEY / OPENAI_API_KEY) or constructor option. VisionSignatureDetector - Composes any RenderBackend (Poppler/pdfjs) with any VisionBackend. - Cost-conscious by default: pages='last' scans only the last 2 pages where signatures usually live. Override via 'all' / 'flagged' (pages whose extracted text contains a signature label) / number[]. - Renders → vision JSON-schema query → DetectedSignature[] with bbox converted from normalized 0-1 to PDF user-space. - Min confidence threshold filters low-confidence vision detections. - Per-page failures don't abort the run. Validation contract - Schema enforces kind enum + 0-1 ranges on bbox + confidence so the vision model's response can be trusted as structured. - "unknown" role is dropped (not surfaced to the user). - inferred_role is lowercased for downstream consistency. Tests (9 new, 262 total) - 'last' / 'all' / 'flagged' / explicit page-list modes - bbox → rect coordinate conversion - minConfidence filtering - role lowercase + 'unknown' drop - per-page failure tolerance Wired into the public API - Exported from src/pdf/vision/index.ts and re-exported at src/pdf and src/index. Both Claude + OpenAI backends ship as ready-to-use reference implementations; bring-your-own (Anthropic Claude Code, Apple Vision, Gemini, etc.) by implementing the interface. Not yet wired - VisionSignatureDetector is NOT in defaultDetectors() — it costs API $$ per page. Callers opt in by adding it to the chain. Future MCP pdf_open could expose enable_vision_signatures=true. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the full audio surface:
audio bytes → TranscriptionBackend → AgentMark snapshot (kind: 'audio')
with timestamps, optional speaker diarization, and the same wire format
any AI client already knows. Customer calls, voicemails, meeting
recordings, podcasts — all become structured AgentMark.
Spec v0.3
- kind enum extended: webpage|document|form|audio|video
- New `media_meta` envelope field: duration_sec, format, language,
transcribed, transcription_backend, vision_backend, speaker_count,
frame_count
- New `speakers` map: speaker IDs → display names
- Three new body tags:
[TIME:t_N] timestamp marker (N = seconds-from-start)
[SPEAKER:s_X] speaker label, resolves to envelope.speakers
[FRAME:f_N] video frame ref (reserved for v0.11)
- agentmark-v0.3.json schema; validator picks v0.1/v0.2/v0.3 by
declared version. v0.2 docs continue to validate unchanged.
- AGENTMARK_VERSION bumped 0.2 → 0.3.
TranscriptionBackend interface
- transcribe({ data, mimeType, language, diarize }) → segments + full_text
- WhisperApiBackend: OpenAI /v1/audio/transcriptions
multipart/form-data + verbose_json for segment-level timestamps
no SDK dep, fetch-based, OPENAI_API_KEY auth
- Sniffs MIME from magic bytes: mp3/wav/ogg/m4a/flac/webm
- Bring-your-own AssemblyAI / Deepgram / Apple Speech / whisper.cpp
convertAudio()
- Same contract as convertPdf — bytes in, ConversionResult out.
- Body builder emits one [TIME:t_N] per segment + [SPEAKER:s_X] when
diarized. Same-speaker continuing segments omit the SPEAKER tag for
compactness. Body text escaped (matches web body builder rules) so
transcripts containing literal "[PAGE:p_1]" don't become bogus refs.
- Falls back to URL basename for title; counts distinct speakers when
no speakers map provided; preserves vendor (x-) extensions.
Tests (271 passing, 9 new for audio)
- Round-trip: backend → snapshot → parsed → validated v0.3
- Diarized + non-diarized paths
- Empty-segments fallback to full_text
- Speaker dedup logic
- Title fallback
- Vendor extensions
- Backend-failure SnapshotError wrapping
- Body-text escaping (literal [TAG] doesn't become a ref)
- Updated 3 prior tests to expect new agentmark version 0.3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(v0.11)
The final media surface. Video bytes become an AgentMark snapshot with
kind: 'video' that interleaves transcript and visually-captioned frames
in timeline order — so an LLM reading the snapshot sees both audio and
visual context aligned in time.
[TIME:t_0]
[SPEAKER:s_alice] Welcome to the demo.
[TIME:t_30] [FRAME:f_2]
(frame caption: Architecture diagram with three boxes labeled A, B, C.)
[TIME:t_60]
[SPEAKER:s_alice] Today we'll cover the three pieces.
Reuses the v0.10 audio + v0.9 vision pipelines — no new long-lived
backend interfaces. Just a frame extractor + an orchestrator.
FrameExtractionBackend interface
- extractFrames({ data, sampling, width, format }) → ExtractedFrame[]
- Sampling strategies: { every: N seconds } | { count: N frames } | { keyframes: true }
FfmpegFrameBackend (reference impl)
- Shells out to system `ffmpeg` (and `ffprobe` for duration).
- Required: brew install ffmpeg / apt-get install ffmpeg.
- Throws clean SnapshotError with install instructions if missing.
- Atomic temp dir per call; cleaned up on completion.
- Emits jpegs by default at 800px width — good vision-LLM input.
convertVideo() orchestrator
- Runs frame extraction + transcription in parallel (when transcribe
is configured).
- Captions each extracted frame in series via the caller-provided
VisionBackend (the same interface introduced in v0.9 for signature
detection — no new vision impl needed).
- Builds a unified timeline of {transcript, frame} events sorted by
timestamp, emits [TIME] / [SPEAKER] / [FRAME] tags as appropriate.
- Frames go into the existing `media` map as `image` entries with
captions; [FRAME:f_n] resolves through the existing MEDIA-resolving
validator path (no new validator code).
- Per-frame caption failures are logged + skipped (graceful).
- transcribe=null → frames-only video (no audio extraction).
- caption=null → [FRAME] markers without descriptions (cheaper).
- Both null → fail fast with SnapshotError ("no frames and no transcript").
Validator
- [FRAME] body tag added to MEDIA_RESOLVING set — must reference an
envelope.media entry (cross-field check).
Tests (278 total, 7 new for video — 252 → 268 → 271 → 278)
- Interleaved transcript + frames in correct timeline order
- v0.3 schema validation passes
- caption=null path
- transcribe=null path
- both=null fails fast
- Per-frame caption failure tolerated (graceful, frame still emitted)
- Out-of-order insertion still produces ordered timeline
Public API
- convertVideo + FfmpegFrameBackend + types exported at /src and /index.
- ffmpeg is NOT a peer dep (it's a system binary). Documented in
TESTING.md / README.
Version bump
- @thinkfleet/agentmark: 0.7.0 → 0.11.0 (skipping 0.8/0.9/0.10 since
signatures + vision sigs + audio + video all landed together in this
iteration cycle and we haven't published yet).
- Description rewritten to capture the full surface coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four milestones in one PR — bundles all the AI-readable surfaces AgentMark didn't yet cover: signatures (heuristic + vision-based), audio (transcription), video (transcription + frame captioning).
After this PR, AgentMark covers every consumable AI surface under one wire format:
What's new
v0.8 — Heuristic signature detection
v0.9 — Vision signatures
v0.10 — Audio support
v0.11 — Video support
Tests (278 total — 252 → 271 → 278)
Known limitations / explicitly deferred
Test plan
🤖 Generated with Claude Code