Extraction pipeline redesign: parallel extract → LLM validates

[JordanCoin]

Current Architecture (wrong)

LLM (14 min) → done
OR GLiNER → done  
OR spaCy → done
OR Regex → done

Only ONE method runs. LLM processes the entire document text (78K chars for a 32-page contract). Slow, wasteful, no cross-validation.

Proposed Architecture

Phase 1: Extract (parallel, ~30 sec)
├── GLiNER (zero-shot NER, chunked)     → candidates with confidence
├── spaCy (NLP pipeline, chunked)       → candidates with confidence  
└── Regex (pattern matching, instant)   → dates, money, phones, emails

Phase 2: Merge + Dedup (~instant)
├── Fuzzy match duplicates (Clearview AI ≈ Clearvicn Al)
├── Boost confidence for entities found by multiple methods
├── Keep best spelling/normalization per entity
└── Remove obvious junk (< 3 chars, > 80 chars, common words)

Phase 3: LLM Validation (~30 sec)
├── Input: cleaned entity list (~2K chars, not 78K)
├── "Remove junk. Fix OCR errors. What's missing? Score confidence."
├── Identify relationships between validated entities
└── Output: final entity list with relationships

Why this is better

• 40x less LLM input — 2K entity list vs 78K document text
• Cross-validation — entities found by GLiNER AND spaCy get higher confidence
• LLM does judgment, not scanning — what it's actually good at
• Parallel phase 1 — GLiNER + spaCy + Regex run simultaneously
• Total time: ~60 sec instead of 14 min
• Better quality — multiple extraction methods catch different things

LLM validation prompt (compact)

Review these entities extracted from a government document.
Remove junk (boilerplate, OCR artifacts, generic words).
Fix OCR errors in names. Add confidence scores.
Identify relationships between entities.

Entities: [list of ~50 candidates with types]

Return cleaned JSON with entities and relationships.

Implementation notes

• Phase 1 extractors already exist — just need to run all three, not pick one
• Merge logic: group by fuzzy-matched name, keep highest confidence, prefer GLiNER type
• LLM validation is optional — if no LLM available, skip to Phase 2 output
• Regex always runs (catches structured patterns NER models miss)

[JordanCoin]

Codex Adversarial Review

Ran /codex challenge on this proposal. 848K tokens of analysis. Verdict: direction is right, architecture is incomplete. Key issues and a better counter-proposal below.

Findings (from codex)

1. LLM can't validate without evidence context.
"What's missing?" from a flat entity list is guesswork. Relationship extraction needs sentence evidence — the current code already carries raw_text, context, and page_number per entity. A bare list throws away exactly what makes relationships discoverable.

2. Merge stage is too lossy for investigative work.
Fuzzy-collapsing "Clearvicn Al" into "Clearview AI" without preserving both raw mentions + page numbers is a data-integrity bug. In FOIA investigations, the raw OCR text IS evidence — you can't silently rewrite it.

3. Offline path regresses relationships.
Proposal makes LLM optional, but LLM is where relationships come from in the proposal. Current code already extracts relationships in GLiNER (co-occurrence) and spaCy (syntactic parsing). Can't regress that.

4. RAM pressure on low-resource machines.
Running GLiNER (2GB model) + spaCy simultaneously on a $300 laptop could cause swapping. THREAT_MODEL.md explicitly warns about sensitive text hitting swap. Parallel execution should be opt-in, not default.

5. Confidence boosting across methods is uncalibrated.
GLiNER uses model scores, spaCy hardcodes 0.85, regex hardcodes 0.6. "Boost confidence when multiple methods agree" is numerology unless scores are calibrated to the same scale first.

6. Junk filters will drop real entities.
"< 3 chars" kills NYC, FBI, EPA. "Common words" could kill Smith, Baker. FOIA text is messy — aggressive filtering hurts recall.

7. Proposal overstates the delta.
Current LLM extraction already chunks at 8K chars (not 78K). spaCy already supplements with regex. The hard part isn't concurrency — it's merge semantics.

---

Codex's Counter-Proposal (adopted)

Phase 1: Extract MENTIONS (not entities)

{type, raw_text, start_offset, end_offset, page, sentence, backend, backend_score}

Every extractor produces positional mentions with provenance. Nothing is lost.

Phase 2: Conservative Merge

• Exact typed normalization → auto-merge (keep all raw mentions as aliases)
• Fuzzy matches → create alias clusters for review, don't silently collapse
• Boost confidence only when same type + overlapping position from multiple backends
• Preserve every raw mention, page number, and context window

Phase 3: Evidence-Grounded LLM (optional)

• NOT a flat entity list. Send small evidence windows for:
  • Ambiguous alias clusters ("Is Clearvicn Al the same as Clearview AI?")
  • Candidate relationship pairs with surrounding sentences
  • Low-confidence entities that need a judgment call
• Input stays small (specific questions, not full document)
• Relationships still come from syntax/co-occurrence in Phase 1 — LLM only validates ambiguous ones

Adaptive execution:

• Default: regex + best available NER backend (single backend, safe for low RAM)
• --ensemble flag: run all available backends in parallel
• --validate flag: add LLM validation pass on top
• Never require LLM for core functionality

Measure before shipping:

• Run tests/test_extraction.py and tests/benchmark_extraction.py at every step
• Track precision, recall, wall-clock, and peak memory — not just anecdotal latency