Entity extraction: filter junk entities from regex fallback

[JordanCoin]

Problem

When using the regex fallback (no GLiNER/spaCy/LLM), extraction produces junk entities from OCR artifacts and document boilerplate. These pollute the graph and waste crossref API calls.

Examples from a 32-page Clearview AI contract:

• To: — picked up as organization
• Bill To: — picked up as organization
• APPROVED — picked up as organization
• EC .Orchestrating a or g-?te' 22 — OCR garbage picked up as organization
• CID SES Criminal lnte — truncated OCR artifact
• Justification of request: Clcorwcw will assist the department with identifying suspects through facial recognition. — entire sentence picked up as person
• Sgt. Lakca Gaither, 09/94/2019, For FMU Use Only — name + metadata jammed together
• Clearvicn Al. Inc. — OCR misspelling of "Clearview AI"

Fixes needed

1. Blocklist — common boilerplate words that aren't entities: "To:", "From:", "Bill To:", "APPROVED", "RE:", "CC:", etc.
2. Length filter — skip entities shorter than 3 chars or longer than 80 chars (full sentences aren't entities)
3. Character ratio — if >30% special characters or digits, probably OCR garbage
4. Fuzzy dedup — "Clearview AI" and "Clearvicn Al" should merge (Levenshtein distance)
5. Sentence detection — if the "entity" contains a verb, it's probably a sentence not a name

Context

GLiNER and spaCy handle these correctly — this only affects the regex fallback tier. But regex is the safety net that always works, so it needs to be cleaner.

[JordanCoin]

Codex Implementation Spec (406K tokens of analysis)

Hard Drop Rules (score = 0.0)

• Exact matches in blocklist: "to", "from", "approved", "bill to", "invoice", "qty", "subtotal", etc.
• NER types (person/org/location) that are digits only: "880", "32", "35"
• NER types with ≤2 letters unless all-caps known acronym (FBI, CIA, EPA, NYC, etc.)
• Tokens ending with ":"

Soft Penalty (reduce confidence, don't drop)

• Generic roles: "Seller", "Buyer", "Contractor" → -0.45
• Concept phrases: "human trafficking", "narcotics" → -0.60
• Bare generic org nouns: "Office", "Department", "Board" → -0.45
• No uppercase in NER-type entity → -0.35
• Single word ≤3 chars (non-acronym) → -0.45

Confidence Recalibration

• spaCy: 0.55 (was 0.85 — too generous)
• GLiNER: 0.15 + 0.85 * raw_score (calibrated)
• Regex: 0.92 for email/phone/doc_id, 0.78 for other structured
• LLM: raw score * 0.9

OCR-Aware Fuzzy Merge

• OCR fold: & → 8, 0 → o, 1/l → i, 5 → s then strip non-alphanumeric
• Person merge: same last name + first initial match
• Org merge: token Jaccard ≥ 0.80 on core tokens (strip Inc/LLC/Corp)
• Location merge: Jaccard ≥ 0.80 or SequenceMatcher ≥ 0.95
• Keep all aliases in metadata, canonicalize to cleanest variant

Protected Entities (never drop)

FBI, CIA, DOJ, DHS, DOD, EPA, IRS, DEA, ICE, CBP, TSA, SEC, FTC, FCC, OIG, NYC, DC

Final Keep Threshold

• 0.30 for PERSON/ORG/LOCATION
• 0.25 for structured types (DATE/MONEY/EMAIL/PHONE)