fix(text): Filter external email banner boilerplate

[mlissner]

Summary

• Filter "CAUTION - EXTERNAL EMAIL" banners hidden behind yellow rectangles in court filings
• Matches both the full banner and its common tail fragment ("attachments or clicking on links") since the text spans multiple rectangles

Test plan

• New test test_external_email_banner_no_results verifies the banner is filtered
• All 26 tests pass
• mypy and pre-commit pass
• Merge after fix(text): Filter single-character redactions as false positives #200 lands

🤖 Generated with Claude Code

[claude]

🔴 The new email-banner alternative caution\\s*-?\\s*external\\s+e-?mail[^|]* uses an unbounded [^|]* that greedily consumes everything to the end of the string (pipes never appear in extracted PDF text), instead of stopping at the end of the banner phrase like every other alternative in this regex. If a rectangle's extracted text starts with the banner phrase but also contains trailing real content (plausible since the PR notes the banner spans multiple, irregularly-sized/overlapping rectangles), that real content is silently erased, is_ok_words wrongly returns False for it as an OK-word, and a genuine bad redaction gets filtered out as a false negative. Bounding the match (e.g. to [^|\\n]* up to a period/newline, similar to how the tail-fragment alternative is bounded) would fix it.

Extended reasoning...

The bug: In is_ok_words (xray/text_utils.py:57), the newly added alternative is:

caution\\s*-?\\s*external\\s+e-?mail[^|]*

[^|] is a negated character class matching any character except a literal pipe. Pipe characters essentially never occur in text extracted from a PDF, so [^|]* is functionally unbounded — it greedily consumes everything from the end of "external email" to the end of the string being checked (including across newlines, since a negated character class matches newlines too even without re.DOTALL).

Verified empirically by extracting the exact substitution logic from is_ok_words and running it directly:

is_ok_words('CAUTION - EXTERNAL EMAIL: SSN 123-45-6789 is confidential')  # -> False
is_ok_words('CAUTION - EXTERNAL EMAIL John Smith SSN 999-99-9999')        # -> False

In both cases the entire string — including the trailing SSN / name — is stripped to an empty string by the regex substitution, so len(text) > 0 evaluates to False.

Why this matters for the pipeline: is_ok_words is used as a filter to decide whether a redaction's underlying text is "just boilerplate" (and therefore not a real bad redaction) versus genuine content. get_intersecting_chars (xray/pdf_utils.py) builds each rectangle's extracted text by concatenating all characters that fall inside that rectangle, and filter_redactions_by_text keeps only redactions where is_ok_words is False (i.e., discards ones where it's True, meaning "this is just an OK word/phrase, not real content"). Since redaction rectangles are per-rectangle and — per this PR's own description — the banner "often spans multiple rectangles" with irregular boundaries, it's entirely plausible for a single rectangle's text to start with the banner phrase but also pick up trailing real content (an adjacent word, a name, an SSN, etc. that overlaps the same rectangle). In that case, the greedy [^|]* doesn't just remove the banner boilerplate — it silently erases the real content along with it, causing is_ok_words to return False... wait — to be precise: after substitution the string becomes empty, so len(text) > 0 is False, meaning is_ok_words reports "not an OK word" is False, i.e. the function returns False. Tracing the call site, redactions are only classified as bad (kept) when is_ok_words returns True for their leftover text after stripping known-boilerplate; when everything is stripped away (including real content) the function returns False, and the redaction is treated as OK/boilerplate and dropped from the bad-redactions result. The net effect is a false negative: a genuine bad redaction containing real sensitive text is hidden from the user because it happened to share a rectangle with (or be adjacent to) the email banner text.

Why nothing currently catches this: every other alternative in this regex — confidential, redacted, the district-of boilerplate, and even the sibling tail-fragment attachments or clicking on links\.? — matches a fixed, bounded phrase. Only the new caution-email alternative uses an open-ended negated character class as its terminator, which is a qualitatively different (and much riskier) construct. The included test (test_external_email_banner_no_results) only checks that a banner-only PDF returns no redactions, which passes regardless of whether the match is bounded or unbounded — it doesn't exercise the case where real content trails the banner phrase in the same extracted-text string, so the overly greedy match ships undetected.

Fix: bound the match the same way the tail-fragment is bounded — e.g. caution\\s*-?\\s*external\\s+e-?mail[^\\n]* (stop at a newline) or better, bound it to the end of the actual banner sentence (e.g. up to a period or a fixed max length), so it can only strip the boilerplate text itself and never runs on into unrelated trailing content.