fix(analyzer): stop quadratic backtracking in in_pan low pattern#2068
fix(analyzer): stop quadratic backtracking in in_pan low pattern#2068uwezkhan wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR updates the India PAN low-confidence regex to avoid expensive scanning/backtracking behavior and adds regression tests to validate both correctness and runtime characteristics.
Changes:
- Refined the low-confidence PAN regex lookaheads to scope matching to the token character class.
- Added a regression test for low-confidence detection with embedded digits.
- Added a performance/regression test intended to catch quadratic-time behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/india/in_pan_recognizer.py | Adjusts the low-confidence PAN regex lookaheads to avoid scanning beyond the candidate token. |
| presidio-analyzer/tests/test_in_pan_recognizer.py | Adds functional + runtime regression tests for the updated low-confidence PAN regex. |
| @@ -1,3 +1,5 @@ | |||
| import time | |||
| assert_result(results[0], entities[0], 0, 10, 0.01) | ||
|
|
||
|
|
||
| def test_low_confidence_pattern_does_not_backtrack(recognizer, entities): |
| text = "a " * 50000 | ||
| start = time.time() | ||
| results = recognizer.analyze(text, entities) | ||
| elapsed = time.time() - start | ||
| assert results == [] | ||
| assert elapsed < 10 |
There was a problem hiding this comment.
Good call. Switched the guard to perf_counter (monotonic, higher resolution) and pushed. The threshold stays at 10s on purpose: the patched pattern runs this input in single-digit milliseconds, so it is a coarse did-not-blow-up guard rather than a tight perf assertion, which keeps it stable across CI machines.
| Pattern( | ||
| "PAN (Low)", | ||
| r"\b((?=.*?[a-zA-Z])(?=.*?[0-9]{4})[\w@#$%^?~-]{10})\b", | ||
| r"\b((?=[\w@#$%^?~-]*?[a-zA-Z])(?=[\w@#$%^?~-]*?[0-9]{4})[\w@#$%^?~-]{10})\b", |
There was a problem hiding this comment.
I looked at bounding both scans to {0,9}, but kept the token-class ? form. The quadratic cost came from . walking across whitespace, so each lookahead rescanned the rest of the text at every word boundary. Scoping them to [\w@#$%^?~-] makes each scan stop at the first non-token character, so the work is already linear in input length (100 KB dropped from ~32 s to ~9 ms).
Before: {0,9}? would cap the scan at 9 characters. After: the current form also caps it, because the trailing \b forces the token to end at a boundary. The difference is that the class contains non-word symbols (@#$%^?~-), so an explicit {0,9} would change which spans match for tokens built from those symbols, while the *? form preserves the existing matches exactly. Given the scan is already bounded by the token class, I'd rather not shift match semantics.
time.time() is wall-clock and non-monotonic, which can make the timing guard flaky. Switch to perf_counter for a monotonic, higher-resolution measurement. Signed-off-by: Uwez Khan <uwezkhan053@gmail.com>
Change Description
in_pan_recognizer.py, PAN (Low) pattern, matched against a boundary-rich string with no four-digit run:Both lookaheads use
.*?, so each one rescans the rest of the text at every word boundary and match time is quadratic in input length. IN_PAN is enabled by default for English, so this sits on the analyze path for any analyzed text.Before:
(?=.*?[a-zA-Z])(?=.*?[0-9]{4})[\w@#$%^?~-]{10}..walks across non-token characters, so the lookahead scan is unbounded.After:
(?=[\w@#$%^?~-]*?[a-zA-Z])(?=[\w@#$%^?~-]*?[0-9]{4})[\w@#$%^?~-]{10}. The lookaheads only traverse the PAN token's own character class, so each stops at the first non-token character. Matching is linear (100 KB now ~9 ms) and every valid PAN whose four digits sit inside the token still matches at the same span and score.Tradeoff: scoping the lookaheads also drops one previous match. A ten-letter token is no longer reported as a PAN when four digits happen to appear later in the text; a PAN's digits live inside the number, so that case was a false positive.
Issue reference
N/A
Checklist