feat: Recursive word condensation — normalize possessives, plurals, and case variants into canonical tokens

[aj1126]

Summary

Implement a recursive word condensation layer that normalizes morphological variants of the same root term into a single canonical token before they enter the analytics tiers. Currently, the pipeline treats Roswell, ROSWELL, roswell, Roswell's, Roswells, and roswell's as six separate tokens, fragmenting term frequencies, inflating TF-IDF vectors, and breaking correlation matrices.

Current State

The only normalization happening today is in analyzer.js:L50-52:

const rawWords = text
    .replace(/[^\w\s]/g, "")   // strips punctuation (but also strips apostrophes, killing possessives)
    .toLowerCase()              // case fold
    .split(/\s+/)

Problems

1. Possessive stripping is destructive — replace(/[^\w\s]/g, "") removes ' entirely, turning Roswell's into roswells (a non-word), not roswell.
2. No plural collapsing — locations and location are counted as separate tokens.
3. No stemming or lemmatization — observed, observing, observation, observer are all counted separately instead of collapsing to observ or observe.
4. Entity extraction is case-sensitive upstream — NLP #Place and #Date tags from compromise may produce "Georgia" and "GEORGIA" as separate Set entries depending on how the source text was formatted.
5. Location entity fragmentation — Sprint 2 Task 1 (Entity Unification) was marked complete, but the current implementation only does .toLowerCase() — no fuzzy matching, no normalization of formatting symbols or possessives on extracted entities.

Proposed Solution

Layer 1: Pre-Tokenization Normalization (Before Word Split)

Applied to raw text before tokenization:

• Possessive stripping — 's, 's, s' → remove before tokenization (e.g., Roswell's → Roswell)
• Contraction expansion (optional) — don't → do not, can't → cannot
• Unicode normalization — NFD/NFC normalization to collapse accented character variants

Layer 2: Post-Tokenization Condensation (After Word Split)

Applied to each token after splitting:

• Case folding — already done (.toLowerCase())
• Plural collapsing — strip trailing s, es, ies→y for simple English plurals
• Suffix stemming — lightweight Porter-style stemmer or compromise's built-in .root() to collapse inflectional variants (observed → observe, flying → fly)
• Minimum token length — enforce ≥2 chars after stemming (already partially done)

Layer 3: Entity-Level Deduplication (NLP Entities)

Applied to extracted dates[] and locations[] Sets:

• Case-insensitive dedup — "Georgia" and "GEORGIA" → "Georgia"
• Possessive/article stripping — "the Pentagon's" → "Pentagon"
• Whitespace normalization — collapse multiple spaces, trim
• Canonical form selection — when duplicates are found, prefer Title Case for locations, ISO format for dates

Implementation Options

Approach	Pros	Cons
compromise .root() method	Already a dependency, zero new packages	Limited stemming accuracy
Custom Porter stemmer	No dependencies, ~80 lines of code	English-only, no lemmatization
natural npm package	Full Porter/Lancaster stemmers + tokenizers	New dependency (MIT, FOSS compliant)
Custom regex pipeline	Zero dependencies, full control	Fragile, hard to maintain for edge cases

Recommended: Use compromise's built-in .root() for entity normalization (it's already installed) + a lightweight custom regex layer for possessive/plural stripping before tokenization. Avoids new dependencies.

Affected Analytics Tiers

Tier	Impact
Descriptive	Term frequency counts will be more accurate — no more split counts
Diagnostic	TF-IDF vectors and cosine similarity matrices will produce tighter, more meaningful clusters
Predictive	Time-series keyword forecasting will track consolidated trends instead of fragmented variants
Prescriptive	Metadata validation recommendations will be more precise

Acceptance Criteria

• Roswell, ROSWELL, roswell, Roswell's all collapse to a single canonical token
• Simple English plurals are collapsed (locations → location)
• At least basic suffix stemming is applied (observed → observe or observ)
• NLP-extracted entities (dates[], locations[]) are deduplicated case-insensitively
• Possessives ('s, 's) are stripped before tokenization, not destroyed by blanket punctuation removal
• No new paid/proprietary dependencies (FOSS mandate)
• Existing test suite passes with no regressions
• New test cases verify condensation edge cases (possessives, plurals, mixed case, Unicode)
• docs/architecture.md updated to document the normalization layer

Related Files

• src/analytics/analyzer.js — word frequency + NLP entity extraction (primary target)
• src/analytics/diagnostic.js — TF-IDF vector generation (downstream consumer)
• src/analytics/descriptive.js — term frequency aggregation
• docs/ROADMAP.md — feature tracking (relates to "Entity Unification" and "Advanced Stop-Word Culling" optional modules)