Add wordlist analyzer tool, fix dedup missing subdirectory files and encoding corruption

.github/workflows/validate.yml

@@ -86,6 +86,31 @@ jobs:
           echo "⚠️  Large file detected: $line"
         done || echo "✓ All files within reasonable size limits"
 
+  analyze:
+    name: Wordlist Intelligence
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+
+    - name: Run intelligence analysis
+      run: |
+        echo "📊 Running wordlist intelligence analysis..."
+        python3 scripts/analyze.py --report --report-output intelligence-report.json
+
+    - name: Upload intelligence report
+      uses: actions/upload-artifact@v4
+      with:
+        name: intelligence-report
+        path: intelligence-report.json
+        retention-days: 30
+
   integrity:
     name: Integrity Verification
     runs-on: ubuntu-latest

README.md

@@ -286,8 +286,51 @@ python3 scripts/validate.py --file passwords.txt
 # Deduplicate wordlists
 python3 scripts/deduplicate.py passwords.txt
 
-# Deduplicate all
+# Deduplicate all wordlists (including forced-browsing/ subdirectories)
 python3 scripts/deduplicate.py --all
+
+# Case-insensitive deduplication (treat 'Password' and 'password' as same)
+python3 scripts/deduplicate.py --all --case-insensitive
+
+# Sort entries while deduplicating
+python3 scripts/deduplicate.py --all --sort
+```
+
+### Intelligence & Analysis Tool
+
+The `scripts/analyze.py` tool provides deep insights into wordlist composition, cross-referencing, and smart merging.
+
+**Character composition analysis** — Understand the makeup of a password list:
+```bash
+# Analyze character class breakdown (upper, lower, digit, special)
+python3 scripts/analyze.py --composition 1000000-password-seclists.txt
+
+# Compare multiple files
+python3 scripts/analyze.py --composition cain.txt 8-more-passwords.txt
+```
+
+**Cross-reference** — Find overlapping entries between wordlists:
+```bash
+# See which passwords appear in multiple sources
+python3 scripts/analyze.py --cross-reference 7-more-passwords.txt 8-more-passwords.txt
+
+# Save detailed pairwise Jaccard similarity to JSON
+python3 scripts/analyze.py --cross-reference file1.txt file2.txt --output xref.json
+```
+
+**Smart merging** — Combine wordlists with frequency tracking:
+```bash
+# Merge and deduplicate, sorted alphabetically
+python3 scripts/analyze.py --merge --output combined.txt list1.txt list2.txt
+
+# Rank by frequency (entries in more sources appear first)
+python3 scripts/analyze.py --merge --rank-by-frequency --output merged.txt *.txt
+```
+
+**Full repository intelligence report:**
+```bash
+# Comprehensive analysis of all wordlists in the repo
+python3 scripts/analyze.py --report --report-output report.json
 ```
 
 ### CI/CD Pipeline

scripts/README.md

@@ -8,11 +8,12 @@ This directory contains tools for maintaining the quality and integrity of the w
 Validates all wordlists and generates a comprehensive manifest.
 
 **Features:**
-- Encoding detection and validation
+- Encoding detection and validation (UTF-8, latin-1, cp1252)
 - Duplicate detection
 - File integrity checks (SHA256)
 - Statistics generation (line counts, lengths, etc.)
-- Manifest generation with full metadata
+- Binary content detection via raw byte analysis
+- Manifest generation with full metadata (dynamic date)
 
 **Usage:**
 ```bash
@@ -30,9 +31,13 @@ python3 validate.py --file wordlist.txt
 
 ### `deduplicate.py`
 Remove duplicate entries from wordlists while preserving order.
+Now handles wordlists in subdirectories (e.g., `forced-browsing/`).
 
 **Features:**
 - Order-preserving deduplication (keeps first occurrence)
+- Recursive discovery of all wordlists (including nested dirs)
+- Case-insensitive deduplication (`--case-insensitive`)
+- Alphabetical sorting (`--sort`)
 - Batch processing of all wordlists
 - Statistics reporting (duplicates removed, percentage)
 
@@ -44,12 +49,43 @@ python3 deduplicate.py wordlist.txt
 # Deduplicate with output to new file
 python3 deduplicate.py input.txt output.txt
 
-# Deduplicate all wordlists in place
+# Deduplicate all wordlists in place (recursive)
 python3 deduplicate.py --all
+
+# Case-insensitive dedup across entire repo
+python3 deduplicate.py --all --case-insensitive
+
+# Sort entries while deduplicating
+python3 deduplicate.py --all --sort
 ```
 
 **Note:** Use `--all` with caution as it modifies files in place.
 
+### `analyze.py`
+Wordlist Intelligence & Analysis Tool — new in v2.0.
+
+**Features:**
+- **Composition analysis:** character class breakdown (upper, lower, digit, special), entropy estimation
+- **Cross-reference:** pairwise Jaccard similarity, intersection/union, per-file uniqueness
+- **Pattern detection:** date-like patterns, keyboard walks, repeated substrings
+- **Smart merging:** combine wordlists with frequency tracking, rank by source count
+- **Full repository report:** comprehensive intelligence report with per-file breakdowns
+
+**Usage:**
+```bash
+# Character composition analysis
+python3 analyze.py --composition passwords.txt
+
+# Cross-reference two or more wordlists
+python3 analyze.py --cross-reference file1.txt file2.txt
+
+# Smart merge with frequency ranking
+python3 analyze.py --merge --rank-by-frequency --output merged.txt *.txt
+
+# Full repo intelligence report
+python3 analyze.py --report --report-output report.json
+```
+
 ## Requirements
 
 - Python 3.8+