Add wordlist analyzer tool, fix dedup missing subdirectory files and encoding corruption#71
Open
Mayne-X wants to merge 1 commit into
Open
Add wordlist analyzer tool, fix dedup missing subdirectory files and encoding corruption#71Mayne-X wants to merge 1 commit into
Mayne-X wants to merge 1 commit into
Conversation
- New scripts/analyze.py: composition analysis, cross-referencing, smart merging, and full repo intelligence reports - Fix deduplicate.py: --all mode now finds wordlists in subdirectories (forced-browsing/ was being skipped entirely) - Fix deduplicate.py: encoding corruption bug - original code used errors='ignore' which silently dropped non-UTF8 characters from latin-1 encoded files (7-more-passwords.txt, 38650-username-*) - Fix deduplicate.py: added --case-insensitive and --sort flags - Fix validate.py: dynamic date, multi-encoding support (utf-8, latin-1, cp1252), better binary detection via raw byte analysis - Fix validate.py: --file mode no longer crashes on bare filenames - CI: new analyze job generates intelligence report on every push - Docs: README and scripts/README updated with new tool usage
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
📝 WalkthroughWalkthroughRewrites ChangesWordlist Tooling Enhancements
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/analyze.py`:
- Around line 265-268: The non-ranked path in the else block constructs
write_lines by doing sorted(merged + ['\n']), which adds only a single newline
to the entire list instead of separating each entry with a newline. This causes
entries to be written without proper separators, producing a corrupted merged
file. Fix this by ensuring each entry in the merged list has its own trailing
newline character before sorting, so that each entry is properly separated when
written to the file.
- Around line 41-50: The load_entries() function uses errors='ignore' on the
UTF-8 encoding attempt, which silently drops undecodable bytes instead of
raising an exception that would trigger the fallback to latin-1 decoding. Change
the errors parameter from 'ignore' to 'strict' (or remove it entirely since
strict is the default) in the first open() call so that invalid UTF-8 sequences
cause an exception and allow the except block to execute the latin-1 fallback,
ensuring consistent behavior with the validator/deduplicate modules.
- Around line 424-440: The cross_reference() function returns an error dict when
fewer than two files are provided, but the current code path does not check for
this error condition before printing results. Add a validation check immediately
after calling cross_reference(filepaths) to detect if the returned dictionary
contains an error key, and if so, print the error message using the
processLogger or print statement and exit with a non-zero status code (such as
1) before attempting to access and display the cross-reference results. Only
proceed with printing the success report and exiting with status 0 if the
validation confirms no error exists.
In `@scripts/deduplicate.py`:
- Around line 117-124: The find_wordlists() function docstring claims to skip
the scripts directory, but the skip_dirs set does not include 'scripts'. Add
'scripts' to the skip_dirs set (currently contains '.git', 'node_modules',
'__pycache__') to align the actual behavior with the documented behavior and
match the file discovery logic used in scripts/validate.py.
- Around line 99-101: The print statement with the f-string that computes the
reduction percentage will raise a ZeroDivisionError when original_nonempty is 0
(i.e., files with no non-empty lines). Guard against this by checking if
original_nonempty is greater than 0 before performing the division in the
`removed / original_nonempty * 100` calculation. If original_nonempty is 0,
provide an alternative output (such as showing 0% reduction or skipping the
percentage display entirely) to prevent the crash during --all runs.
In `@scripts/validate.py`:
- Line 36: The docstring describing the function behavior claims to use
streaming reads for large files to minimize memory, but the actual
implementation at line 54 loads the entire file into memory using f.read(). Fix
this by either updating the docstring to accurately reflect that the function
reads entire files into memory, or implement true streaming by reading the file
in chunks and updating the hash incrementally using .update() calls instead of
loading everything at once with f.read().
- Line 188: Remove the `f` prefix from the string in the print statement at the
validation summary line since the string contains no placeholder expressions
that need interpolation. Change the f-string to a regular string literal while
keeping the newline character and text content intact.
- Line 140: The inline use of __import__('datetime') at line 140 where the today
variable is assigned is non-idiomatic Python and violates PEP 8 standards. Add a
standard import statement for datetime at the top of the file (after the
existing imports around line 20), then replace the
__import__('datetime').date.today().isoformat() call with
datetime.date.today().isoformat() to use the properly imported module instead of
the dynamic import.
- Around line 57-76: The encoding detection loop in the file has a critical
flaw: latin-1 is a single-byte encoding that always succeeds for any byte
sequence without raising UnicodeDecodeError, making the cp1252 fallback
unreachable dead code. Remove cp1252 from the encoding tuple in the for loop
since it will never be reached, or replace the entire encoding detection logic
with a proper library like chardet to genuinely detect the file's actual
encoding when UTF-8 fails. The current implementation will misidentify actual
cp1252 files as latin-1, causing character corruption in the 0x80-0x9F byte
range.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 920fdbc8-44d0-4a69-bf2e-e94f0dd490e8
📒 Files selected for processing (6)
.github/workflows/validate.ymlREADME.mdscripts/README.mdscripts/analyze.pyscripts/deduplicate.pyscripts/validate.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes two real bugs and adds a new analysis tool:
Bug 1: deduplicate.py --all was skipping half the repo The --all mode only looked for wordlist files in the root folder. All 50+ files inside forced-browsing/ and its sub-folders were never seen — so users running dedup got a false sense of completeness. Fixed by using recursive rglob().
Bug 2: Dedup was silently eating characters from some files Two wordlists (7-more-passwords.txt and 38650-username-sktorrent.txt) are stored in latin-1 encoding, not UTF-8. The old code used errors='ignore' which just dropped any byte that wasn't valid UTF-8 — characters like ø, è, č were deleted without any warning. Now the tool auto-detects the encoding (tries UTF-8 first, then latin-1, then cp1252) so nothing gets lost.
Also improved:
What's NOT in this PR: No wordlist data files were touched. Just the tooling. A maintainer can run deduplicate.py --all separately to clean up the ~81K duplicate entries that exist in the data.
Summary by CodeRabbit
New Features
Improvements
Documentation