Refactor RegexTokenizer to handle empty stats and update max function usage - #17
Open
Conceptaniel wants to merge 4 commits into
Open
Refactor RegexTokenizer to handle empty stats and update max function usage#17Conceptaniel wants to merge 4 commits into
Conceptaniel wants to merge 4 commits into
Conversation
… parser, and ignore derived/exported files
… usage - Added a check to break the loop if stats is empty in RegexTokenizer. - Updated the way the pair with the highest count is determined using max with items. Update execution counts in DataCleaning and BytePairEncoding notebooks - Adjusted execution counts to maintain consistency after code changes. - Updated Python version in notebook metadata. Fix execution counts and add new cell in TransformerModel notebook - Corrected execution counts for several cells to reflect the current order. - Added a new empty code cell for future use. - Updated Python version in notebook metadata. Add read_plain_chat function to DataCleaning notebook - Introduced a new function to clean and parse plain text chat exports. - The function handles various message formats and normalizes the output into a DataFrame.
Owner
|
Hi @Conceptaniel, Thank you for contributing to this project. I will take a look at your PR this weekend. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update execution counts in DataCleaning and BytePairEncoding notebooks
Fix execution counts and add new cell in TransformerModel notebook
Add read_plain_chat function to DataCleaning notebook