Skip to content

PP-MX-FT Emilio y Jorge #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
evillalobos1123 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

PP-MX-FT Emilio y Jorge #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 254 additions & 0 deletions your-code/.ipynb_checkpoints/Q1-checkpoint.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the cell below, create a Python function that wraps your previous solution for the Bag of Words lab.\n",
"\n",
"Requirements:\n",
"\n",
"1. Your function should accept the following parameters:\n",
" * `docs` [REQUIRED] - array of document paths.\n",
" * `stop_words` [OPTIONAL] - array of stop words. The default value is an empty array.\n",
"\n",
"1. Your function should return a Python object that contains the following:\n",
" * `bag_of_words` - array of strings of normalized unique words in the corpus.\n",
" * `term_freq` - array of the term-frequency vectors."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"# Import required libraries\n",
"import re\n",
"import pandas as pd\n",
"import math\n",
"import os\n",
"# Define function\n",
"docs = ['doc1.txt', 'doc2.txt', 'doc3.txt']\n",
"def get_bow_from_docs(docs, stop_words=[]):\n",
" corpus = []\n",
" bag_of_words = []\n",
" term_freq = []\n",
" corpus2 = []\n",
" # In the function, first define the variables you will use such as `corpus`, `bag_of_words`, and `term_freq`.\n",
" for x in docs:\n",
" abrir = open(x).read()\n",
" corpus.append(abrir)\n",
" for x in corpus:\n",
" y = x.lower().replace(\".\", \"\")\n",
" corpus2.append(y)\n",
" \n",
" \"\"\"\n",
" Loop `docs` and read the content of each doc into a string in `corpus`.\n",
" Remember to convert the doc content to lowercases and remove punctuation.\n",
" \"\"\"\n",
" for x in corpus2:\n",
" y = re.split(\" \", x)\n",
" for z in y:\n",
" if z not in bag_of_words and z not in stop_words:\n",
" bag_of_words.append(z)\n",
" else:\n",
" pass\n",
" \n",
" \n",
" \n",
" pieces = []\n",
" for x in corpus2:\n",
" pieces.append(x.split(\" \"))\n",
" \n",
" \n",
" \n",
" \n",
" \"\"\"\n",
" Loop `corpus`. Append the terms in each doc into the `bag_of_words` array. The terms in `bag_of_words` \n",
" should be unique which means before adding each term you need to check if it's already added to the array.\n",
" In addition, check if each term is in the `stop_words` array. Only append the term to `bag_of_words`\n",
" if it is not a stop word.\n",
" \"\"\"\n",
" \n",
" \n",
" \n",
" \n",
" for s in pieces:\n",
" temp = []\n",
" for b in bag_of_words:\n",
" y = s.count(b)\n",
" temp.append(y)\n",
" term_freq.append(temp)\n",
"\n",
" \n",
" \n",
" \n",
" \"\"\"\n",
" Loop `corpus` again. For each doc string, count the number of occurrences of each term in `bag_of_words`. \n",
" Create an array for each doc's term frequency and append it to `term_freq`.\n",
" \"\"\"\n",
"\n",
" \n",
" \n",
" # Now return your output as an object\n",
" return {\n",
" \"bag_of_words\": bag_of_words,\n",
" \"term_freq\": term_freq\n",
" }\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}\n"
]
}
],
"source": [
"print(get_bow_from_docs(docs, stop_words=[]))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Test your function without stop words. You should see the output like below:\n",
"\n",
"```{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}```"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}\n"
]
}
],
"source": [
"# Define doc paths array\n",
"docs = ['doc1.txt', 'doc2.txt', 'doc3.txt']\n",
"\n",
"# Obtain BoW from your function\n",
"bow = get_bow_from_docs(docs)\n",
"\n",
"# Print BoW\n",
"print(bow)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If your attempt above is successful, nice work done!\n",
"\n",
"Now test your function again with the stop words. In the previous lab we defined the stop words in a large array. In this lab, we'll import the stop words from Scikit-Learn."
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"frozenset({'for', 'up', 'please', 'were', 'so', 'besides', 'beyond', 'against', 'these', 'any', 'has', 'ours', 'although', 'that', 'why', 'then', 'none', 'detail', 'latter', 'herein', 'twelve', 'find', 'myself', 'further', 'which', 'though', 'should', 'he', 'thence', 'yourself', 'side', 'itself', 'are', 'very', 'ourselves', 'your', 'empty', 'hence', 'put', 'across', 'elsewhere', 'this', 'un', 'former', 'out', 'found', 'moreover', 'still', 'rather', 'most', 'afterwards', 'have', 'at', 'now', 'either', 'themselves', 'therein', 'until', 'from', 'both', 'less', 'four', 'i', 'whereas', 'keep', 'even', 'becoming', 're', 'nine', 'been', 'ie', 'around', 'everywhere', 'anyhow', 'con', 'sometimes', 'take', 'where', 'can', 'front', 'several', 'hereafter', 'others', 'over', 'else', 'such', 'before', 'during', 'because', 'they', 'do', 'latterly', 'hers', 'almost', 'under', 'part', 'nobody', 'whither', 'bottom', 'about', 'move', 'done', 'became', 'sincere', 'whereby', 'anything', 'name', 'serious', 'seemed', 'much', 'her', 'through', 'below', 'whatever', 'and', 'being', 'among', 'them', 'therefore', 'show', 'our', 'sometime', 'herself', 'nor', 'after', 'there', 'top', 'least', 'nowhere', 'might', 'whose', 'must', 'every', 'to', 'no', 'also', 'somewhere', 'two', 'enough', 'him', 'on', 'seeming', 'us', 'we', 'without', 'per', 'thereby', 'eleven', 'when', 'ltd', 'those', 'would', 'mostly', 'mine', 'thru', 'amongst', 'perhaps', 'hereupon', 'within', 'never', 'forty', 'along', 'hundred', 'cant', 'becomes', 'due', 'more', 'anyone', 'inc', 'five', 'same', 'down', 'back', 'formerly', 'yours', 'describe', 'etc', 'full', 'neither', 'each', 'could', 'last', 'or', 'whereupon', 'behind', 'it', 'in', 'few', 'noone', 'via', 'thus', 'thereafter', 'alone', 'thick', 'towards', 'somehow', 'twenty', 'anyway', 'three', 'beforehand', 'me', 'call', 'ever', 'amount', 'will', 'whole', 'onto', 'namely', 'she', 'everyone', 'yourselves', 'wherein', 'whether', 'with', 'made', 'cry', 'whenever', 'something', 'his', 'mill', 'first', 'give', 'get', 'become', 'interest', 'than', 'eight', 'had', 'by', 'bill', 'am', 'as', 'go', 'but', 'hereby', 'all', 'nothing', 'thin', 'here', 'beside', 'meanwhile', 'ten', 'was', 'an', 'always', 'who', 'only', 'third', 'fifty', 'toward', 'the', 'see', 'eg', 'since', 'of', 'many', 'some', 'their', 'amoungst', 'system', 'couldnt', 'whence', 'everything', 'one', 'what', 'fire', 'next', 'whom', 'my', 'off', 'six', 'between', 'upon', 'wherever', 'cannot', 'throughout', 'once', 'whoever', 'whereafter', 'indeed', 'seem', 'seems', 'fill', 'himself', 'too', 'its', 'may', 'fifteen', 'you', 'nevertheless', 'except', 'while', 'de', 'sixty', 'thereupon', 'again', 'a', 'co', 'be', 'above', 'often', 'already', 'someone', 'yet', 'together', 'hasnt', 'how', 'anywhere', 'another', 'well', 'if', 'not', 'own', 'otherwise', 'into', 'is', 'however', 'other'})\n"
]
}
],
"source": [
"from sklearn.feature_extraction import stop_words\n",
"\n",
"print(stop_words.ENGLISH_STOP_WORDS)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You should have seen a large list of words that looks like:\n",
"\n",
"```frozenset({'across', 'mine', 'cannot', ...})```\n",
"\n",
"`frozenset` is a type of Python object that is immutable. In this lab you can use it just like an array without conversion."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, test your function with supplying `stop_words.ENGLISH_STOP_WORDS` as the second parameter."
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'cool', 'love', 'student'], 'term_freq': [[1, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1]]}\n"
]
}
],
"source": [
"bow = get_bow_from_docs(docs, stop_words.ENGLISH_STOP_WORDS)\n",
"\n",
"print(bow)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You should have seen:\n",
"\n",
"```{'bag_of_words': ['ironhack', 'cool', 'love', 'student'], 'term_freq': [[1, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1]]}```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading