Skip to content

PP-MX-FT (Brisa Araujo, Paola Alean) #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
brisa-araujo wants to merge 2 commits into
base: master
Choose a base branch
from
Open

PP-MX-FT (Brisa Araujo, Paola Alean) #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f261176
soluciones al primer cuaderno
brisa-araujo Jul 5, 2019
9054a2a
solución para P2 usando html2text para transformar archivos en .txt
brisa-araujo Jul 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 199 additions & 0 deletions your-code/.ipynb_checkpoints/Q1-checkpoint.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the cell below, create a Python function that wraps your previous solution for the Bag of Words lab.\n",
"\n",
"Requirements:\n",
"\n",
"1. Your function should accept the following parameters:\n",
" * `docs` [REQUIRED] - array of document paths.\n",
" * `stop_words` [OPTIONAL] - array of stop words. The default value is an empty array.\n",
"\n",
"1. Your function should return a Python object that contains the following:\n",
" * `bag_of_words` - array of strings of normalized unique words in the corpus.\n",
" * `term_freq` - array of the term-frequency vectors."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}\n"
]
}
],
"source": [
"# Import required libraries\n",
"\n",
"import os\n",
"import pandas as pd\n",
"\n",
"docs = ['doc1.txt', 'doc2.txt', 'doc3.txt']\n",
"\n",
"# Define function\n",
"def get_bow_from_docs(docs, stop_words=[]):\n",
" docs_open = []\n",
" corpus = []\n",
" bag_of_words = []\n",
" term_freq = []\n",
" for path in docs:\n",
" docs_open.append(open(path, \"r\"))\n",
" for archivo in docs_open:\n",
" corpus.append(archivo.read().lower().strip(\".\"))\n",
" for i in corpus:\n",
" i = i.split()\n",
" for j in i:\n",
" if j not in bag_of_words and j not in stop_words:\n",
" bag_of_words.append(j)\n",
" for frase in corpus:\n",
" tmp = []\n",
" for palabra in bag_of_words:\n",
" tmp.append(frase.split().count(palabra))\n",
" term_freq.append(tmp)\n",
" return {\n",
" \"bag_of_words\": bag_of_words,\n",
" \"term_freq\": term_freq\n",
" }"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Test your function without stop words. You should see the output like below:\n",
"\n",
"```{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}```"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}\n"
]
}
],
"source": [
"# Obtain BoW from your function\n",
"bow = get_bow_from_docs(docs)\n",
"\n",
"# Print BoW\n",
"print(bow)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If your attempt above is successful, nice work done!\n",
"\n",
"Now test your function again with the stop words. In the previous lab we defined the stop words in a large array. In this lab, we'll import the stop words from Scikit-Learn."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"frozenset({'have', 'give', 'done', 'everywhere', 'were', 'inc', 'yours', 'us', 'another', 'whoever', 'anyhow', 'un', 'but', 'within', 'herein', 'my', 'few', 'please', 'mostly', 'where', 'cry', 'besides', 'moreover', 'system', 'also', 'had', 'yet', 'onto', 'never', 'due', 'five', 'in', 'our', 'i', 'much', 'what', 'this', 'sometimes', 'since', 'other', 'found', 'a', 'hence', 'her', 'describe', 'however', 'him', 'myself', 'after', 'beyond', 'formerly', 'until', 'towards', 'not', 'here', 'own', 'name', 'along', 'it', 'keep', 'several', 'upon', 'via', 'then', 'namely', 'one', 'con', 'thereupon', 're', 'per', 'me', 'am', 'detail', 'now', 'front', 'alone', 'perhaps', 'more', 'ever', 'both', 'do', 'he', 'above', 'thereby', 'whether', 'whereas', 'and', 'the', 'together', 'none', 'full', 'thence', 'was', 'except', 'empty', 'ie', 'thus', 'eg', 'else', 'around', 'whole', 'neither', 'although', 'mine', 'such', 'become', 'your', 'yourselves', 'still', 'etc', 'will', 'indeed', 'amoungst', 'himself', 'whence', 'hereupon', 'part', 'interest', 'six', 'whose', 'co', 'these', 'somehow', 'wherein', 'why', 'most', 'them', 'may', 'before', 'de', 'twenty', 'fire', 'meanwhile', 'across', 'twelve', 'enough', 'whatever', 'there', 'beforehand', 'find', 'noone', 'anything', 'by', 'thru', 'only', 'whenever', 'behind', 'under', 'seems', 'their', 'hereafter', 'seem', 'out', 'thereafter', 'nothing', 'nobody', 'is', 'could', 'when', 'no', 'seeming', 'less', 'at', 'someone', 'amongst', 'should', 'whom', 'thick', 'she', 'whither', 'an', 'eight', 'anyone', 'sometime', 'each', 'ours', 'already', 'take', 'ltd', 'into', 'be', 'herself', 'toward', 'themselves', 'fifteen', 'without', 'least', 'how', 'some', 'any', 'fifty', 'either', 'that', 'would', 'made', 'eleven', 'seemed', 'ten', 'afterwards', 'even', 'therefore', 'ourselves', 'fill', 'many', 'hers', 'go', 'cannot', 'first', 'anywhere', 'nine', 'every', 'bill', 'nowhere', 'everything', 'move', 'rather', 'can', 'if', 'whereupon', 'something', 'while', 'because', 'two', 'again', 'further', 'whereafter', 'must', 'bottom', 'almost', 'up', 'sixty', 'throughout', 'through', 'anyway', 'who', 'to', 'everyone', 'though', 'call', 'well', 'became', 'so', 'they', 'its', 'which', 'his', 'top', 'side', 'below', 'among', 'itself', 'too', 'last', 'those', 'all', 'former', 'wherever', 'becoming', 'latter', 'amount', 'once', 'with', 'next', 'whereby', 'between', 'been', 'latterly', 'back', 'being', 'sincere', 'hundred', 'elsewhere', 'during', 'against', 'are', 'thin', 'see', 'cant', 'hereby', 'put', 'yourself', 'somewhere', 'mill', 'for', 'forty', 'same', 'nevertheless', 'has', 'off', 'about', 'therein', 'others', 'serious', 'than', 'or', 'of', 'three', 'four', 'couldnt', 'over', 'very', 'from', 'nor', 'we', 'beside', 'might', 'becomes', 'often', 'get', 'otherwise', 'always', 'as', 'down', 'hasnt', 'you', 'show', 'on', 'third'})\n"
]
}
],
"source": [
"from sklearn.feature_extraction import stop_words\n",
"print(stop_words.ENGLISH_STOP_WORDS)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You should have seen a large list of words that looks like:\n",
"\n",
"```frozenset({'across', 'mine', 'cannot', ...})```\n",
"\n",
"`frozenset` is a type of Python object that is immutable. In this lab you can use it just like an array without conversion."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, test your function with supplying `stop_words.ENGLISH_STOP_WORDS` as the second parameter."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'cool', 'love', 'student'], 'term_freq': [[1, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1]]}\n"
]
}
],
"source": [
"bow = get_bow_from_docs(docs, stop_words.ENGLISH_STOP_WORDS)\n",
"\n",
"print(bow)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You should have seen:\n",
"\n",
"```{'bag_of_words': ['ironhack', 'cool', 'love', 'student'], 'term_freq': [[1, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1]]}```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
173 changes: 173 additions & 0 deletions your-code/.ipynb_checkpoints/Q2-checkpoint.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

111 changes: 66 additions & 45 deletions your-code/Q1.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,49 +19,49 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 11,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}\n"
]
}
],
"source": [
"# Import required libraries\n",
"\n",
"# Define function\n",
"def get_bow_from_docs(docs, stop_words=[]):\n",
" \n",
" # In the function, first define the variables you will use such as `corpus`, `bag_of_words`, and `term_freq`.\n",
" \n",
" \n",
" \n",
" \"\"\"\n",
" Loop `docs` and read the content of each doc into a string in `corpus`.\n",
" Remember to convert the doc content to lowercases and remove punctuation.\n",
" \"\"\"\n",
"import os\n",
"import pandas as pd\n",
"\n",
" \n",
" \n",
" \"\"\"\n",
" Loop `corpus`. Append the terms in each doc into the `bag_of_words` array. The terms in `bag_of_words` \n",
" should be unique which means before adding each term you need to check if it's already added to the array.\n",
" In addition, check if each term is in the `stop_words` array. Only append the term to `bag_of_words`\n",
" if it is not a stop word.\n",
" \"\"\"\n",
"docs = ['doc1.txt', 'doc2.txt', 'doc3.txt']\n",
"\n",
" \n",
" \n",
" \n",
" \"\"\"\n",
" Loop `corpus` again. For each doc string, count the number of occurrences of each term in `bag_of_words`. \n",
" Create an array for each doc's term frequency and append it to `term_freq`.\n",
" \"\"\"\n",
"\n",
" \n",
" \n",
" # Now return your output as an object\n",
"# Define function\n",
"def get_bow_from_docs(docs, stop_words=[]):\n",
" docs_open = []\n",
" corpus = []\n",
" bag_of_words = []\n",
" term_freq = []\n",
" for path in docs:\n",
" docs_open.append(open(path, \"r\"))\n",
" for archivo in docs_open:\n",
" corpus.append(archivo.read().lower().strip(\".\"))\n",
" for i in corpus:\n",
" i = i.split()\n",
" for j in i:\n",
" if j not in bag_of_words and j not in stop_words:\n",
" bag_of_words.append(j)\n",
" for frase in corpus:\n",
" tmp = []\n",
" for palabra in bag_of_words:\n",
" tmp.append(frase.split().count(palabra))\n",
" term_freq.append(tmp)\n",
" return {\n",
" \"bag_of_words\": bag_of_words,\n",
" \"term_freq\": term_freq\n",
" }\n",
" "
" }"
]
},
{
Expand All @@ -75,13 +75,18 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 12,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}\n"
]
}
],
"source": [
"# Define doc paths array\n",
"docs = []\n",
"\n",
"# Obtain BoW from your function\n",
"bow = get_bow_from_docs(docs)\n",
"\n",
Expand All @@ -100,9 +105,17 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 13,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"frozenset({'have', 'give', 'done', 'everywhere', 'were', 'inc', 'yours', 'us', 'another', 'whoever', 'anyhow', 'un', 'but', 'within', 'herein', 'my', 'few', 'please', 'mostly', 'where', 'cry', 'besides', 'moreover', 'system', 'also', 'had', 'yet', 'onto', 'never', 'due', 'five', 'in', 'our', 'i', 'much', 'what', 'this', 'sometimes', 'since', 'other', 'found', 'a', 'hence', 'her', 'describe', 'however', 'him', 'myself', 'after', 'beyond', 'formerly', 'until', 'towards', 'not', 'here', 'own', 'name', 'along', 'it', 'keep', 'several', 'upon', 'via', 'then', 'namely', 'one', 'con', 'thereupon', 're', 'per', 'me', 'am', 'detail', 'now', 'front', 'alone', 'perhaps', 'more', 'ever', 'both', 'do', 'he', 'above', 'thereby', 'whether', 'whereas', 'and', 'the', 'together', 'none', 'full', 'thence', 'was', 'except', 'empty', 'ie', 'thus', 'eg', 'else', 'around', 'whole', 'neither', 'although', 'mine', 'such', 'become', 'your', 'yourselves', 'still', 'etc', 'will', 'indeed', 'amoungst', 'himself', 'whence', 'hereupon', 'part', 'interest', 'six', 'whose', 'co', 'these', 'somehow', 'wherein', 'why', 'most', 'them', 'may', 'before', 'de', 'twenty', 'fire', 'meanwhile', 'across', 'twelve', 'enough', 'whatever', 'there', 'beforehand', 'find', 'noone', 'anything', 'by', 'thru', 'only', 'whenever', 'behind', 'under', 'seems', 'their', 'hereafter', 'seem', 'out', 'thereafter', 'nothing', 'nobody', 'is', 'could', 'when', 'no', 'seeming', 'less', 'at', 'someone', 'amongst', 'should', 'whom', 'thick', 'she', 'whither', 'an', 'eight', 'anyone', 'sometime', 'each', 'ours', 'already', 'take', 'ltd', 'into', 'be', 'herself', 'toward', 'themselves', 'fifteen', 'without', 'least', 'how', 'some', 'any', 'fifty', 'either', 'that', 'would', 'made', 'eleven', 'seemed', 'ten', 'afterwards', 'even', 'therefore', 'ourselves', 'fill', 'many', 'hers', 'go', 'cannot', 'first', 'anywhere', 'nine', 'every', 'bill', 'nowhere', 'everything', 'move', 'rather', 'can', 'if', 'whereupon', 'something', 'while', 'because', 'two', 'again', 'further', 'whereafter', 'must', 'bottom', 'almost', 'up', 'sixty', 'throughout', 'through', 'anyway', 'who', 'to', 'everyone', 'though', 'call', 'well', 'became', 'so', 'they', 'its', 'which', 'his', 'top', 'side', 'below', 'among', 'itself', 'too', 'last', 'those', 'all', 'former', 'wherever', 'becoming', 'latter', 'amount', 'once', 'with', 'next', 'whereby', 'between', 'been', 'latterly', 'back', 'being', 'sincere', 'hundred', 'elsewhere', 'during', 'against', 'are', 'thin', 'see', 'cant', 'hereby', 'put', 'yourself', 'somewhere', 'mill', 'for', 'forty', 'same', 'nevertheless', 'has', 'off', 'about', 'therein', 'others', 'serious', 'than', 'or', 'of', 'three', 'four', 'couldnt', 'over', 'very', 'from', 'nor', 'we', 'beside', 'might', 'becomes', 'often', 'get', 'otherwise', 'always', 'as', 'down', 'hasnt', 'you', 'show', 'on', 'third'})\n"
]
}
],
"source": [
"from sklearn.feature_extraction import stop_words\n",
"print(stop_words.ENGLISH_STOP_WORDS)"
Expand All @@ -128,11 +141,19 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 15,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'bag_of_words': ['ironhack', 'cool', 'love', 'student'], 'term_freq': [[1, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1]]}\n"
]
}
],
"source": [
"bow = get_bow_from_docs(bow, stop_words.ENGLISH_STOP_WORDS)\n",
"bow = get_bow_from_docs(docs, stop_words.ENGLISH_STOP_WORDS)\n",
"\n",
"print(bow)"
]
Expand Down Expand Up @@ -170,7 +191,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
"version": "3.7.3"
}
},
"nbformat": 4,
Expand Down
Loading