From 718ff301045bd2cb6b2971604f83f768af7b5fd1 Mon Sep 17 00:00:00 2001
From: TGLSpain
Date: Wed, 20 Jul 2022 21:02:47 -0300
Subject: [PATCH 1/2] first commit
---
.../Learning-checkpoint.ipynb | 310 +++++
.../.ipynb_checkpoints/Q1-checkpoint.ipynb | 240 ++++
.../.ipynb_checkpoints/Q2-checkpoint.ipynb | 161 +++
.../.ipynb_checkpoints/Q3-checkpoint.ipynb | 285 +++++
.../.ipynb_checkpoints/main-checkpoint.ipynb | 1049 +++++++++++++++++
.../your-code/Learning.ipynb | 2 +-
lab-functional-programming/your-code/Q1.ipynb | 114 +-
lab-functional-programming/your-code/Q2.ipynb | 60 +-
lab-functional-programming/your-code/Q3.ipynb | 116 +-
.../your-code/main.ipynb | 645 +++++++++-
10 files changed, 2897 insertions(+), 85 deletions(-)
create mode 100644 lab-functional-programming/your-code/.ipynb_checkpoints/Learning-checkpoint.ipynb
create mode 100644 lab-functional-programming/your-code/.ipynb_checkpoints/Q1-checkpoint.ipynb
create mode 100644 lab-functional-programming/your-code/.ipynb_checkpoints/Q2-checkpoint.ipynb
create mode 100644 lab-functional-programming/your-code/.ipynb_checkpoints/Q3-checkpoint.ipynb
create mode 100644 lab-functional-programming/your-code/.ipynb_checkpoints/main-checkpoint.ipynb
diff --git a/lab-functional-programming/your-code/.ipynb_checkpoints/Learning-checkpoint.ipynb b/lab-functional-programming/your-code/.ipynb_checkpoints/Learning-checkpoint.ipynb
new file mode 100644
index 0000000..e838c99
--- /dev/null
+++ b/lab-functional-programming/your-code/.ipynb_checkpoints/Learning-checkpoint.ipynb
@@ -0,0 +1,310 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Functional Programming\n",
+ "\n",
+ "\n",
+ "Lesson Goals\n",
+ "\n",
+ " Learn the difference between procedural and functional programming style.\n",
+ " Learn about the components of functions and how to write them in Python.\n",
+ " Learn the difference between global and local variables and how they apply to functions.\n",
+ " Learn how to apply functions to Pandas data frames.\n",
+ "\n",
+ "\n",
+ "Introduction\n",
+ "\n",
+ "Up until this point, we have been writing Python code in a procedural manner and using functions and methods that are either in Python's standard library or part of some other library such as Numpy or Pandas. This has worked fine for us thus far, but in order to truly graduate from beginner to intermediate Python programmer, you must have a basic understanding of functional programming and be able to write your own functions.\n",
+ "\n",
+ "In this lesson, we will cover how to write Python code in a functional style and how to construct and call user-defined functions.\n",
+ "Procedural vs. Functional Programming\n",
+ "\n",
+ "Procedural code is code that is written line by line and executed in the order it is written. It is the style in which most people learn to program due to its straightforward sequence of actions to solve a problem - do this, then do this, then do this.\n",
+ "\n",
+ "More mature programmers frame problems a little differently. They take more of a functional approach to problem solving, where they create functions consisting of multiple lines of code that accomplish some task and often return a result that can be used by another function.\n",
+ "\n",
+ "You can think of a function as a tool - something that helps you perform a job. In a sense, functional programming allows you to create your own tools. Some of these tools will be designed specifically for the current set of tasks you need to perform while other functions you write will be more general and able to be used for other projects. Imagine if a chef were able to construct their own set of knives or a carpenter were able to construct their own set of saws that were perfect for the jobs they needed to perform. That is the power that functional programming gives you as a programmer. Over time, you will develop a personal library of functions that allow you to get your work done more efficiently.\n",
+ "\n",
+ "\n",
+ "Writing Functions in Python\n",
+ "\n",
+ "Functions in Python are defined by the def keyword followed by the function name and one or more inputs the function will need enclosed in parentheses. Within the function, you write code that performs the task you want the function to execute followed by a return statement if there is anything you would like the function to return. Below is an example of a list2string function that converts lists to strings. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'John was a man of many talents'"
+ ]
+ },
+ "execution_count": 1,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "def list2string(lst):\n",
+ " string = ' '.join(lst)\n",
+ " return string\n",
+ "\n",
+ "to_string = ['John', 'was', 'a', 'man', 'of', 'many', 'talents']\n",
+ "list2string(to_string)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In this example, we defined our list2string function and told it to expect one input. Within the function, we used the join method to convert the list to a string consisting of the list elements joined together by spaces. We then had the function return the string of space-separated elements.\n",
+ "\n",
+ "You can include all the concepts we have covered in past lessons (loops, conditional logic, list comprehensions, etc.) inside a function.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Global vs. Local Variables\n",
+ "\n",
+ "When writing functional code, it is important to keep track of which variables are global and which variables are local. Global variables are variables that maintain their value outside of functions. Local variables maintain their value within the function where they are defined and then cease to exist after we exit that function. In the example below, the variables a and c are global variables and variable b is a local variable that does not exist outside of the multiply function. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "9 18\n"
+ ]
+ }
+ ],
+ "source": [
+ "a = 9\n",
+ "\n",
+ "def multiply(number, multiplier=2):\n",
+ " b = number * multiplier\n",
+ " return b\n",
+ "\n",
+ "c = multiply(a)\n",
+ "\n",
+ "print(a, c)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The multiply function above accepts a number and a multiplier (that defaults to 2), creates a local variable b that has a value of the number times the multiplier, and the function returns that value. We can see from the results of the example that global variable a (9) did get multiplied, returned, and assigned to another global variable c, so we know that b did exist within the function and was returned. However, now that the function has completed its job, we receive an error when we try to print b because it does not exist in the global variable namespace. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "ename": "NameError",
+ "evalue": "name 'b' is not defined",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;31mNameError\u001b[0m: name 'b' is not defined"
+ ]
+ }
+ ],
+ "source": [
+ "print(b)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can enter any number into the function above and also override the default multiplier and our function will return the product of those two numbers."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "500"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiply(100, multiplier=5)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Applying Functions to Data Frames\n",
+ "\n",
+ "\n",
+ "Now that we have a basic understanding of how functions and variables work, let's look at how we can apply a function that we have created to a Pandas data frame. For this example, we will import Pandas and the vehicles data set we have been working with in this module. We will then write a get_means function that identifies all the numeric columns in the data set and returns a data frame containing each column name and its mean. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/home/iudh/.local/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3044: DtypeWarning: Columns (70,71,72,73) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+ " interactivity=interactivity, compiler=compiler, result=result)\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Column
\n",
+ "
Mean
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
0
\n",
+ "
barrels08
\n",
+ "
17.779002
\n",
+ "
\n",
+ "
\n",
+ "
1
\n",
+ "
barrelsA08
\n",
+ "
0.202128
\n",
+ "
\n",
+ "
\n",
+ "
2
\n",
+ "
charge120
\n",
+ "
0.000000
\n",
+ "
\n",
+ "
\n",
+ "
3
\n",
+ "
charge240
\n",
+ "
0.009125
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
city08
\n",
+ "
17.574601
\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Column Mean\n",
+ "0 barrels08 17.779002\n",
+ "1 barrelsA08 0.202128\n",
+ "2 charge120 0.000000\n",
+ "3 charge240 0.009125\n",
+ "4 city08 17.574601"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "data = pd.read_csv('vehicles.csv')\n",
+ "\n",
+ "def get_means(df):\n",
+ " numeric = df._get_numeric_data()\n",
+ " means = pd.DataFrame(numeric.mean()).reset_index()\n",
+ " means.columns = ['Column', 'Mean']\n",
+ " return means\n",
+ "\n",
+ "mean_df = get_means(data)\n",
+ "mean_df.head()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that we performed several steps with a single function call:\n",
+ "\n",
+ " Getting the numeric columns\n",
+ " Calculating the means\n",
+ " Resetting the indexes\n",
+ " Setting the data frame's column names\n",
+ "\n",
+ "Also note that the only input necessary for this function was a data frame. The function took care of everything else. That means that it should work for any data frame that you pass it, as long as the data frame has numeric columns. We have just created a tool by writing code once that we can use again and again.\n",
+ "\n",
+ "Challenge: Import another data set that contains numeric variables and run the get_means function on it. It should return the means for the numeric columns in that data set. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lab-functional-programming/your-code/.ipynb_checkpoints/Q1-checkpoint.ipynb b/lab-functional-programming/your-code/.ipynb_checkpoints/Q1-checkpoint.ipynb
new file mode 100644
index 0000000..ad517d5
--- /dev/null
+++ b/lab-functional-programming/your-code/.ipynb_checkpoints/Q1-checkpoint.ipynb
@@ -0,0 +1,240 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In the cell below, create a Python function that wraps your previous solution for the Bag of Words lab.\n",
+ "\n",
+ "Requirements:\n",
+ "\n",
+ "1. Your function should accept the following parameters:\n",
+ " * `docs` [REQUIRED] - array of document paths.\n",
+ " * `stop_words` [OPTIONAL] - array of stop words. The default value is an empty array.\n",
+ "\n",
+ "1. Your function should return a Python object that contains the following:\n",
+ " * `bag_of_words` - array of strings of normalized unique words in the corpus.\n",
+ " * `term_freq` - array of the term-frequency vectors."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'bag_of_words': ['i',\n",
+ " 'student',\n",
+ " 'at',\n",
+ " 'ironhack',\n",
+ " 'am',\n",
+ " 'cool',\n",
+ " 'is',\n",
+ " 'a',\n",
+ " 'love'],\n",
+ " 'term_freq': [[0, 0, 0, 1, 0, 1, 1, 0, 0],\n",
+ " [1, 0, 0, 1, 0, 0, 0, 0, 1],\n",
+ " [1, 1, 1, 1, 1, 0, 0, 1, 0]]}"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Import required libraries\n",
+ "import re\n",
+ "\n",
+ "pattern = r\"[,\\.\\n\\s]\"\n",
+ "docs = [\"doc1.txt\", \"doc2.txt\", \"doc3.txt\"]\n",
+ " \n",
+ "# Define function\n",
+ "def get_bow_from_docs(docs, stop_words=[]):\n",
+ " \n",
+ " # In the function, first define the variables you will use such as `corpus`, `bag_of_words`, and `term_freq`.\n",
+ " corpus = []\n",
+ " bag_of_words = set()\n",
+ " term_freq = []\n",
+ " \n",
+ " \"\"\"\n",
+ " Loop `docs` and read the content of each doc into a string in `corpus`.\n",
+ " Remember to convert the doc content to lowercases and remove punctuation.\n",
+ " \"\"\"\n",
+ " for doc in docs:\n",
+ " with open(doc, \"r\") as f:\n",
+ " text = f.read()\n",
+ " patterned_string = re.split(pattern, text)\n",
+ " patterned_string_no_spaces = [element.lower() for element in patterned_string if element != \"\"]\n",
+ " corpus.append(patterned_string_no_spaces)\n",
+ " \n",
+ " \"\"\"\n",
+ " Loop `corpus`. Append the terms in each doc into the `bag_of_words` array. The terms in `bag_of_words` \n",
+ " should be unique which means before adding each term you need to check if it's already added to the array.\n",
+ " In addition, check if each term is in the `stop_words` array. Only append the term to `bag_of_words`\n",
+ " if it is not a stop word.\n",
+ " \"\"\"\n",
+ " for vector in corpus: \n",
+ " for word in vector:\n",
+ " if word not in stop_words:\n",
+ " bag_of_words.add(word)\n",
+ " final_bag_of_words = list(bag_of_words)\n",
+ " \n",
+ " \"\"\"\n",
+ " Loop `corpus` again. For each doc string, count the number of occurrences of each term in `bag_of_words`. \n",
+ " Create an array for each doc's term frequency and append it to `term_freq`.\n",
+ " \"\"\"\n",
+ " for vector in corpus:\n",
+ " vector_freq = []\n",
+ " for word in final_bag_of_words:\n",
+ " vector_freq.append(vector.count(word))\n",
+ " term_freq.append(vector_freq)\n",
+ " \n",
+ " # Now return your output as an object\n",
+ " return {\n",
+ " \"bag_of_words\": final_bag_of_words,\n",
+ " \"term_freq\": term_freq\n",
+ " }\n",
+ " \n",
+ " \n",
+ "get_bow_from_docs(docs) "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Test your function without stop words. You should see the output like below:\n",
+ "\n",
+ "```{'bag_of_words': ['ironhack', 'is', 'cool', 'i', 'love', 'am', 'a', 'student', 'at'], 'term_freq': [[1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 1, 0, 0, 0, 0], [1, 0, 0, 1, 0, 1, 1, 1, 1]]}```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'bag_of_words': ['i', 'student', 'at', 'ironhack', 'am', 'cool', 'is', 'a', 'love'], 'term_freq': [[0, 0, 0, 1, 0, 1, 1, 0, 0], [1, 0, 0, 1, 0, 0, 0, 0, 1], [1, 1, 1, 1, 1, 0, 0, 1, 0]]}\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Define doc paths array\n",
+ "docs = docs = [\"doc1.txt\", \"doc2.txt\", \"doc3.txt\"]\n",
+ "\n",
+ "# Obtain BoW from your function\n",
+ "bow = get_bow_from_docs(docs)\n",
+ "\n",
+ "# Print BoW\n",
+ "print(bow)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If your attempt above is successful, nice work done!\n",
+ "\n",
+ "Now test your function again with the stop words. In the previous lab we defined the stop words in a large array. In this lab, we'll import the stop words from Scikit-Learn."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "frozenset({'many', 'became', 'without', 'your', 'few', 'because', 'we', 'bottom', 'himself', 'anything', 'moreover', 'of', 'mine', 'what', 'them', 'whether', 'seemed', 'my', 'whereas', 'un', 'across', 'another', 'by', 'itself', 'well', 'even', 'yet', 'never', 'both', 'here', 'interest', 'to', 'latterly', 'rather', 'five', 'beyond', 'some', 'it', 'he', 'beforehand', 'system', 'however', 'can', 'been', 'and', 'per', 'any', 'ten', 'nowhere', 'too', 'while', 'twelve', 'whereupon', 'enough', 'except', 'since', 'alone', 'thereafter', 'noone', 'onto', 'in', 'where', 'i', 'though', 'due', 'whence', 'co', 'once', 'all', 'front', 'sixty', 'neither', 'whose', 'this', 'couldnt', 'show', 'why', 'yourselves', 'everywhere', 'eg', 'four', 'hence', 'thick', 'six', 'being', 'ours', 'mill', 'side', 'for', 'thereupon', 'sometimes', 'fire', 'formerly', 'meanwhile', 'whereafter', 'towards', 'anyhow', 'ltd', 'whenever', 'up', 'get', 'first', 'cant', 'none', 'yours', 'hasnt', 'several', 'more', 'between', 're', 'has', 'nor', 'down', 'behind', 'ourselves', 'through', 'afterwards', 'hers', 'hereby', 'before', 'throughout', 'whereby', 'keep', 'ever', 'her', 'on', 'might', 'amongst', 'becomes', 'fifty', 'above', 'nothing', 'inc', 'during', 'that', 'three', 'eleven', 'such', 'than', 'from', 'whom', 'together', 'eight', 'therein', 'was', 'anywhere', 'over', 'his', 'often', 'until', 'next', 'do', 'fill', 'describe', 'forty', 'detail', 'only', 'somehow', 'amount', 'were', 'top', 'are', 'hereafter', 'hundred', 'upon', 'who', 'me', 'nevertheless', 'must', 'had', 'move', 'everything', 'myself', 'very', 'seeming', 'someone', 'wherein', 'him', 'within', 'whole', 'herein', 'will', 'amoungst', 'cannot', 'cry', 'former', 'full', 'namely', 'you', 'whither', 'back', 'seems', 'not', 'be', 'have', 'other', 'am', 'somewhere', 'others', 'else', 'they', 'herself', 'call', 'should', 'elsewhere', 'may', 'serious', 'an', 'toward', 'already', 'as', 'still', 'further', 'de', 'most', 'at', 'either', 'no', 'perhaps', 'beside', 'empty', 'under', 'thin', 'besides', 'twenty', 'sincere', 'therefore', 'give', 'into', 'thus', 'sometime', 'go', 'take', 'two', 'about', 'our', 'nobody', 'own', 'indeed', 'or', 'via', 'otherwise', 'every', 'anyone', 'etc', 'with', 'last', 'third', 'would', 'again', 'mostly', 'con', 'see', 'thereby', 'against', 'the', 'around', 'seem', 'which', 'done', 'when', 'everyone', 'less', 'anyway', 'one', 'now', 'after', 'part', 'could', 'name', 'there', 'us', 'below', 'find', 'ie', 'those', 'along', 'a', 'whoever', 'whatever', 'its', 'always', 'if', 'out', 'bill', 'hereupon', 'wherever', 'found', 'thru', 'put', 'then', 'among', 'something', 'these', 'yourself', 'become', 'how', 'each', 'same', 'made', 'off', 'nine', 'their', 'is', 'much', 'also', 'please', 'although', 'becoming', 'but', 'themselves', 'latter', 'fifteen', 'she', 'almost', 'thence', 'so', 'least'})\n"
+ ]
+ }
+ ],
+ "source": [
+ "from sklearn.feature_extraction import stop_words\n",
+ "print(stop_words.ENGLISH_STOP_WORDS)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You should have seen a large list of words that looks like:\n",
+ "\n",
+ "```frozenset({'across', 'mine', 'cannot', ...})```\n",
+ "\n",
+ "`frozenset` is a type of Python object that is immutable. In this lab you can use it just like an array without conversion."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, test your function with supplying `stop_words.ENGLISH_STOP_WORDS` as the second parameter."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'bag_of_words': ['love', 'student', 'cool', 'ironhack'], 'term_freq': [[0, 0, 1, 1], [1, 0, 0, 1], [0, 1, 0, 1]]}\n"
+ ]
+ }
+ ],
+ "source": [
+ "bow = get_bow_from_docs(docs, stop_words.ENGLISH_STOP_WORDS)\n",
+ "\n",
+ "print(bow)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You should have seen:\n",
+ "\n",
+ "```{'bag_of_words': ['ironhack', 'cool', 'love', 'student'], 'term_freq': [[1, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1]]}```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/lab-functional-programming/your-code/.ipynb_checkpoints/Q2-checkpoint.ipynb b/lab-functional-programming/your-code/.ipynb_checkpoints/Q2-checkpoint.ipynb
new file mode 100644
index 0000000..d764aaa
--- /dev/null
+++ b/lab-functional-programming/your-code/.ipynb_checkpoints/Q2-checkpoint.ipynb
@@ -0,0 +1,161 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we want to enhance the `get_bow_from_docs` function so that it will work with HTML webpages. In HTML, there are a lot of messy codes such as HTML tags, Javascripts, [unicodes](https://www.w3schools.com/charsets/ref_utf_misc_symbols.asp) that will mess up your bag of words. We need to clean up those junk before generating BoW.\n",
+ "\n",
+ "Next, what you will do is to define several new functions each of which is specialized to clean up the HTML codes in one aspect. For instance, you can have a `strip_html_tags` function to remove all HTML tags, a `remove_punctuation` function to remove all punctuation, a `to_lower_case` function to convert string to lowercase, and a `remove_unicode` function to remove all unicodes.\n",
+ "\n",
+ "Then in your `get_bow_from_doc` function, you will call each of those functions you created to clean up the HTML before you generate the corpus.\n",
+ "\n",
+ "Note: Please use Python string operations and regular expression only in this lab. Do not use extra libraries such as `beautifulsoup` because otherwise you loose the purpose of practicing."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Define your string handling functions below\n",
+ "\n",
+ "import re\n",
+ "\n",
+ "# Minimal 3 functions\n",
+ "def remove_spacing(text):\n",
+ " pattern = r\"\\s\"\n",
+ " words_no_spaces = re.split(pattern, text)\n",
+ " return words_no_spaces\n",
+ "\n",
+ "def remove_punctuation(words_no_spaces):\n",
+ " pattern = r\"[\\.,:;]\"\n",
+ " words_no_punctuation = [re.sub(pattern, \"\", word_no_spaces) for word_no_spaces in words_no_spaces]\n",
+ " return words_no_punctuation \n",
+ " \n",
+ "def to_lower_case(words_no_punctuation):\n",
+ " words_lower_case = [word_no_punctuation.lower() for word_no_punctuation in words_no_punctuation]\n",
+ " return words_lower_case"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, paste your previously written `get_bow_from_docs` function below. Call your functions above at the appropriate place."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def get_bow_from_docs(docs, stop_words=[]):\n",
+ " # In the function, first define the variables you will use such as `corpus`, `bag_of_words`, and `term_freq`.\n",
+ " corpus = []\n",
+ " bag_of_words = set()\n",
+ " term_freq = []\n",
+ " \n",
+ " for doc in docs:\n",
+ " with open(doc, \"r\", encoding = \"utf-8\") as f:\n",
+ " text = f.read()\n",
+ " words_no_spaces = remove_spacing(text)\n",
+ " words_no_punctuation = remove_punctuation(words_no_spaces)\n",
+ " words_lower_case = to_lower_case(words_no_punctuation)\n",
+ " corpus.append(words_lower_case)\n",
+ " \n",
+ " for vector in corpus: \n",
+ " for word in vector:\n",
+ " if word not in stop_words:\n",
+ " bag_of_words.add(word)\n",
+ " final_bag_of_words = list(bag_of_words)\n",
+ " \n",
+ " for vector in corpus:\n",
+ " vector_freq = []\n",
+ " for word in final_bag_of_words:\n",
+ " vector_freq.append(vector.count(word))\n",
+ " term_freq.append(vector_freq)\n",
+ " \n",
+ " return {\n",
+ " \"bag_of_words\": final_bag_of_words,\n",
+ " \"term_freq\": term_freq\n",
+ " }\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, read the content from the three HTML webpages in the `your-codes` directory to test your function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'bag_of_words': ['', 'fulfilling', 'chinese\"', 'representation', 'width=\"100%\"', 'toggle', 'module