Update practicals

Author: johnsamuelwrites

en/Project/project.md

@@ -11,15 +11,23 @@
     "\n",
     "-   Parsing and working with CSV, TSV and JSON files\n",
     "-   Querying external data sources\n",
-    "-   Data analyses\n",
+    "-   Data analysis with pandas\n",
     "\n",
-    "### Exercises\n",
+    "## Prerequisites\n",
     "\n",
-    "1.  Parsing and reading CSV/TSV files\n",
-    "2.  Parsing and reading JSON files\n",
-    "3.  Querying external data sources (Query endpoints and API)\n",
-    "4.  Performing classical data analyses\n",
-    "\n"
+    "-   Completion of [Practical 0](../practical0/practical0.ipynb)\n",
+    "-   Basic understanding of Python data structures\n",
+    "\n",
+    "## Exercises Overview\n",
+    "\n",
+    "| Exercise | Difficulty | Topics |\n",
+    "|----------|------------|--------|\n",
+    "| Exercise 1 | ★ | Setup and package installation |\n",
+    "| Exercise 2 | ★ | Parsing CSV/TSV files with NumPy |\n",
+    "| Exercise 3 | ★★ | Parsing JSON files with pandas |\n",
+    "| Exercise 4 | ★★ | Querying external data sources (Wikidata) |\n",
+    "| Exercise 5 | ★★★ | Data grouping and aggregation |\n",
+    "| Exercise 6 | ★★★ | Downloading and processing images |"
    ]
   },
   {
@@ -58,7 +66,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!pip3 install numpy pandas matplotlib sparqlwrapper"
+    "!pip3 install numpy pandas matplotlib sparqlwrapper opencv-python"
    ]
   },
   {
@@ -108,7 +116,7 @@
    "id": "805f1ac0-711b-42a1-90fd-f45a33b69d50",
    "metadata": {},
    "source": [
-    "Practise the exercises given in [practicals 0](../practical0/practical0.ipynb)."
+    "Practise the exercises given in [Practical 0](../practical0/practical0.ipynb) before continuing."
    ]
   },
   {
@@ -482,7 +490,27 @@
    "id": "2c698304-8f67-439e-b3ca-b350cfa29430",
    "metadata": {},
    "source": [
-    "**Question**: What is the difference between np.loadtxt() and np.genfromtxt() when reading CSV files? When should you prefer one over the other?"
+    "**Question 1**: What is the difference between `np.loadtxt()` and `np.genfromtxt()` when reading CSV files? When should you prefer one over the other?\n",
+    "\n",
+    "**Key Differences:**\n",
+    "\n",
+    "| Feature | `np.loadtxt()` | `np.genfromtxt()` |\n",
+    "|---------|----------------|-------------------|\n",
+    "| Missing values | Raises error | Handles gracefully |\n",
+    "| Speed | Faster | Slower |\n",
+    "| Flexibility | Less flexible | More options |\n",
+    "| Use case | Clean, well-formatted data | Data with potential issues |\n",
+    "\n",
+    "**When to use which:**\n",
+    "- Use `loadtxt()` when you're confident your data is clean and complete\n",
+    "- Use `genfromtxt()` when dealing with real-world data that may have missing values or inconsistencies\n",
+    "\n",
+    "**Question 2**: Write a data validation function that checks the loaded dataset for:\n",
+    "1. Any year values that are negative or in the future (> current year)\n",
+    "2. Any names that are empty or contain only whitespace\n",
+    "3. Return a report showing how many invalid entries were found\n",
+    "\n",
+    "**Hint:** Use a loop to iterate through the dataset and check each entry."
    ]
   },
   {
@@ -628,6 +656,7 @@
    "outputs": [],
    "source": [
     "# Get some descriptive summary of the dataframe\n",
+    "# By default, describe() only shows statistics for numerical columns\n",
     "dataframe.describe()"
    ]
   },
@@ -815,12 +844,15 @@
     "dataframe[\"year\"] = pd.to_numeric(dataframe[\"year\"], errors=\"coerce\")\n",
     "\n",
     "# Identify missing values\n",
+    "print(\"Missing values per column:\")\n",
     "print(dataframe.isnull().sum())\n",
     "\n",
     "# Fill missing values in \"year\" with the median value\n",
-    "dataframe[\"year\"].fillna(dataframe[\"year\"].median(), inplace=True)\n",
+    "# Note: Using assignment instead of inplace=True (which is deprecated)\n",
+    "dataframe[\"year\"] = dataframe[\"year\"].fillna(dataframe[\"year\"].median())\n",
+    "\n",
     "# Fill missing values in \"languageLabel\" with \"Unknown\"\n",
-    "dataframe[\"languageLabel\"].fillna(\"Unknown\", inplace=True)"
+    "dataframe[\"languageLabel\"] = dataframe[\"languageLabel\"].fillna(\"Unknown\")"
    ]
   },
   {
@@ -946,7 +978,24 @@
    "id": "9a53a3ac-d287-4a8b-8b62-4d7cac4063b4",
    "metadata": {},
    "source": [
-    "**Question:**  What is the difference between `json.load()` and `json.loads()` in Python?  "
+    "**Question 1:**  What is the difference between `json.load()` and `json.loads()` in Python?\n",
+    "\n",
+    "**Question 2:** Create a new column `decade` that groups programming languages by their decade of creation (1950s, 1960s, etc.):\n",
+    "\n",
+    "```python\n",
+    "# Example: 1962 should become \"1960s\"\n",
+    "dataframe['decade'] = (dataframe['year'] // 10 * 10).astype(str) + 's'\n",
+    "```\n",
+    "\n",
+    "Then:\n",
+    "1. Count how many languages were created in each decade\n",
+    "2. Find the decade with the most programming languages\n",
+    "3. Create a bar chart showing the number of languages per decade\n",
+    "\n",
+    "**Question 3:** Clean and transform the `languageLabel` column:\n",
+    "1. Convert all names to title case (first letter capitalized)\n",
+    "2. Remove any leading/trailing whitespace\n",
+    "3. Identify any duplicate entries (same language, same year)"
    ]
   },
   {
@@ -980,7 +1029,13 @@
    "cell_type": "markdown",
    "id": "c01a458e-d30f-4b75-aa33-0932a183b23f",
    "metadata": {},
-   "source": []
+   "source": [
+    "**What is SPARQL?**\n",
+    "\n",
+    "SPARQL (SPARQL Protocol and RDF Query Language) is a query language used to retrieve and manipulate data stored in RDF (Resource Description Framework) format. Wikidata uses RDF to store its knowledge graph, and SPARQL allows us to query this data.\n",
+    "\n",
+    "The URL above contains an encoded SPARQL query that retrieves programming languages and their inception years from Wikidata."
+   ]
   },
   {
    "cell_type": "code",
@@ -994,8 +1049,16 @@
     "import pandas as pd\n",
     "\n",
     "url = \"https://query.wikidata.org/sparql?query=SELECT%20%3FlanguageLabel%20(YEAR(%3Finception)%20as%20%3Fyear)%0AWHERE%0A%7B%0A%20%20%23instances%20of%20programming%20language%0A%20%20%3Flanguage%20wdt%3AP31%20wd%3AQ9143%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20wdt%3AP571%20%3Finception%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20rdfs%3Alabel%20%3FlanguageLabel.%0A%20%20FILTER(lang(%3FlanguageLabel)%20%3D%20%22en%22)%0A%7D%0AORDER%20BY%20%3Fyear%0ALIMIT%20100&format=json\"\n",
-    "response = urllib.request.urlopen(url)\n",
-    "responsedata = json.loads(response.read().decode(\"utf-8\"))\n",
+    "\n",
+    "req = urllib.request.Request(\n",
+    "    url,\n",
+    "    headers={\n",
+    "        \"User-Agent\": \"GitHubAction/1.0 (https://github.com/johnsamuelwrites/DataMining)\"\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "with urllib.request.urlopen(req) as response:\n",
+    "    responsedata = json.loads(response.read().decode(\"utf-8\"))\n",
     "\n",
     "array = []\n",
     "\n",
@@ -1359,15 +1422,39 @@
     "    the above query results) since 2010.\n",
     "\n",
     "**Hint**:Take a look at functions groupby, reset_index, head, tail, sort_values, count of Pandas\n",
-    "\n"
+    "\n",
+    "**Question 3: Merging DataFrames**\n",
+    "\n",
+    "You now have two datasets from this practical:\n",
+    "- Programming languages data (language name, year)\n",
+    "- Population data (country, year, population)\n",
+    "\n",
+    "Create a summary DataFrame that shows, for each year:\n",
+    "1. Number of programming languages created\n",
+    "2. Any population data available\n",
+    "\n",
+    "Use `pd.merge()` to combine these datasets:\n",
+    "```python\n",
+    "# Example of merging on year\n",
+    "merged = pd.merge(lang_df, pop_df, on='year', how='outer')\n",
+    "```\n",
+    "\n",
+    "**Questions to answer:**\n",
+    "- Which years have both programming language and population data?\n",
+    "- What type of merge (inner, outer, left, right) is most appropriate here? Why?"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "99eb67df-0773-4a58-be20-6d22d0dfaa3b",
    "metadata": {},
    "source": [
-    "**Note**: If you get time-out errors, please change the LIMIT to some lower values (1000, 2000, 5000)."
+    "**Troubleshooting:**\n",
+    "\n",
+    "- If you get timeout errors, reduce the `LIMIT` value in the query (try 1000, 2000, or 3000)\n",
+    "- If the query service is unavailable, try again after a few minutes\n",
+    "- Make sure you have a stable internet connection\n",
+    "- The Wikidata query service may have usage limits during peak hours"
    ]
   },
   {
@@ -1536,7 +1623,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": ".venv",
    "language": "python",
    "name": "python3"
   },
@@ -1550,7 +1637,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.13.0"
   }
  },
  "nbformat": 4,