Document GitHub Sentiment Dataset Pipeline

[splimon]

Purpose

To create clear and reproducible documentation so future contributors can understand and rerun the GitHub sentiment pipeline without reverse‑engineering prior work. The notebooks should explain the commit/PR comment flow and produce outputs compatible with Kaiaulu.

Process

Create a documentation PR that:

• Adds the .yml files for the GitHub projects used in the dataset
• Adds the Python script used to load sentiment labels into MySQL
• Converts existing MySQL Workbench queries into Jupyter notebooks
• Organizes notebooks in a clear way:
  • Load sentiment CSV into MySQL
  • Explore DB + identify relevant tables
  • Scale/automate across all projects
• Export each notebook to a .py file so reviewers can read the code directly in GitHub (in addition to the .ipynb)
• Does not commit any data files (e.g., CSV exports, database dumps, generated data, etc.), only code and documentation (scripts, notebooks, configs)

The final notebook must output a file that can be inner joined with Kaiaulu output for the same project.

Task List

• Add .yml config project files to the repo
• Add Python loader script for loading sentiment labels into MySQL
• Create Notebook 1: load sentiment CSV into MySQL
• Create Notebook 2: explore DB + locate relevant tables/joins
• Create Notebook 3: scale/automate pipeline across all projects
• Export notebooks to .py for code review
• Ensure final notebook outputs a file that is joinable with Kaiaulu

[splimon]

4/3/26 Capstone Meeting Notes (Sentiment Classifier)

Hi @carlosparadis,

Next Steps

Notebook Documentation

• Use Python library calls to access and run SQL queries automatically in the notebooks (no manual SQL copy/paste)
• Display query results inline so users can validate outputs quickly
• Finalize the GitHub commit + PR comment documentation workflow. Use the notebooks as a guide so a new contributor can follow the notebook steps end-to-end without additional clarification
• After GitHub documentation is complete, apply the same workflow and documentation pattern to the Jira emotion dataset: Implement sentiment workflow with Jira Dataset emotion_jira_dataset#1

[splimon]

4/10/26 Meeting Notes

Hi @carlosparadis ,

Next Steps

Notebook Documentation

• Notebooks 1-2 narrative:
  • Problem: the Gold Standard CSV only has ID, polarity, and text. It can't connect to Kaiaulu as-is
  • Goal: add columns from GHTorrent so the CSV can be INNER JOINed to Kaiaulu output
• Notebook 2 (Exploring GHTorrent Tables) revisions:
  • End with two DataFrames: (1) original Gold Standard header, (2) contextualized dataset header with an explanation of why each new column is needed. Also add final contextualized CSV at the end. This will be the CSV that will be INNER JOINed to the Kaiaulu output and is also the input for Notebook 4.
• Notebook 3 (Scale config files) narrative:
  • Reframe intro: "Now that we have the contextualized CSV, let's generate configs so we can get Kaiaulu data to join it with"
• Notebook 4 (NEW NOTEBOOK):
  • Inputs: contextualized CSV from Notebook 2 + parsed Kaiaulu output
  • Make clear how the Kaiaulu output was produced: Name the downloader/refresher function and parser function used
  • Load both files, INNER JOIN on comment_id, show the result
  • Calling exec in Kaiaulu to do this programmatically is a future implementation