Improve Automation for adding new mentors

.github/workflows/add_new_mentors.yml

@@ -0,0 +1,76 @@
+name: Add Newly Accepted Mentors
+
+on:
+  workflow_dispatch:
+    inputs:
+      file_id:
+        description: "Google Drive file ID of the Excel sheet with the new mentors data"
+        required: true
+      current_period:
+        description: "Current period (specify \"long-term\" if during long-term mentorship registration; otherwise, default is used)"
+        required: false
+        default: 'default'
+
+jobs:
+  add-new-mentors:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Cache pip
+        uses: actions/cache@v4
+        with:
+            path: ~/.cache/pip
+            key: ${{ runner.os }}-pip-${{ hashFiles('tools/requirements.txt') }}
+            restore-keys: |
+                ${{ runner.os }}-pip-
+
+      - name: Install dependencies
+        run: |
+            python -m pip install --upgrade pip
+            pip install -r tools/requirements.txt
+
+      - name: Install and Configure rclone with Google Cloud service account
+        run: |
+          curl https://rclone.org/install.sh | sudo bash
+          echo '${{ secrets.GOOGLECLOUD_SERVICE_KEY_RETRIEVE_ADHOC_FILE_JSON }}' > service_account.json
+          rclone config create gdrive drive scope=drive service_account_file=service_account.json
+
+      - name: Download spreadsheet from Google Drive
+        run: |
+          rclone backend copyid gdrive: ${{ github.event.inputs.file_id }} tools/samples/new_mentors.xlsx
+
+      - name: Run scripts to append new mentors to mentors.yml and download their profile picture(s)
+        run: |
+          cd tools
+          python3 automation_mentors.py samples/new_mentors.xlsx ../_data/mentors.yml ${{ github.event.inputs.current_period }} a 0
+          python3 download_image.py samples/new_mentors.xlsx
+
+      - name: Cleanup files
+        if: always()
+        run: rm -f service_account.json tools/samples/new_mentors.xlsx
+
+      - name: Create or Update Pull Request
+        uses: peter-evans/create-pull-request@v7
+        with:
+          token: ${{ secrets.GHA_ACTIONS_ALLOW_TOKEN }}
+          commit-message: "added new mentors"
+          branch: "automation/add-new-mentors"
+          team-reviewers: |
+            Women-Coding-Community/leaders
+          title: "[WCC Bot] Add New Mentors"
+          body: |
+            This PR was created automatically by a GitHub Action that handles adding new mentors.
+            `_data/mentors.yml` should be updated with the new data. Images for the new mentors should also have been downloaded and included in this PR.
+
+            Please review the changes and ensure that the changes are as expected before merging.
+          labels: |
+            automation
+            new-mentors

tools/README.md

@@ -1,9 +1,8 @@
 ## How to Run Python Scripts
 
-There are two automation scripts:
-1) `automation.py`: appends new mentors in `samples/mentors.xslx` to `_data/mentor.yml`
+1) `automation_mentors.py`: appends new mentors in `samples/mentors.xslx` to `_data/mentor.yml` (or updates all existing mentors if using WRITE mode)
 
-2) `download_image.py`: downloads image from a specified URL and saves in `assets/images/mentors`
+2) `download_image.py`: downloads image for each mentor from a specified URL and saves in `assets/images/mentors`. It uses data from `samples/mentors.xlsx` sheetname `Mentors Images`.
 
 3) `meetup_import.py`: imports new upcoming events from the WCC MeetUp page using the iCal feed: https://www.meetup.com/women-coding-community/events/ical/
 
@@ -17,21 +16,32 @@ python 3.11 or above
 
 ### How to Execute on Mac
 
-#### A) `automation.py`
+#### A) `automation_mentors.py`
 
 ```shell
-sh run_automation.sh
+sh run_mentor_automation.sh
 ```
-**Note:** 
-- Ensure to update `mentors.xslx` with the new spreadsheet containing the mentors to be added, **OR** 
-- adjust the `FILE_PATH_MENTORS_XLSX` parameter in [the script](run_automation.sh) to match the file path for the new spreadsheet.
+**Notes:**  
+If running locally:
+- Ensure to update `mentors.xslx` sheetname: `WCC All Approved Mentors` with new data containing the mentors to be added, **OR** 
+- If using another file source, adjust the `FILE_PATH_MENTORS_XLSX` parameter in [the script](run_mentor_automation.sh) to match the file path.
+- If running this script during long-term registration period, adjust the `CURRENT_PERIOD` parameter in [the script](run_mentor_automation.sh) to "long-term"
+
+- After running the script, you **HAVE** to run the [run_download_automation script](run_download_automation.sh) to download images for the new mentors. Else, the image links will be broken as they do not exist yet. Read the instructions for the download script usage below.
+
+**If using GitHub Actions**, the GHA workflow is **ONLY** for adding new mentors.
+It uses a Google Cloud service account setup to retrieve the Excel file from Google Drive. The service key has been configured for womencodingcommunity Google Drive account and the file to be used/updated has been shared with the service account email.
+  Hence, to run the GHA workflow, you only need to provide:
+  - the file ID for the excel sheet to use
+  - (Optional) the current period
+
+For more information on the GC service account configurations, you can read the [documentation](blog_automation/README.md) in the blog automation folder.
 
 
 #### B) `download_image.py`
 
-**Before running the script, make sure** to update the `IMAGE_URL` and `MENTOR_NAME` parameters in the [run_download_automation script](run_download_automation.sh) with:
-- the URL you want to download the mentor's image from, **AND**
-- the mentor's name as it appears in the spreadsheet e.g 'Adriana Zencke'
+**Before running the script, make sure** to update `mentors.xslx` sheetname: `Mentors Images` with the data for the new mentors that you want to download their images
+If you want to use another file source, adjust `XLSX_FILE_PATH` parameter in the [script](run_download_automation.sh) to match the file path.
 
 You can then run: 
 ```shell

tools/automation.py → tools/automation_mentors.py

@@ -9,6 +9,7 @@
 import textwrap
 from enum import Enum
 
+import numpy as np
 import pandas as pd
 from ruamel.yaml import YAML
 from ruamel.yaml.scalarstring import LiteralScalarString
@@ -37,6 +38,10 @@
 IMAGE_FILE_PATH = "assets/images/mentors"
 IMAGE_SUFFIX = ".jpeg"
 
+# Mentorship cycle periods
+LONG_TERM_REG_PERIOD = "long-term" # long-term registrations period only
+DEFAULT_PERIOD = "default" # rest of the cycle, ad-hoc periods
+
 
 class WriteMode(Enum):
     # Create new a file
@@ -130,23 +135,40 @@ def get_multiline_string(long_text_arg):
         multiline_str = LiteralScalarString(textwrap.dedent(long_text_arg))
     return multiline_str
 
-def get_sort(mentorship_type, num_mentee):
+def is_available_for_long_term(mentorship_type):
+    return mentorship_type == type_long_term[0] or mentorship_type == TYPE_BOTH
+
+def is_available_for_ad_hoc(mentorship_type):
+    return mentorship_type == type_ad_hoc[0] or mentorship_type == TYPE_BOTH
+
+def sort_for_long_term_reg(num_mentee):
+    """
+    Return sort value for mentors available for long-term, based on number of mentees they can take.
+    Applies only during long-term registration period.
+    if no mentees, sort to 10; if num_mentees is 1, sort to 100; 2, sort to 200; if >2, sort to highest 500
+    """
+
+    mentee_sort_map = {
+        0: 10, 
+        1: 100, 
+        2: 200
+    }
+    return mentee_sort_map.get(num_mentee, 500)
+
+
+def get_sort(mentorship_type, current_period, num_mentee):
     """
-    Get mentor's sort value
+    Get sort value for a new mentor
+    Rules: https://docs.google.com/document/d/1GwlleBNScHCQ3K8rgvYIB3upIr1BylgWjGR2jxwYWtI/edit?usp=sharing
     """
 
-    if mentorship_type == TYPE_BOTH or mentorship_type == type_long_term[0]:
-        if num_mentee > 2:
-            return 600
-        if num_mentee == 2:
-            return 550
-        if num_mentee == 1:
-            return 500
-        return 200
-    if mentorship_type == type_ad_hoc[0]:
-        #todo: (if availability == next month) then adjust the sort value:
-        return 100
+    if current_period == LONG_TERM_REG_PERIOD and is_available_for_long_term(mentorship_type):
+        return sort_for_long_term_reg(num_mentee)
+
+    if current_period == DEFAULT_PERIOD and is_available_for_ad_hoc(mentorship_type):
+        return 500
 
+    # else the mentor is not available for any periods
     return 10
 
 def get_mentorship_type(mentorship_type_str):
@@ -237,34 +259,48 @@ def read_yml_file(file_path):
 
     return yml_dict
 
+def get_num_mentee_from_row(mentor_row):
+    """
+    Gets the 'num_mentee' value for a new mentor from mentor_row, or use a default value if invalid.
+    """
+    val = mentor_row.iloc[44]
+
+    return int(val) if pd.notna(val) else 0
+
+def get_mentor_position(mentor_row):
+    """
+    Returns formatted value for mentor role and company
+    """
+    if not pd.isna(mentor_row.iloc[9]):
+        return f"{mentor_row.iloc[8].strip()}, {mentor_row.iloc[9].strip()}"
+    else:
+        return mentor_row.iloc[8].strip()
+
 
 def xlsx_to_yaml_parser(mentor_row,
                         mentor_index,
+                        current_period,
                         mentor_disabled=False,
                         mentor_sort=0,
                         mentor_matched=False,
-                        num_mentee=1):
+                        num_mentee=0):
     """
     Prepare mentor's excel data for yaml format
     """
     areas = get_yaml_block_sequence(mentor_row, AREAS_START_INDEX, AREAS_END_INDEX)
     focus = get_yaml_block_sequence(mentor_row, FOCUS_START_INDEX, FOCUS_END_INDEX)
     programming_languages = get_yaml_block_sequence(mentor_row, PROG_LANG_START_INDEX, PROG_LANG_END_INDEX)
 
-    # Left commented since the code might be used in the later versions
-    # to add default picture until the mentor's image is not available
-    # mentor_image = os.path.join(IMAGE_FILE_PATH, str(mentor_index) + IMAGE_SUFFIX)
-    mentor_image = f"{IMAGE_FILE_PATH}/{mentor_row.iloc[2].strip().lower().replace(' ', '_')}{IMAGE_SUFFIX} # TODO: Run download_image script to actually download the image"
+    mentor_image = f"{IMAGE_FILE_PATH}/{mentor_row.iloc[2].strip().lower().replace(' ', '_')}{IMAGE_SUFFIX}"
+    # Format mentor role and company
+    mentor_position = get_mentor_position(mentor_row)
 
     mentor_type = get_mentorship_type(mentor_row.iloc[4])
 
+    # If mentor is new i.e mentor_sort is 0 (from default input), get the correct num_mentees and sort values
     if mentor_sort == 0:
-        mentor_sort = get_sort(mentor_type, num_mentee)
-
-    if not pd.isna(mentor_row.iloc[9]):
-        mentor_position = f"{mentor_row.iloc[8].strip()}, {mentor_row.iloc[9].strip()}"
-    else:
-        mentor_position = mentor_row.iloc[8].strip()
+        num_mentee = get_num_mentee_from_row(mentor_row)
+        mentor_sort = get_sort(mentor_type, current_period, num_mentee)
 
     mentor = {
         'name': mentor_row.iloc[2].strip(),
@@ -329,7 +365,7 @@ def get_yml_data(yml_file_path):
     return df_yml_data
 
 
-def get_all_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=0):
+def get_all_mentors_in_yml_format(yml_file_path, xlsx_file_path, current_period, skip_rows=0):
     """
     Read all mentors from Excel sheet:
      - if mentor is in current mentors.yml, use existing values for index, disabled, sort, matched and num_mentee.
@@ -356,14 +392,16 @@ def get_all_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=0):
         if not df_yml_row.empty:
             mentor = xlsx_to_yaml_parser(df_mentors.iloc[row],
                                         df_yml_row['Index'].item(),
+                                        current_period,
                                         df_yml_row['Disabled'].item(),
                                         df_yml_row['Sort'].item(),
                                         df_yml_row['Matched'].item(),
                                         df_yml_row['Num_mentee'].item())
             logging.info(f"For {mentor_name} use index, disabled and sort from mentors.yml file")
         else:
             mentor = xlsx_to_yaml_parser(df_mentors.iloc[row],
-                                        new_index)
+                                        new_index,
+                                        current_period)
             new_index += 1
         mentors.append(mentor)
 
@@ -372,7 +410,7 @@ def get_all_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=0):
     return mentors
 
 
-def get_new_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=1):
+def get_new_mentors_in_yml_format(yml_file_path, xlsx_file_path, current_period, skip_rows=1):
     """
     Read just new mentors from Excel sheet:
      - start reading xlsx Mentors from the row 1 (from the date 03/04/2024)
@@ -397,7 +435,7 @@ def get_new_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=1):
             mentor_name = df_mentors.iloc[row].values[2].strip().lower()
 
             if df_yml.loc[df_yml.Name == mentor_name].empty:
-                mentor = xlsx_to_yaml_parser(df_mentors.iloc[row], new_index)
+                mentor = xlsx_to_yaml_parser(df_mentors.iloc[row], new_index, current_period)
                 new_index += 1
                 mentors.append(mentor)
 
@@ -411,25 +449,27 @@ def get_new_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=1):
 def run_automation():
     logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
 
-    if len(sys.argv) == 5:
+    if len(sys.argv) == 6:
         xlsx_file_path = sys.argv[1]
         yml_file_path = sys.argv[2]
-        mode = WriteMode(sys.argv[3])
-        skip_rows = int(sys.argv[4])
+        current_period = sys.argv[3]
+        mode = WriteMode(sys.argv[4])
+        skip_rows = int(sys.argv[5])
 
-        logging.info("Params: xlsx: %s yml: %s mode: %s skip_rows: %s", xlsx_file_path, yml_file_path, mode, skip_rows)
+        logging.info("Params: xlsx: %s yml: %s current_period: %s mode: %s skip_rows: %s", xlsx_file_path, yml_file_path, current_period, mode, skip_rows)
     else:
         xlsx_file_path = "samples/mentors.xlsx"
         yml_file_path = "samples/mentors.yml"
+        current_period = "default"
         mode = WriteMode.APPEND
         skip_rows = 0
 
-        logging.info("Default values: xlsx: %s yml:: %s mode: %s", xlsx_file_path, yml_file_path, mode)
+        logging.info("Default values: xlsx: %s yml:: %s current_period: %s mode: %s", xlsx_file_path, yml_file_path, current_period, mode)
 
     if mode == WriteMode.APPEND:
         logging.info("Appending option selected.")
 
-        list_of_mentors = get_new_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=skip_rows)
+        list_of_mentors = get_new_mentors_in_yml_format(yml_file_path, xlsx_file_path, current_period, skip_rows=skip_rows)
 
         logging.info("New Mentors size: %d", len(list_of_mentors))
 
@@ -439,7 +479,7 @@ def run_automation():
     elif mode == WriteMode.WRITE:
         logging.info("Recreate yml - Write option selected.")
 
-        list_of_mentors = get_all_mentors_in_yml_format(yml_file_path, xlsx_file_path, skip_rows=skip_rows)
+        list_of_mentors = get_all_mentors_in_yml_format(yml_file_path, xlsx_file_path, current_period, skip_rows=skip_rows)
         write_yml_file(yml_file_path, list_of_mentors, WriteMode.WRITE)