feat(i18n): centralize translations in Google Sheets and document wor…

add-a-new-language.md

@@ -2,103 +2,78 @@
 
 This guide is divided into two main parts:
 
-1. **[Adding a New Language](#adding-a-new-language)**: Instructions for introducing a new locale across the backend and frontend.
+1. **[Adding a New Language](#adding-a-new-language)**: Instructions for introducing a new locale using the automated Google Sheets workflow.
 2. **[Switching or Using Supported Languages](#switching-or-using-supported-languages)** : Guidance for configuring or switching between languages that are already supported.
 
 ## Adding a New Language
 
-This section shows how to introduce a new locale end-to-end across the backend and the frontend. It includes what files to add, the expected templates, which configs to update, and how to verify everything.
+Compass uses an automated translation workflow based on Google Sheets. This allows non-technical users to add translations from a familiar spreadsheet, while developers simply run an import script to automatically apply them to the codebase.
 
-Locale identifiers follow [IETF BCP 47](https://www.ietf.org/rfc/bcp/bcp47.txt) (e.g., `en-GB`, `es-AR`). Keep language in lowercase and region in uppercase. Keep codes consistent across config and folders.
+Locale identifiers must follow [IETF BCP 47](https://www.ietf.org/rfc/bcp/bcp47.txt) (e.g., `en-GB`, `es-AR`, `fr-FR`). Keep language codes in lowercase and region codes in uppercase.
 
-Notes
+### 1. For Translators (Non-Technical)
 
-* Use BCP‑47 language tags (e.g., `en-GB`, `en-US`, `es-ES`, `es-AR`). Keep codes consistent across config and folders.
-* Backend uses locale directories under `backend/app/i18n/locales/<locale>/` with domain JSON files (e.g., `messages.json`).
-* Frontend-new uses `frontend-new/src/i18n/locales/<locale>/translation.json` and maps them in [`frontend-new/src/i18n/i18n.ts`](frontend-new/src/i18n/i18n.ts).
-* Supported languages are enabled via environment variables on both backend and frontend.
+1. **Open the Compass Translation Sheet:** Access the shared Google Sheet provided by the development team.
+2. **Add a New Language Column:** Find the first empty column header after the existing language codes. Type the BCP-47 language code for your new language (e.g., `fr-FR` for French) and press Enter.
+   > **Note:** Do NOT modify the `Platform` or `Key` columns, as these are strict technical identifiers required for the application to function and **must only be edited by developers or other authorised maintainers**.
+3. **Fill in the Translations:** Scroll through the rows. Use the English (`en-GB` or `en-US`) columns as a reference. Type your translated text into your new column. If you leave a cell blank, the application will automatically fall back to English for that specific text.
+4. **Notify the Team:** Once translations are complete, notify the development team to run the import script.
 
-### Prerequisites
+### 2. For Developers (Technical)
 
-* Decide the language code(s) you want to support (e.g., `es-AR`).
-* Pick a reference language to copy from (English is recommended):
-  - Backend reference: [`backend/app/i18n/locales/en-US/messages.json`](backend/app/i18n/locales/en-US/messages.json)
-  - Frontend-new reference: [`frontend-new/src/i18n/locales/en-GB/translation.json`](frontend-new/src/i18n/locales/en-GB/translation.json)
+Once translations or new keys have been added to the Google Sheet, follow these steps to pull them into the codebase.
 
-### 1. Backend
-
-The backend uses `LocaleProvider` to determine the current locale. The default locale in `BACKEND_LANGUAGE_CONFIG` configuration is used as the default.
-
-#### 1.1 Create Locale directory and messages file
-
-* Create a folder for your new locale under `backend/app/i18n/locales/<locale>/` and add a `messages.json` file with the same keys as English.
-* Translate backend-facing strings in `backend/app/i18n/locales/<locale>/messages.json`. Use [`en-US/messages.json`](backend/app/i18n/locales/en-US/messages.json) as the reference.
-* Key consistency is enforced by [`backend/app/i18n/test_i18n.py`](backend/app/i18n/test_i18n.py).
-
-Example 
-```
-backend/app/i18n/locales/en-US/messages.json     # reference
-backend/app/i18n/locales/es-AR/messages.json     # new 
-```
-For the message structure and keys, please refer to the reference file: [`backend/app/i18n/locales/en-US/messages.json`](backend/app/i18n/locales/en-US/messages.json).
+> **Supported platforms**
+>
+> Currently, only three platforms are powered by this workflow:
+> - **Frontend** (React app)
+> - **Backend** (Python services)
+> - **Feedback** (survey/questions content)
+>
+> Only rows/keys that belong to one of these platforms will be generated. If a *new* platform is ever introduced, the import script must be extended to support it before any keys for that platform will have an effect.
 
-#### 1.2 Update Supported Constants and Environment Variables
+#### 2.1 Run the Import Script
 
-  - Add the locale to [`backend/app/i18n/types.py`](backend/app/i18n/types.py) (`Locale` enum and `SUPPORTED_LOCALES`).
-  - Update [`backend/app/i18n/constants.py`](backend/app/i18n/constants.py) only if the default fallback changes.
-  - Enable the new locale in your backend environment via `BACKEND_LANGUAGE_CONFIG`.
-
-#### 1.3 Verify Backend Key Consistency
-
-`I18nManager` can verify that all locales contain the same keys per domain; [`backend/app/i18n/test_i18n.py`](backend/app/i18n/test_i18n.py) enforces key parity.
-
-Optional commands
+Navigate to the `backend` directory and run the automated import script:
 
 ```bash
 cd backend
-poetry run python scripts/verify_i18n_keys.py --verify
-```
-
-### 2. Frontend 
-
-The frontend uses `i18next` with resources defined in locale files and mapped in [`frontend-new/src/i18n/i18n.ts`](frontend-new/src/i18n/i18n.ts).
-
-#### 2.1 Create the locale folder and translation file
-
-Create a folder for your new locale under `frontend-new/src/locales/<locale>/` and copy the reference `translation.json` from English (`en-GB`). Translate values, keeping all keys identical.
-
-Example
+poetry run python scripts/import_translations_from_sheets.py
 ```
-frontend-new/src/i18n/locales/en-GB/translation.json   # reference
-frontend-new/src/i18n/locales/es-AR/translation.json   # new
-```
-
-For the translation structure and keys, please refer to the reference file: [`frontend-new/src/i18n/locales/en-GB/translation.json`](frontend-new/src/i18n/locales/en-GB/translation.json).
-
-* **Add other locale files:**
-  - Add `frontend-new/src/feedback/overallFeedback/feedbackForm/questions-<locale>.json` mirroring the existing files, e.g., [`questions-en-GB.json`](frontend-new/src/feedback/overallFeedback/feedbackForm/questions-en-GB.json).
-  - Add `frontend-new/public/data/config/fields-<locale>.yaml`; keep the same structure/keys as [`fields-en-GB.yaml`](frontend-new/public/data/config/fields-en-GB.yaml).
 
-#### 2.2 Register Locale Resources
+**What this script does:**
+* Connects to the Google Sheet and downloads the translation matrices.
+* Generates or updates the corresponding JSON files in `frontend-new/src/i18n/locales/`, `backend/app/i18n/locales/`, and `frontend-new/src/feedback/overallFeedback/feedbackForm/`.
+* Ensures that **new translation keys** defined in the sheet are created for the supported platforms, and keeps existing keys in sync.
+* **Auto-registers the new locale:** It automatically patches the codebase to recognize the new language by modifying:
+  * `config/default.json` (Adds locale to `supportedLocales` and injects English fallbacks into `sensitiveData`)
+  * `backend/app/i18n/types.py` (Locale enum and `SUPPORTED_LOCALES`)
+  * `frontend-new/src/i18n/constants.ts` (Locale enum, `LocalesLabels`, `SupportedLocales`)
+  * `frontend-new/src/i18n/i18n.ts` (Import statements and internal resource mapping)
 
-* Update [`frontend-new/src/i18n/constants.ts`](frontend-new/src/i18n/constants.ts) (add to `Locale`, `LocalesLabels`, `SupportedLocales`, and adjust `FALL_BACK_LOCALE` only if the fallback changes).
-* Import and register the translation and feedback JSON files in  [`frontend-new/src/i18n/i18n.ts`](frontend-new/src/i18n/i18n.ts).
+> **When to re-run the script**
+>
+> Any time the shared translation sheet is updated (new language, updated strings, or new keys), a developer must:
+> 1. Re-run `scripts/import_translations_from_sheets.py`.
+> 2. Commit the regenerated locale files and configuration changes.
+> 3. Deploy the updated services so the changes are visible in all environments.
 
-#### 2.3 Environment Config
+#### 2.2 Manual Steps Required
 
-Enable the new locale via environment variables:
-* `FRONTEND_SUPPORTED_LOCALES`: a JSON array of enabled locale codes (see [`parseEnvSupportedLocales.ts`](frontend-new/src/i18n/languageContextMenu/parseEnvSupportedLocales.ts) for validation rules).
-* `FRONTEND_DEFAULT_LOCALE`: default language if user preference not set.
+While the script handles JSON translations and locale registration, one manual step remains:
 
-Environment variables are Base64-encoded and read from [`frontend-new/public/data/env.example.js`](frontend-new/public/data/env.example.js).
+1. **Embeddings:** Generate taxonomy embeddings for the new language using the appropriate model ID (see [Generate Embeddings](./deployment-procedure.md#step-43-generate-embeddings)).
 
-#### 2.4 Verify Frontend Key Consistency
+#### 2.3 Verify Key Consistency
 
-There is an automated test that ensures all locales share the same keys as English (`en-GB`): [`frontend-new/src/i18n/locales/locales.test.ts`](frontend-new/src/i18n/locales/locales.test.ts).
-
-Optional commands
+You can optionally run the automated tests to ensure exact key consistency across platforms:
 
 ```bash
+# Backend verification
+cd backend
+poetry run python scripts/verify_i18n_keys.py --verify
+
+# Frontend verification
 cd frontend-new
 yarn test -- src/i18n/locales/locales.test.ts
 ```
@@ -107,30 +82,13 @@ yarn test -- src/i18n/locales/locales.test.ts
 
 Supported languages are enabled via environment variables on both backend and frontend.
 
-* **Add environment variables for the frontend** ([see deployment procedure](./deployment-procedure.md) and [upload to secret manager](./deployment-procedure.md):
+* **Add environment variables for the frontend** ([see deployment procedure](./deployment-procedure.md) and [upload to secret manager](./deployment-procedure.md)):
   - `FRONTEND_SUPPORTED_LOCALES`: JSON string array of supported locales, e.g., `["en-GB","es-ES"]`.
   - `FRONTEND_DEFAULT_LOCALE`: Default locale string.
 
 * **Add environment variables for the backend** ([see deployment procedure](./deployment-procedure.md) and [upload to secret manager](./deployment-procedure.md)):
   - `BACKEND_LANGUAGE_CONFIG`: Default backend locale (BCP 47).
 
-### 4.  Quick checklist (summary)
-
-* **Backend**
-  * [ ] Create `backend/app/i18n/locales/<locale>/messages.json` with the same keys as English
-  * [ ] Add language configurations to `BACKEND_LANGUAGE_CONFIG`
-  * [ ] Add the locale to `backend/app/i18n/types.py` (`Locale` enum and `SUPPORTED_LOCALES`)
-  * [ ] Generate embeddings for the new language using the new taxonomy model ID \([Generate Embeddings](./deployment-procedure.md#step-43-generate-embeddings)\)
-  * [ ] (Optional) Run backend i18n verify script
-
-* **Frontend-new**
-  * [ ] Create `frontend-new/src/i18n/locales/<locale>/translation.json` with the same keys as English (`en-GB`)
-  * [ ] Import and register resources in `frontend-new/src/i18n/i18n.ts`
-  * [ ] Update `frontend-new/src/i18n/constants.ts` (`Locale`, `LocalesLabels`, `SupportedLocales`)
-  * [ ] Add `frontend-new/src/feedback/overallFeedback/feedbackForm/questions-<locale>.json`
-  * [ ] Add `frontend-new/public/data/config/fields-<locale>.yaml`
-  * [ ] Update `env` config: `FRONTEND_SUPPORTED_LOCALES` and `FRONTEND_DEFAULT_LOCALE`
-  * [ ] (Optional) Run the locales consistency test
 
 ## Switching or Using Supported Languages
 
@@ -144,12 +102,12 @@ This section explains how to enable or switch between languages that are already
 ### 1. Backend
 
 * **Language Config:** `BACKEND_LANGUAGE_CONFIG` determines the default backend language and other configurations.
-* **Supported languages:** Listed in the `SUPPORTED_LOCALES` enum in [`backend/app/i18n/types.py`](backend/app/i18n/types.py).
+* **Supported languages:** Listed in the `SUPPORTED_LOCALES` list in [`backend/app/i18n/types.py`](backend/app/i18n/types.py).
 
 ### 2. Frontend
 
 * **Registered languages:** Defined in [`frontend-new/src/i18n/i18n.ts`](frontend-new/src/i18n/i18n.ts) (translation resources) and [`frontend-new/src/i18n/constants.ts`](frontend-new/src/i18n/constants.ts) (`Locale`, `LocalesLabels`, `SupportedLocales`).
-* **Default language:** `FRONTEND_DEFAULT_LOCALE` in `public/data/env.js` sets the default UI language.
+* **Default language:** `FRONTEND_DEFAULT_LOCALE` in `public/data/env.js` sets the default UI language if no user preference is set.
 * **Switching languages at runtime:** Users can select a language from the UI menu. Only locales listed in `FRONTEND_SUPPORTED_LOCALES` are available. To change the default language in the frontend, update `FRONTEND_DEFAULT_LOCALE` in the environment configuration.
 
-> **Note:** Backend and frontend are not automatically synchronized. To ensure a consistent language across the application, configure both layers to support and use the same language.
+> **Note:** Backend and frontend active languages are not automatically synchronized. To ensure a completely consistent language across the application, configure both layers to support and default to the same language.

backend/.env.example

@@ -63,3 +63,7 @@ BACKEND_LANGUAGE_CONFIG='{"default_locale":"en-US","available_locales":[{"locale
 
 # Branding settings
 GLOBAL_PRODUCT_NAME=Compass
+
+# Google Sheets translations
+GOOGLE_SHEETS_CREDENTIALS=keys/credentials-sheets.json
+GOOGLE_SHEET_ID=<GOOGLE_SHEET_ID>

backend/app/i18n/locales/en-GB/messages.json

@@ -62,4 +62,4 @@
       "unseenUnpaid": "What do you think is most important when helping out in the community or caring for others?"
     }
   }
-}
+}

backend/app/i18n/locales/en-US/messages.json

@@ -62,4 +62,4 @@
       "unseenUnpaid": "What do you think is most important when helping out in the community or caring for others?"
     }
   }
-}
+}

backend/scripts/import_translations_from_sheets.py

@@ -0,0 +1,147 @@
+"""
+Script to download the latest translations from the Google Sheet and automatically
+update the frontend and backend JSON language files.
+"""
+
+import json
+import os
+import sys
+from pathlib import Path
+
+from dotenv import load_dotenv
+from google.oauth2 import service_account
+from googleapiclient.discovery import build
+
+from locale_registration import register_locales
+
+# Configuration
+load_dotenv()
+GOOGLE_SHEETS_CREDENTIALS = os.getenv("GOOGLE_SHEETS_CREDENTIALS")
+GOOGLE_SHEET_ID = os.getenv("GOOGLE_SHEET_ID")
+SCOPES = ["https://www.googleapis.com/auth/spreadsheets.readonly"]
+
+PLATFORM_MAPPINGS = {
+    "Frontend": "frontend-new/src/i18n/locales/{locale}/translation.json",
+    "Backend":  "backend/app/i18n/locales/{locale}/messages.json",
+    "Feedback": "frontend-new/src/feedback/overallFeedback/feedbackForm/questions-{locale}.json",
+}
+
+def build_service(credentials_path: str):
+    """Builds a Google Sheets service object."""
+    creds = service_account.Credentials.from_service_account_file(
+        credentials_path, scopes=SCOPES
+    )
+    return build("sheets", "v4", credentials=creds)
+
+def fetch_rows(service, sheet_id: str, range_name: str = "A:ZZZ") -> list[list[str]]:
+    """Fetches all rows from the specified spreadsheet range."""
+    result = service.spreadsheets().values().get(
+        spreadsheetId=sheet_id, range=range_name
+    ).execute()
+    return result.get("values", [])
+
+def set_nested(d: dict, key_path: str, value):
+    """Sets a value in a nested dictionary using a dot-notated key path."""
+    parts = key_path.split(".")
+    for part in parts[:-1]:
+        if part not in d or not isinstance(d[part], dict):
+            d[part] = {}
+        d = d[part]
+    # Format value: restore newlines and handle nulls
+    d[parts[-1]] = value.replace("\\n", "\n") if isinstance(value, str) else value
+
+def _get_translations_data(header: list[str], rows: list[list[str]], locales: list[str]) -> dict:
+    """Groups spreadsheet rows by target (platform, locale)."""
+    all_data = {}
+    for row in rows[1:]:
+        padded = row + [""] * (len(header) - len(row))
+        platform, key_path = padded[0].strip(), padded[1].strip()
+
+        if platform not in PLATFORM_MAPPINGS or not key_path:
+            continue
+
+        for i, locale in enumerate(locales):
+            value = padded[2 + i]
+            # If value is empty, use None so it becomes 'null' in JSON
+            val_to_set = value if value != "" else None
+
+            target = (platform, locale)
+            all_data.setdefault(target, {})[key_path] = val_to_set
+    return all_data
+
+def _sync_dict(existing: dict, incoming: dict) -> dict:
+    """Order-preserving sync of two nested dicts"""
+    result: dict = {}
+
+    for key, old_val in existing.items():
+        if key not in incoming:
+            continue
+        new_val = incoming[key]
+        if isinstance(old_val, dict) and isinstance(new_val, dict):
+            result[key] = _sync_dict(old_val, new_val)
+        else:
+            result[key] = new_val
+
+    for key, new_val in incoming.items():
+        if key not in result:
+            result[key] = new_val
+
+    return result
+
+
+def _write_json_files(root_dir: Path, all_data: dict):
+    """Writes translation data to JSON files using an order-preserving sync"""
+    for (platform, locale), updates in all_data.items():
+        file_path = root_dir / PLATFORM_MAPPINGS[platform].format(locale=locale)
+        file_path.parent.mkdir(parents=True, exist_ok=True)
+
+        # Build the incoming dict from sheet data (dot-notation → nested)
+        incoming: dict = {}
+        for key, val in updates.items():
+            set_nested(incoming, key, val)
+
+        # Load an existing file (empty dict for new locales)
+        existing: dict = {}
+        if file_path.exists():
+            try:
+                existing = json.loads(file_path.read_text(encoding="utf-8") or "{}")
+            except json.JSONDecodeError:
+                pass
+
+        synced = _sync_dict(existing, incoming)
+
+        with open(file_path, "w", encoding="utf-8") as f:
+            json.dump(synced, f, indent=2, ensure_ascii=False)
+            f.write("\n")
+
+def main():
+    if not GOOGLE_SHEETS_CREDENTIALS or not GOOGLE_SHEET_ID:
+        print("ERROR: GOOGLE_SHEETS_CREDENTIALS or GOOGLE_SHEET_ID not set in .env")
+        sys.exit(1)
+
+    root_dir = Path(__file__).resolve().parent.parent.parent
+    service = build_service(GOOGLE_SHEETS_CREDENTIALS)
+    rows = fetch_rows(service, GOOGLE_SHEET_ID)
+
+    if not rows:
+        print("ERROR: Sheet is empty.")
+        sys.exit(1)
+
+    header = rows[0]
+    if len(header) < 3 or header[0] != "Platform" or header[1] != "Key":
+        print("ERROR: Invalid header. Expected ['Platform', 'Key', <locale>, ...]")
+        sys.exit(1)
+
+    locales = header[2:]
+    all_data = _get_translations_data(header, rows, locales)
+    _write_json_files(root_dir, all_data)
+
+    print(f"Import complete. Processed {len(all_data)} translation files.")
+
+    # Auto-register any new locales found in the sheet
+    new_locs = register_locales(root_dir, locales)
+    if new_locs:
+        print(f"Successfully registered {len(new_locs)} new languages: {', '.join(new_locs)}")
+
+if __name__ == "__main__":
+    main()