feat(ci): org-control plane quality pass — security, tests, DRY, observability

.github/forks.yml

@@ -104,9 +104,14 @@ forks:
     install_forward_port: true
 
   - repo: ledoent/OpenUpgrade
-    # Default branch is "ledoent" (= 19.0 + CI customizations). Sync that
-    # alongside the upstream version-pinned branches.
-    branches: ["ledoent", "19.0"]
+    # NOTE: the fork's default branch `ledoent` is a CI overlay
+    # (= 19.0 + lab CI customizations). It is deliberately NOT in
+    # `branches:` — upstream PRs always target OCA's `19.0`, never
+    # the overlay, so merging upstream into `ledoent` is the wrong
+    # direction (would drop the overlay's customizations under a
+    # conflict). The overlay is rebased on top of 19.0 manually
+    # when it drifts too far. Only `19.0` gets auto-synced here.
+    branches: ["19.0"]
     upstream_org: OCA
     upstream_track: "19.0"
     install_forward_port: false

.github/scripts/README.md

@@ -1,68 +1,73 @@
-# Org workflow scripts
+# Org-control scripts
 
-Helpers invoked by workflows in `../workflows/`. Two scripts run daily as a
-single chain inside `fork-sync-and-digest.yml`:
+Helpers invoked by `.github/workflows/fork-sync-and-digest.yml` and
+`.github/workflows/tests.yml`. The repo-root [`../../README.md`](../../README.md)
+has the high-level overview; this file documents internals.
 
-## 1. `fork_sync_digest.py` — sync + digest assembly
+## Three-stage daily pipeline
 
-- Reads `../forks.yml`.
-- For each fork, calls `POST /repos/{repo}/merge-upstream` on every branch
-  in `branches:` (e.g. `["18.0", "19.0"]`). Non-existent branches return
-  HTTP 422 and are marked as skipped, not failed.
-- Queries each unique upstream `(org, repo)` once for PRs with `[MIG]` in
-  the title merged in the last 24h on `upstream_track`.
-- Writes `digest.html`, `digest.subject`, `digest.exit` for the email step.
-- Writes `forks-parsed.json` for the next stage.
+| Stage | Script | Reads | Writes |
+|---|---|---|---|
+| Collect | `fork_sync_digest.py` | `../forks.yml` | `forks-parsed.json`, `sync-results.json`, `mig-buckets.json` |
+| Distribute | `distribute_forward_port.py` | `forks-parsed.json`, `../templates/forward-port.yml` | `forward-port-distribution.json` |
+| Render | `render_digest.py` | `sync-results.json`, `mig-buckets.json`, `forward-port-distribution.json` | `digest.html`, `digest.subject`, `digest.exit` |
 
-## 2. `distribute_forward_port.py` — push `forward-port.yml` to opted-in forks
+The render stage is separate from collect so the distributor's
+outcomes land in the email body — without the split, a 14/14
+distributor failure would only appear in the artifact tarball.
 
-- Reads `forks-parsed.json`.
-- For each fork with `install_forward_port: true`, PUTs
-  `.github/workflows/forward-port.yml` (from `../templates/forward-port.yml`)
-  onto the fork's default branch via the Contents API.
-- Idempotent: no-ops when the file already matches the template.
+## Shared helper
 
-## The forward-port workflow itself
+`_github.py` holds the stdlib-only HTTP wrapper used by both scripts.
+Calls `require_token()` at startup to fail fast with an actionable
+message when `GH_TOKEN` is unset, then `make_headers()` and
+`request()` for the actual API calls. Add retry / rate-limit handling
+here (one place) if it becomes necessary.
 
-Lives at `../templates/forward-port.yml`. Once installed on a fork, label
-any PR with `port:<branch>` (e.g. `port:19.0`) and on merge the
-[`korthout/backport-action`](https://github.com/korthout/backport-action)
-cherry-picks the squash commit onto the named branch and opens a
-follow-up PR. Conflicts are reported in the PR body for manual resolution.
+## Forward-port workflow template
 
-## Running locally
+`../templates/forward-port.yml` is the file the distributor pushes
+to every opted-in fork. It uses `pull_request_target` (deliberate;
+see comment in the file) and SHA-pinned actions.
+
+## Tests
+
+`tests/` covers the parser, the merge-upstream state mapping, and the
+digest rendering. Run with:
 
 ```sh
-export GH_TOKEN=...   # PAT with Contents:write on ledoent/* + public_repo
-python3 .github/scripts/fork_sync_digest.py
-python3 .github/scripts/distribute_forward_port.py
-cat digest.html
+python3 -m pytest .github/scripts/tests/ -v
 ```
 
+CI runs them via `.github/workflows/tests.yml` on every PR that
+touches `.github/scripts/**`, `.github/forks.yml`, or the tests
+workflow itself.
+
 ## Adding a fork
 
-Edit `.github/forks.yml`. Each entry:
+Edit `../forks.yml`. Each entry:
 
 | Key | Purpose |
 |---|---|
 | `repo` | `ledoent/<name>` |
-| `branches` | List of branches to keep synced. Add `20.0` here when OCA cuts it. |
+| `branches` | List of branches to keep synced. Add `"20.0"` here when OCA cuts it. |
 | `upstream_org` | `OCA` for OCA forks, `null` to skip MIG digest |
 | `upstream_track` | Branch on upstream to scan for `[MIG]` PRs (typically the next major) |
 | `install_forward_port` | `true` to push `forward-port.yml` to this fork |
 
+## Running locally
+
+```sh
+export GH_TOKEN=$(gh auth token)   # or a fine-grained PAT
+python3 .github/scripts/fork_sync_digest.py
+python3 .github/scripts/distribute_forward_port.py
+python3 .github/scripts/render_digest.py
+cat digest.subject && open digest.html  # macOS
+```
+
 ## Secrets
 
-Set at the org or `ledoent/.github` repo level:
-
-- `LEDOENT_FORK_SYNC_TOKEN` — fine-grained PAT, scopes:
-  - `Contents: write` on `ledoent/*`
-  - `Metadata: read`
-  - `public_repo` (for searching OCA upstream PRs)
-- `SMTP_SERVER` / `SMTP_USERNAME` / `SMTP_PASSWORD` / `SMTP_TO` / `SMTP_FROM`
-  - `SMTP_USERNAME`: SES SMTP-credential AKID
-  - `SMTP_PASSWORD`: derived from the IAM secret via SES's signing
-    algorithm (NOT the raw IAM secret)
-  - `SMTP_FROM`: full From: header, e.g. `Ledoent CI <ci@ledoweb.com>`
-    on a domain SES has verified (`ledoweb.com` is already verified
-    in account 058264328562, region us-east-1)
+See [`../../README.md`](../../README.md). The single load-bearing
+detail: the PAT's **resource owner must be the `ledoent` org**, not
+a personal account. A PAT issued under a personal account can't see
+ledoent-owned forks even with full permissions.

.github/scripts/_github.py

@@ -0,0 +1,72 @@
+"""Minimal GitHub REST helper shared by the org-control scripts.
+
+Kept dependency-free (stdlib only) because the workflow runs without
+pip and adding a setup-python step for one HTTP call isn't worth it.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import urllib.error
+import urllib.request
+
+
+GH = "https://api.github.com"
+_USER_AGENT_DEFAULT = "ledoent-org-control/1.0"
+
+
+def require_token(var: str = "GH_TOKEN") -> str:
+    """Read a token from env or exit with a useful pointer.
+
+    Cryptic `KeyError: 'GH_TOKEN'` tracebacks waste a workflow run when
+    the secret name is mis-spelled. Print the actionable message and
+    exit non-zero instead.
+    """
+    token = os.environ.get(var)
+    if not token:
+        sys.stderr.write(
+            f"error: {var} is not set in the environment.\n"
+            f"In the workflow, ensure the step has\n"
+            f"    env:\n"
+            f"      {var}: ${{{{ secrets.LEDOENT_FORK_SYNC_TOKEN }}}}\n"
+            f"and that the secret exists at the repo or org level.\n"
+        )
+        sys.exit(2)
+    return token
+
+
+def make_headers(token: str, user_agent: str = _USER_AGENT_DEFAULT) -> dict:
+    return {
+        "Authorization": f"Bearer {token}",
+        "Accept": "application/vnd.github+json",
+        "X-GitHub-Api-Version": "2022-11-28",
+        "User-Agent": user_agent,
+    }
+
+
+def request(
+    method: str,
+    path: str,
+    *,
+    headers: dict,
+    body: dict | None = None,
+    timeout: int = 30,
+) -> tuple[int, dict]:
+    """Single HTTP call returning (status, parsed_json_or_error_dict)."""
+    url = path if path.startswith("http") else f"{GH}{path}"
+    data = json.dumps(body).encode() if body else None
+    req = urllib.request.Request(url, data=data, method=method, headers=headers)
+    if body:
+        req.add_header("Content-Type", "application/json")
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as r:
+            payload = r.read().decode()
+            return r.status, (json.loads(payload) if payload else {})
+    except urllib.error.HTTPError as e:
+        try:
+            payload = json.loads(e.read().decode())
+        except Exception:
+            payload = {"message": str(e)}
+        return e.code, payload

.github/scripts/distribute_forward_port.py

@@ -17,42 +17,19 @@
 
 import base64
 import json
-import os
 import sys
-import urllib.error
-import urllib.request
 from pathlib import Path
 
-GH = "https://api.github.com"
-TOKEN = os.environ["GH_TOKEN"]
-HEADERS = {
-    "Authorization": f"Bearer {TOKEN}",
-    "Accept": "application/vnd.github+json",
-    "X-GitHub-Api-Version": "2022-11-28",
-    "User-Agent": "ledoent-fp-distributor/1.0",
-}
+from _github import make_headers, request, require_token
+
+HEADERS = make_headers(require_token(), user_agent="ledoent-fp-distributor/1.0")
 
 TEMPLATE = Path(".github/templates/forward-port.yml")
 DEST_PATH = ".github/workflows/forward-port.yml"
 
 
 def gh(method: str, path: str, body: dict | None = None) -> tuple[int, dict]:
-    data = json.dumps(body).encode() if body else None
-    req = urllib.request.Request(
-        f"{GH}{path}", data=data, method=method, headers=HEADERS
-    )
-    if body:
-        req.add_header("Content-Type", "application/json")
-    try:
-        with urllib.request.urlopen(req, timeout=30) as r:
-            payload = r.read().decode()
-            return r.status, (json.loads(payload) if payload else {})
-    except urllib.error.HTTPError as e:
-        try:
-            payload = json.loads(e.read().decode())
-        except Exception:
-            payload = {"message": str(e)}
-        return e.code, payload
+    return request(method, path, headers=HEADERS, body=body)
 
 
 def push_workflow(repo: str, content_bytes: bytes) -> tuple[str, str]: