From 9a500b165fbb470134f5ace9554e06279c6d9d2a Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 09:50:21 -0700 Subject: [PATCH 01/17] Add design spec: run AAQ scraper as ArgoCD CronJob on workloads cluster Egress-IP-first scaffolding to move the scraper off GitHub Actions (blocked by the Fastly bot-challenge, #28) onto the workloads EKS cluster, sharing bitergia's static NAT egress IP for the Mozilla allowlist ask (#27). Plain-HTTP image, CSVs stay in git, fine-grained PAT for pushes. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...13-k8s-argocd-scraper-deployment-design.md | 236 ++++++++++++++++++ 1 file changed, 236 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md diff --git a/docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md b/docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md new file mode 100644 index 0000000..17e1be1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md @@ -0,0 +1,236 @@ + + +# Design: run the AAQ scraper as an ArgoCD CronJob on the workloads EKS cluster + +**Date:** 2026-07-13 +**Status:** approved (brainstorming) — pending implementation plan +**Related:** #27 (Mozilla API-access / IP-allowlist request), #28 (move off GitHub +Actions to a self-hosted runner / static egress IP), github-action-thunderbird-aaq#34, +bitergia-deploy#50. + +## Problem + +As of ~2026-07-09 the SUMO `/api/2/` API sits behind a Fastly bot-challenge that +fingerprints the **automated browser itself** — it blocks headless *and* headed +Playwright/real-Chrome, so the "drive a real browser past the challenge" approach +that resolved #34 no longer works, and there is **no path forward on GitHub +Actions** (its shared, rotating egress IPs cannot be allowlisted). The agreed fix +(#27/#28) is: run from an environment with a **single, stable egress IP**, hand +that IP to Mozilla to allowlist at the edge, and then call the API directly over +**plain HTTP** (no browser). + +This spec covers the **egress-IP-first scaffolding**: build the full ArgoCD +CronJob deployment now so it (1) produces the known static egress IP we hand +Mozilla, and (2) starts scraping successfully the moment that IP is allowlisted — +with no further code change required. + +## Scope + +**In scope** + +- A new `aaq-scraper` CronJob app deployed to the **workloads EKS cluster** + (`mzla-eks-workloads01`, eu-central-1, ArgoCD project `workloads`) via the + existing app-of-apps, following the `bamboohr-cal-sync` / + `thundermail-ticket-spike-monitor` pattern in `platform-infrastructure`. +- A lean, browser-free container image built from the `aaq-scraper` repo. +- A modest `sumo.py` refactor: **drop Playwright, fetch over a plain HTTP + client**, preserving the existing `fetch_json` retry/deferral contract. +- Pulumi additions: ECR repo + push-only GitHub-OIDC role; **confirm the + workloads cluster egresses via a stable NAT-gateway EIP** (the IP we hand + Mozilla). +- Secrets via External Secrets Operator (ESO) → AWS Secrets Manager: a + **fine-grained GitHub PAT** the pod uses to commit/push CSVs back to + `aaq-scraper`. +- CronJob-failure alerting via a `VMRule` over existing kube-state-metrics, + shipped **suppressed** (the job fails hourly until allowlisting) with a note to + un-suppress at cutover. + +**Out of scope (tracked separately)** + +- Actually obtaining the Mozilla allowlist — that is #27, an upstream ask. +- Migrating `schema-check.yml` to k8s. It also queries the live API and will + break the same way, so **this effort disables that workflow** (with a + reviewer note in the file) and leaves its migration for a follow-up. +- `kitsune-api-watch.yml` — hits only the GitHub API, unaffected, stays in + Actions. +- Disabling `scrape.yml` — stays running until the k8s job is verified green + post-allowlist, then disabled in a follow-up (no flag-day). + +## Approach + +### 1. Placement & the egress IP + +Deploy to the **workloads cluster**, namespace **`aaq-scraper`** (CreateNamespace +via the ArgoCD Application). This is deliberate: **bitergia-deploy** — the other +Thunderbird consumer of SUMO `/api/2/` — already runs on this cluster, so its +NAT-gateway egress IP is shared. Co-locating means Mozilla allowlists **one IP +for both consumers** (this + bitergia #50), which is exactly the #27 ask. + +**Critical early verification:** confirm (in the Pulumi networking for the +workloads cluster) that outbound traffic leaves via a NAT gateway with a **fixed +Elastic IP**, not a per-AZ ephemeral address. If it is already a stable EIP, that +IP is what #27 hands Mozilla — no new infra. If egress is ephemeral, add/confirm a +NAT EIP. This fact gates the entire effort's value and should be checked first in +the implementation plan. + +### 2. Container image & the `sumo.py` refactor + +- Base: `python:3.12-slim` (or distroless-python), **`linux/arm64`**, run as + nonroot. **No Chromium** — the image is small and browser-free. +- Dependency change: replace `playwright` with a plain HTTP client (`httpx`). +- **`sumo.py`:** replace the Playwright `SumoBrowser` (launch Chromium → acquire + challenge cookies → in-page `page.evaluate(fetch)`) with a direct HTTP client + that calls `API_BASE` endpoints. **Preserve the existing `fetch_json` + contract** so nothing downstream changes: + - exponential backoff on HTTP 429 (honour `Retry-After`), 5xx, and a + 200-but-non-JSON hiccup; + - `RateLimitDeferral` when a 429 wait exceeds `max_429_wait_s`; + - `ChallengeError` on a persistent 200-but-HTML (this is how the job will + "fail loudly" every hour until the IP is allowlisted — the expected + pre-allowlist state). + - `DEFERRAL_EXIT_CODE` (75) and atomic CSV writes unchanged. +- `run_refresh.py`, `scrape_questions.py`, `scrape_answers.py`, + `find_updated_days.py`, CSV flattening/escaping/redaction, and determinism are + **unchanged**. The `SumoBrowser(headless=…)` constructor call sites adapt to the + new client; the `--headless` CLI flag becomes a no-op (kept for compatibility) + or is removed — decided in the plan. + +### 3. Image build & release (aaq-scraper repo) + +- New `.github/workflows/aaq-scraper-image.yml` (copy the + `thundermail-ticket-spike-monitor-image.yml` external-repo-trust variant): + build on push to `main` touching scraper source / `Dockerfile` / deps; auth via + **GitHub OIDC** assuming a push-only role; `amazon-ecr-login` + + `docker/build-push-action` (`platforms: linux/arm64`); push immutable + `git-` tags to the shared ECR + `826971876779.dkr.ecr.us-east-1.amazonaws.com/aaq-scraper`. +- **Deploys stay human-gated:** bump the `image:` tag in + `services/aaq-scraper/deploy/cronjob.yaml` via a `platform-infrastructure` PR; + ArgoCD syncs on merge. No image-updater. +- One-time setup: GitHub Environment `image-aaq-scraper` (main-only) + repo var + `IMAGE_PUSH_ROLE_ARN` in the aaq-scraper repo. + +### 4. Kubernetes manifests (`platform-infrastructure/services/aaq-scraper/deploy/`) + +Raw YAML (no Kustomize/Helm), ordered by `argocd.argoproj.io/sync-wave`: + +- `serviceaccount.yaml` (wave 0) — dedicated SA, `automountServiceAccountToken: + false`, no IRSA (needs only HTTPS egress + ESO-materialized secrets). +- `configmap.yaml` (wave 0) — non-secret env (target repo URL/branch, product + slugs, refresh flags). +- `externalsecret.yaml` (wave 0) — `external-secrets.io/v1`, `ClusterSecretStore` + `aws-secrets-manager`, target Secret `aaq-scraper` mapping SM key + `mzla/shared-services/aaq-scraper` → env-named keys (see §5). +- `cronjob.yaml` (wave 1): + - `schedule: "0 * * * *"`, `timeZone: "Etc/UTC"`. + - `concurrencyPolicy: Forbid` (equivalent intent to today's + `cancel-in-progress`), `startingDeadlineSeconds`, `successfulJobsHistoryLimit` + / `failedJobsHistoryLimit: 3`. + - `jobTemplate`: `backoffLimit`, `activeDeadlineSeconds` ~3300 (≈ the current + 55-min GHA hard cap; keep it above the 40-min soft deadline), + `ttlSecondsAfterFinished`, `restartPolicy: OnFailure`. + - Hardened `securityContext` (`runAsNonRoot`, `readOnlyRootFilesystem`, + `allowPrivilegeEscalation: false`, `capabilities: drop [ALL]`, + `seccompProfile: RuntimeDefault`). Note: git clone/commit needs a **writable + workdir** — mount an `emptyDir` for the checkout since the root FS is + read-only. + - `envFrom` the ConfigMap + the ESO Secret. + - Command runs the entrypoint (see §6) which wraps + `run_refresh.py --soft-deadline 40 --max-429-wait 120` — same flags the + hourly workflow uses today (exit-75 deferral semantics preserved). +- `vmrule.yaml` (wave 1) — see §7. + +### 5. Secrets (ESO → AWS Secrets Manager) + +- SM secret `mzla/shared-services/aaq-scraper` holding a **fine-grained GitHub + PAT** scoped to the `aaq-scraper` repo with **Contents: read/write** only, + materialized to an env var / git credential in the pod. +- Created manually in Secrets Manager before first deploy (documented in the + `externalsecret.yaml` header, per repo convention). +- The `ExternalSecret` Application manifest carries the standard ESO + `ignoreDifferences` block (copied verbatim from an existing app). +- PAT expiry is an operational risk: document a rotation reminder; the job + simply fails to push (loudly) if it expires — no silent data loss. + +### 6. Output & state persistence — stateless pod, CSVs in git + +The pod reproduces the GitHub Actions checkout→run→commit→push flow itself, +preserving the "**CSVs tracked in git**" contract that downstream Ruby reports +depend on. An entrypoint script (baked into the image): + +1. `git clone` (or shallow fetch) `aaq-scraper` `main` into the emptyDir workdir, + using the PAT. +2. Read the high-water mark from **`.refresh-hwm` committed in the repo**. +3. Run `run_refresh.py --soft-deadline 40 --max-429-wait 120`. +4. `git add 20*/ .refresh-hwm`; if nothing changed, exit 0 (the deterministic + "nothing to commit" path). +5. `git commit -m "Hourly refresh "` and push with the **same + rebase-onto-latest-main retry/backoff loop** the workflow uses today + (main receives concurrent pushes from manual backfills). + +**High-water-mark change:** `.refresh-hwm` moves from the (now-unavailable) +Actions cache into the **repo** (un-gitignored, committed each active run) — +CLAUDE.md's documented "make it durable" option. The 26h `--lookback-hours` +fallback in `run_refresh.py` still covers a missing/stale mark, so a first run or +a wiped mark self-heals. + +### 7. Alerting + +- `vmrule.yaml` (VictoriaMetrics `VMRule`) alerting on **CronJob failure / + last-success staleness** using existing **kube-state-metrics** + (`kube_job_status_failed`, last-successful-run age) — **no app-side metrics + push**, so the scraper needs no instrumentation change. +- Because the job **fails every hour until the IP is allowlisted**, ship the + alert **suppressed/inhibited** with an inline comment, and un-suppress it at + cutover (tracked in the cutover follow-up). + +### 8. Cutover (follow-up, not this PR) + +`scrape.yml` keeps running until the k8s CronJob is verified green after +allowlisting. Then, in a separate PR: disable `scrape.yml`, un-suppress the +VMRule, and (optionally) migrate `schema-check.yml`. + +## Repos & files touched + +**aaq-scraper repo** + +- `sumo.py` — Playwright → HTTP client refactor (preserve `fetch_json` contract). +- `pyproject.toml` / `uv.lock` — drop `playwright`, add `httpx`. +- `Dockerfile` — new, arm64, nonroot, browser-free. +- entrypoint script (clone/run/commit/push) — new. +- `.github/workflows/aaq-scraper-image.yml` — new (OIDC ECR push). +- `.github/workflows/schema-check.yml` — **disabled** with a reviewer note. +- `.gitignore` — un-ignore `.refresh-hwm`; commit the current mark. +- `CLAUDE.md` / `README.md` — document the k8s deployment and the state-in-repo + change. + +**platform-infrastructure repo** + +- `services/aaq-scraper/deploy/{serviceaccount,configmap,externalsecret,cronjob,vmrule}.yaml` + — new. +- `argocd/workloads/apps/aaq-scraper.yaml` — new ArgoCD Application (project + `workloads`, `path: services/aaq-scraper/deploy`, standard ESO + `ignoreDifferences` + `syncPolicy`, `CreateNamespace=true`). +- `pulumi/environments/mzla-shared-services/config.prod.yaml` — add + `ecr.repositories.aaq-scraper` and `github_oidc.roles.aaq-scraper-image` + (trusting the `thunderbird/aaq-scraper` repo). +- Pulumi networking — **verify/ensure** the workloads NAT-gateway stable EIP. + +## Risks & open items + +- **Egress-IP assumption:** the whole effort's value depends on the workloads + cluster having a stable, allowlistable NAT EIP. Verify first. +- **Post-allowlist unknown:** we don't yet know whether Mozilla will grant an IP + allowlist vs. an API token, or whether plain HTTP from the allowlisted IP fully + bypasses the challenge. The design accommodates either (a token would become + another ESO secret + a header in the HTTP client). +- **PAT lifecycle:** expiry causes loud push failures (acceptable — no silent + loss), but needs a rotation runbook. +- **Read-only root FS + git:** requires a writable emptyDir workdir; ensure the + git config/credential helper writes there, not `$HOME`. +- **Determinism preserved:** the "no changes → nothing to commit" path must keep + working so idle hours don't create empty commits. From 6b958e8e10a67763c51cc8dbd9bc64c4be27a261 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 10:46:53 -0700 Subject: [PATCH 02/17] Add implementation plan for the ArgoCD CronJob deployment Two-phase, TDD where testable: Phase A (this repo) swaps sumo.py's Playwright engine for httpx behind the frozen SumoBrowser API, packages a browser-free arm64 image whose entrypoint clones/refreshes/commits, and builds it in CI; Phase B (platform-infrastructure) adds the Pulumi ECR+OIDC, the deploy manifests, and the ArgoCD app. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...026-07-13-k8s-argocd-scraper-deployment.md | 1146 +++++++++++++++++ 1 file changed, 1146 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md diff --git a/docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md b/docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md new file mode 100644 index 0000000..3b10c54 --- /dev/null +++ b/docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md @@ -0,0 +1,1146 @@ + + +# AAQ Scraper → ArgoCD CronJob Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Run the SUMO AAQ scraper as an hourly ArgoCD CronJob on the workloads EKS cluster with a stable, allowlistable egress IP, replacing the (now-blocked) GitHub Actions browser approach. + +**Architecture:** Two coordinated changes. **Phase A** (`aaq-scraper` repo, PR #44): swap `sumo.py`'s Playwright engine for a plain `httpx` client behind the *unchanged* `SumoBrowser`/`fetch_json`/`paginate` public API; package a lean, browser-free arm64 image whose entrypoint clones the repo, runs the existing refresh, and commits CSVs back to git; build that image in CI via OIDC → shared ECR. **Phase B** (`platform-infrastructure` repo, separate PR): Pulumi ECR repo + push role, the `services/aaq-scraper/deploy/` manifests, and the ArgoCD Application targeting the workloads cluster. The design doc is `docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md`. + +**Tech Stack:** Python 3.10+ (`httpx`, `pytest`), Docker (`python:3.12-slim`, `linux/arm64`), Kubernetes (`batch/v1` CronJob), ArgoCD (raw-YAML app-of-apps), External Secrets Operator → AWS Secrets Manager, Pulumi (Python), GitHub Actions (OIDC → ECR). + +## Global Constraints + +- **License header on every new file** — MPL-2.0 block; Python after the shebang, YAML/Dockerfile/shell at the top. Copy verbatim from any existing file. +- **`SumoBrowser` public API is frozen.** The class name, constructor keyword args (`headless`, `settle_ms`, `max_attempts`, `backoff_base`, `retry_jitter_min_s`, `retry_jitter_max_s`, `max_429_wait_s`), context-manager protocol, and the `fetch_json(url)` / `paginate(first_url, on_page=None)` methods must keep identical signatures and behavior. Consumers (`scrape_questions.py`, `scrape_answers.py`, `find_updated_days.py`, `check_schema.py`, `run_refresh.py`) must not need edits. +- **`fetch_json` / `paginate` bodies stay unchanged** — only `__init__`, `__enter__`, `__exit__`, `_raw_fetch` are rewritten. `_raw_fetch(url)` must keep returning exactly `{"status": int, "json": dict|None, "snippet": str, "retry_after": str|None}`. +- **Exception contract preserved:** `ChallengeError` on exhausted 200-but-HTML; `RateLimitDeferral` when a 429 wait exceeds `max_429_wait_s`; `RuntimeError` on non-retryable 4xx; `DEFERRAL_EXIT_CODE = 75`. +- **Determinism preserved:** re-running a day yields byte-identical CSVs; the "nothing changed → no commit" path must remain. +- **Package manager is `uv`** — `uv sync` / `uv run`, never pip/venv. +- **Image:** `linux/arm64`, run as nonroot, shared ECR `826971876779.dkr.ecr.us-east-1.amazonaws.com/aaq-scraper`, immutable `git-` tags. +- **Deploys are human-gated:** CI only pushes the image; releasing means bumping the `image:` tag in `services/aaq-scraper/deploy/cronjob.yaml` via a `platform-infrastructure` PR. +- **Cluster/namespace:** workloads cluster `mzla-eks-workloads01` (eu-central-1), ArgoCD project `workloads`, namespace `aaq-scraper` (CreateNamespace via the Application). + +--- + +# Phase A — `aaq-scraper` repo (PR #44) + +Executes in this checkout (`/home/aatchison/src/tb/aaq-scraper`), branch `k8s-argocd-scraper-deployment`. + +## Task A1: Lock the `fetch_json` contract with tests (safety net) + +These regression tests stub `_raw_fetch`, so they pass against the **current Playwright code** and must keep passing after the httpx swap. They are the guardrail for Task A2. + +**Files:** +- Create: `tests/__init__.py` (empty) +- Create: `tests/test_fetch_json.py` +- Modify: `pyproject.toml` (add a dev dependency group with `pytest`) + +**Interfaces:** +- Consumes: `sumo.SumoBrowser`, `sumo.ChallengeError`, `sumo.RateLimitDeferral` (existing). +- Produces: nothing consumed by later tasks (test-only), but establishes the behaviors Task A2 must preserve. + +- [ ] **Step 1: Add the pytest dev dependency** + +In `pyproject.toml`, after the `dependencies = [...]` array, add: + +```toml +[dependency-groups] +dev = [ + "pytest>=8", +] +``` + +- [ ] **Step 2: Sync so pytest is available** + +Run: `uv sync` +Expected: resolves and installs `pytest` into the environment. + +- [ ] **Step 3: Write the contract tests** + +Create `tests/__init__.py` (empty file with just the MPL header as a comment) and `tests/test_fetch_json.py`: + +```python +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +"""Contract/regression tests for SumoBrowser.fetch_json. + +These stub _raw_fetch (no browser, no network), so they pin the retry / +deferral / challenge behavior independently of the transport. They must pass +both before and after the Playwright->httpx refactor (Task A2).""" +import pytest + +import sumo + + +def _resp(status, json=None, snippet="", retry_after=None): + return {"status": status, "json": json, "snippet": snippet, + "retry_after": retry_after} + + +def _browser(monkeypatch, responses, **kwargs): + """A SumoBrowser whose _raw_fetch yields the given canned responses in + order, with sleeping/jitter neutralized so tests run instantly.""" + kwargs.setdefault("max_attempts", 5) + kwargs.setdefault("backoff_base", 2.0) + kwargs.setdefault("retry_jitter_min_s", 0) + kwargs.setdefault("retry_jitter_max_s", 0) + sb = sumo.SumoBrowser(**kwargs) + seq = iter(responses) + monkeypatch.setattr(sb, "_raw_fetch", lambda url: next(seq)) + monkeypatch.setattr(sumo.time, "sleep", lambda _s: None) + monkeypatch.setattr(sumo.random, "uniform", lambda _a, _b: 0.0) + return sb + + +def test_returns_json_on_200(monkeypatch): + sb = _browser(monkeypatch, [_resp(200, {"count": 3})]) + assert sb.fetch_json("u") == {"count": 3} + + +def test_non_retryable_4xx_raises_plain_runtimeerror(monkeypatch): + sb = _browser(monkeypatch, [_resp(404, snippet="nope")]) + with pytest.raises(RuntimeError) as ei: + sb.fetch_json("u") + assert not isinstance(ei.value, sumo.ChallengeError) + assert not isinstance(ei.value, sumo.RateLimitDeferral) + + +def test_5xx_retried_then_success(monkeypatch): + sb = _browser(monkeypatch, [_resp(503), _resp(200, {"ok": 1})]) + assert sb.fetch_json("u") == {"ok": 1} + + +def test_exhausted_200_non_json_raises_challenge_error(monkeypatch): + sb = _browser(monkeypatch, [_resp(200, None, "challenge")] * 5) + with pytest.raises(sumo.ChallengeError): + sb.fetch_json("u") + + +def test_429_over_threshold_defers(monkeypatch): + sb = _browser(monkeypatch, [_resp(429, retry_after="600")], + max_429_wait_s=120) + with pytest.raises(sumo.RateLimitDeferral): + sb.fetch_json("u") + + +def test_429_under_threshold_retries_then_succeeds(monkeypatch): + sb = _browser(monkeypatch, + [_resp(429, retry_after="10"), _resp(200, {"ok": 1})], + max_429_wait_s=1000) + assert sb.fetch_json("u") == {"ok": 1} +``` + +- [ ] **Step 4: Run the tests against the current (Playwright) code** + +Run: `uv run pytest tests/test_fetch_json.py -v` +Expected: **6 passed** — `_raw_fetch` is stubbed, so no browser launches. If any fail, the stub/monkeypatch is wrong; fix before touching `sumo.py`. + +- [ ] **Step 5: Commit** + +```bash +git add pyproject.toml uv.lock tests/__init__.py tests/test_fetch_json.py +git commit -m "test: pin SumoBrowser.fetch_json retry/deferral/challenge contract" +``` + +## Task A2: Swap the `sumo.py` engine Playwright → httpx + +Rewrite only `__init__`, `__enter__`, `__exit__`, `_raw_fetch`; leave `fetch_json` and `paginate` untouched. The Task A1 tests are the pass/fail gate, plus one new test for the httpx mapping. + +**Files:** +- Modify: `sumo.py:13-18` (imports), `sumo.py:75-168` (class through `_raw_fetch`) +- Modify: `pyproject.toml` (`dependencies`: drop `playwright`, add `httpx`) +- Modify: `tests/test_fetch_json.py` (append the `_raw_fetch` mapping test) + +**Interfaces:** +- Consumes: `sumo.API_BASE`, `sumo.HOME_URL`, `sumo.USER_AGENT` (existing module constants, keep them). +- Produces: `SumoBrowser` with the frozen public API (see Global Constraints); `_raw_fetch(url) -> {"status","json","snippet","retry_after"}`. + +- [ ] **Step 1: Add the httpx mapping test (fails until the refactor lands)** + +Append to `tests/test_fetch_json.py`: + +```python +def test_raw_fetch_maps_httpx_response(): + """_raw_fetch turns an httpx response into the dict fetch_json expects.""" + import httpx + + def handler(request): + return httpx.Response(429, headers={"retry-after": "30"}, + text='{"a": 1}') + + sb = sumo.SumoBrowser() + sb._client = httpx.Client(transport=httpx.MockTransport(handler)) + out = sb._raw_fetch("https://support.mozilla.org/api/2/question/") + assert out == {"status": 429, "json": {"a": 1}, + "snippet": '{"a": 1}', "retry_after": "30"} + + +def test_raw_fetch_non_json_body_sets_json_none(): + import httpx + + def handler(request): + return httpx.Response(200, text="challenge") + + sb = sumo.SumoBrowser() + sb._client = httpx.Client(transport=httpx.MockTransport(handler)) + out = sb._raw_fetch("https://support.mozilla.org/api/2/question/") + assert out["status"] == 200 and out["json"] is None + assert out["snippet"] == "challenge" +``` + +- [ ] **Step 2: Run to confirm the new tests fail** + +Run: `uv run pytest tests/test_fetch_json.py -k raw_fetch -v` +Expected: FAIL — current `SumoBrowser` has no `_client` attribute / `_raw_fetch` calls `self._page`. + +- [ ] **Step 3: Swap the dependency** + +In `pyproject.toml`, change the `dependencies` array from: + +```toml +dependencies = [ + "playwright==1.49.1", +] +``` +to: +```toml +dependencies = [ + "httpx==0.28.1", +] +``` + +- [ ] **Step 4: Rewrite the imports (`sumo.py` lines 13-18)** + +Replace: +```python +import csv +import random +import sys +import time + +from playwright.sync_api import sync_playwright, TimeoutError as PWTimeoutError +``` +with: +```python +import csv +import json +import random +import sys +import time + +import httpx +``` + +Also update the module docstring (lines 4-11) to describe the HTTP client instead of the browser: +```python +""" +Shared SUMO (support.mozilla.org) API client. + +Historically drove a real Chromium (Playwright) to pass the Fastly JS/WAF +challenge; that path is now fingerprinted and blocked (issue #28), so this uses +a plain HTTP client (httpx). Once our egress IP is allowlisted by Mozilla +(issue #27) the API is reachable directly. The public API (SumoBrowser, +fetch_json, paginate) is unchanged so callers did not have to change. +""" +``` + +- [ ] **Step 5: Rewrite the class from `__init__` through `_raw_fetch`** + +Replace `sumo.py` lines 75-168 (the class docstring, `__init__`, `__enter__`, `__exit__`, `_raw_fetch`) with: + +```python +class SumoBrowser: + """A SUMO API HTTP session. + + Named `SumoBrowser` for backwards compatibility with existing call sites; + it no longer drives a browser. Use as a context manager: + + with SumoBrowser(headless=True) as sumo: + data = sumo.fetch_json(sumo_url) + for page in sumo.paginate(first_url): + ... + """ + + def __init__(self, headless=False, settle_ms=6000, + max_attempts=5, backoff_base=2.0, + retry_jitter_min_s=60, retry_jitter_max_s=300, + max_429_wait_s=None): + # `headless` and `settle_ms` are accepted for backwards-compatible call + # sites but ignored: there is no browser to run headed/headless and + # nothing to "settle" for a plain HTTP client. + self.max_attempts = max_attempts + self.backoff_base = backoff_base + # If set, a 429 whose required wait (Retry-After / backoff) exceeds this + # many seconds raises RateLimitDeferral instead of sleeping, so a caller + # can defer the work rather than burn the run on one long window. None = + # honour the wait in full (original behaviour). + self.max_429_wait_s = max_429_wait_s + # Extra random pause added on top of a 429 wait (default 1-5 min) so we + # always retry strictly AFTER SUMO's Retry-After window and desync retries. + self.retry_jitter_min_s = retry_jitter_min_s + self.retry_jitter_max_s = retry_jitter_max_s + self._client = None + + def __enter__(self): + # A persistent client keeps a cookie jar across calls, so any edge + # cookie handed out on the first request is reused (mirrors the old + # browser context's cookie reuse). follow_redirects matches a browser. + self._client = httpx.Client( + headers={ + "User-Agent": USER_AGENT, + "Accept-Language": "en-US,en;q=0.9", + }, + timeout=60.0, + follow_redirects=True, + ) + # Warm up by loading the home page once so any cookie is captured before + # the API call. Best-effort: a failure here is not fatal — a genuine + # block surfaces later as ChallengeError from fetch_json. + try: + self._client.get(HOME_URL) + except httpx.HTTPError as e: + print(f"home warm-up failed (continuing): {e}", + file=sys.stderr, flush=True) + return self + + def __exit__(self, exc_type, exc, tb): + if self._client is not None: + self._client.close() + return False + + def _raw_fetch(self, url): + """Do one HTTP GET; return {status, json, snippet, retry_after}. + + Parses the body as JSON the same way the old in-page fetch did (json is + None when the body is not valid JSON — e.g. an HTML challenge page).""" + res = self._client.get(url, headers={"Accept": "application/json"}) + text = res.text + try: + body = json.loads(text) + except ValueError: + body = None + return { + "status": res.status_code, + "json": body, + "snippet": text[:300], + "retry_after": res.headers.get("retry-after"), + } +``` + +- [ ] **Step 6: Regenerate the lockfile** + +Run: `uv sync` +Expected: `playwright` removed, `httpx` added; environment resolves. + +- [ ] **Step 7: Run the full test file — old contract + new mapping** + +Run: `uv run pytest tests/test_fetch_json.py -v` +Expected: **8 passed** (the 6 contract tests from A1 still green, plus the 2 mapping tests). + +- [ ] **Step 8: Verify consumers still import and parse args (no signature drift)** + +Run: +```bash +uv run python -c "import scrape_questions, scrape_answers, find_updated_days, check_schema, run_refresh; print('imports ok')" +uv run python scrape_questions.py --help >/dev/null && echo "questions --help ok" +``` +Expected: `imports ok` and `questions --help ok` (the frozen API means no consumer edits were needed). + +- [ ] **Step 9: Commit** + +```bash +git add sumo.py pyproject.toml uv.lock tests/test_fetch_json.py +git commit -m "refactor(sumo): replace Playwright with httpx behind the frozen SumoBrowser API" +``` + +## Task A3: Confirm the HTTP path reaches the edge (expected pre-allowlist block) + +No new code — a manual verification that the refactor talks to the real API and correctly *classifies the current block*, which is the honest end-to-end check until the IP is allowlisted. + +**Files:** none. + +- [ ] **Step 1: Run a single-day scrape against the live API** + +Run: `uv run python scrape_questions.py 2026 7 1 2026 7 1 2>&1 | tail -20` +Expected (today, pre-allowlist): it fails, and the failure is a **`ChallengeError`** ("200 with HTML" / "browser may not have passed the challenge") — i.e. the HTTP client reached the Fastly edge and got the challenge page, exactly as designed. It must NOT be a Playwright/import error. Record the outcome in the PR description as the known pre-allowlist state. + +- [ ] **Step 2: No commit** (verification only). + +## Task A4: Browser-free container image + +`python:3.12-slim` (not distroless) because the entrypoint (Task A5) needs `git` and a shell at runtime. + +**Files:** +- Create: `Dockerfile` +- Create: `.dockerignore` + +**Interfaces:** +- Produces: an image whose default entrypoint is `deploy/entrypoint.sh` (created in Task A5). This task builds the image with a placeholder CMD and Task A5 wires the entrypoint; to keep tasks independently testable, define the Dockerfile to copy `deploy/entrypoint.sh` now and have Task A5 create the file — so build this image only after A5, OR temporarily set `CMD ["python", "-c", "import sumo; print('ok')"]`. Use the temporary CMD here; A5 flips it to the entrypoint. + +- [ ] **Step 1: Write the Dockerfile** + +Create `Dockerfile`: +```dockerfile +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. + +# Browser-free image: the old Playwright/Chromium approach is dead (issue #28); +# this runs the plain-httpx scraper. git + a shell are needed at runtime because +# the entrypoint clones aaq-scraper, runs the refresh, and commits CSVs back. +FROM python:3.12-slim + +# uv for dependency management (matches local/CI: uv sync / uv run). +COPY --from=ghcr.io/astral-sh/uv:0.5.11 /uv /usr/local/bin/uv + +RUN apt-get update \ + && apt-get install -y --no-install-recommends git ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app +COPY pyproject.toml uv.lock ./ +RUN uv sync --frozen --no-dev + +COPY . . + +# Non-root. The clone/commit workdir is a writable emptyDir mounted at runtime +# (the root FS is read-only in the pod), so this user only needs to read /app. +RUN useradd --create-home --uid 65532 appuser +USER 65532 + +CMD ["python", "-c", "import sumo; print('image ok')"] +``` + +- [ ] **Step 2: Write `.dockerignore`** + +Create `.dockerignore`: +``` +.git +.venv +__pycache__ +*.pyc +20*/ +docs/ +backfill-reports/ +tests/ +``` +(The data dirs `20*/` are excluded from the image — the pod clones fresh data at runtime; baking hundreds of MB of CSVs into the image is wasteful.) + +- [ ] **Step 3: Build for arm64** + +Run: `docker buildx build --platform linux/arm64 -t aaq-scraper:dev --load .` +Expected: build succeeds. (On an arm64 host `--load` works directly; on amd64 it still builds via emulation.) + +- [ ] **Step 4: Smoke-test the image** + +Run: `docker run --rm aaq-scraper:dev` +Expected: prints `image ok` (imports resolve, no Playwright). + +- [ ] **Step 5: Commit** + +```bash +git add Dockerfile .dockerignore +git commit -m "build: browser-free arm64 image for the scraper CronJob" +``` + +## Task A5: Container entrypoint — clone, refresh, commit, push + +Reproduces the GitHub Actions `checkout → run_refresh → commit → push (rebase-retry)` flow inside the pod, using a PAT. Includes an integration test against a **local bare repo** (no network). + +**Files:** +- Create: `deploy/entrypoint.sh` +- Create: `tests/test_entrypoint.py` +- Modify: `Dockerfile` (flip CMD to the entrypoint) + +**Interfaces:** +- Consumes (env vars): `GIT_REPO_URL` (https URL of aaq-scraper), `GIT_BRANCH` (default `main`), `GITHUB_TOKEN` (fine-grained PAT, Contents:rw), `GIT_AUTHOR_NAME`, `GIT_AUTHOR_EMAIL`, `REFRESH_ARGS` (default `--soft-deadline 40 --max-429-wait 120`), and optional `REFRESH_CMD` (override the scrape command; used by tests). +- Produces: a commit `Hourly refresh ` on `GIT_BRANCH` when `20*/` or `.refresh-hwm` changed; clean exit 0 with no commit when nothing changed; exit 75 propagated as "deferred". + +- [ ] **Step 1: Write the entrypoint** + +Create `deploy/entrypoint.sh`: +```bash +#!/usr/bin/env bash +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +# +# Pod entrypoint: clone aaq-scraper, run the incremental refresh, commit changed +# CSVs (and the high-water mark) back to the branch. Mirrors the logic that used +# to live in .github/workflows/scrape.yml (checkout + run_refresh + commit with +# rebase-onto-latest-main retry). The pod is stateless; the repo IS the state. +set -euo pipefail + +: "${GIT_REPO_URL:?set GIT_REPO_URL}" +: "${GITHUB_TOKEN:?set GITHUB_TOKEN}" +GIT_BRANCH="${GIT_BRANCH:-main}" +GIT_AUTHOR_NAME="${GIT_AUTHOR_NAME:-aaq-scraper-bot}" +GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-aaq-scraper-bot@thunderbird.net}" +REFRESH_ARGS="${REFRESH_ARGS:---soft-deadline 40 --max-429-wait 120}" + +WORKDIR="$(mktemp -d)" +# Embed the token in the clone URL (https://x-access-token:TOKEN@github.com/...). +AUTH_URL="$(printf '%s' "$GIT_REPO_URL" | sed -E "s#https://#https://x-access-token:${GITHUB_TOKEN}@#")" + +git clone --branch "$GIT_BRANCH" "$AUTH_URL" "$WORKDIR/repo" +cd "$WORKDIR/repo" +git config user.name "$GIT_AUTHOR_NAME" +git config user.email "$GIT_AUTHOR_EMAIL" + +# Run the refresh. REFRESH_CMD lets tests inject a fake; default is the real one. +# Preserve the deferral exit code (75) as a non-error signal. +set +e +if [ -n "${REFRESH_CMD:-}" ]; then + eval "$REFRESH_CMD" +else + uv run python run_refresh.py $REFRESH_ARGS +fi +rc=$? +set -e +if [ "$rc" -eq 75 ]; then + echo "refresh deferred (exit 75); committing whatever completed" +elif [ "$rc" -ne 0 ]; then + echo "refresh failed (exit $rc)"; exit "$rc" +fi + +git add 20*/ .refresh-hwm 2>/dev/null || git add 20*/ +if git diff --cached --quiet; then + echo "No changes to commit." + exit 0 +fi +git commit -m "Hourly refresh $(date -u +%Y-%m-%dT%H:%MZ)" + +# main also receives pushes from manual backfills, so rebase onto latest and +# retry with backoff rather than failing on a rejected push. +for attempt in 1 2 3 4 5; do + if ! git pull --rebase origin "$GIT_BRANCH"; then + git rebase --abort || true + echo "rebase failed (attempt $attempt/5); retrying"; sleep $((attempt * 5)); continue + fi + if git push origin "$GIT_BRANCH"; then + echo "pushed on attempt $attempt"; exit 0 + fi + echo "push rejected (attempt $attempt/5); retrying"; sleep $((attempt * 5)) +done +echo "ERROR: could not push after 5 attempts"; exit 1 +``` + +- [ ] **Step 2: Make it executable** + +Run: `chmod +x deploy/entrypoint.sh` + +- [ ] **Step 3: Write the integration test (local bare repo, no network, no token)** + +Create `tests/test_entrypoint.py`: +```python +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +"""Integration test for deploy/entrypoint.sh using a local bare repo. + +Exercises the clone -> (fake) refresh -> commit -> push loop without a network +or a real token. The token is stripped from a file:// URL by git, so a dummy +value is fine.""" +import os +import subprocess +from pathlib import Path + + +def _run(cmd, cwd=None, env=None): + return subprocess.run(cmd, cwd=cwd, env=env, check=True, + capture_output=True, text=True) + + +def _git_env(): + env = dict(os.environ) + env.update({ + "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@e", + "GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@e", + }) + return env + + +def test_entrypoint_commits_new_csv(tmp_path): + genv = _git_env() + # A bare "origin" with one commit on main containing an empty data dir marker. + origin = tmp_path / "origin.git" + _run(["git", "init", "--bare", "-b", "main", str(origin)], env=genv) + seed = tmp_path / "seed" + _run(["git", "clone", str(origin), str(seed)], env=genv) + (seed / "2099").mkdir() + (seed / "2099" / ".keep").write_text("") + _run(["git", "add", "-A"], cwd=seed, env=genv) + _run(["git", "commit", "-m", "seed"], cwd=seed, env=genv) + _run(["git", "push", "origin", "main"], cwd=seed, env=genv) + + repo_root = Path(__file__).resolve().parent.parent + env = _git_env() + env.update({ + "GIT_REPO_URL": f"file://{origin}", + "GITHUB_TOKEN": "dummy", + "GIT_BRANCH": "main", + # Fake refresh: write a CSV under 2099/ so there is something to commit. + "REFRESH_CMD": "echo id > 2099/questions-test-2099-01-01.csv", + }) + _run(["bash", str(repo_root / "deploy" / "entrypoint.sh")], env=env) + + # The bare origin now has the new file on main. + log = _run(["git", "log", "--oneline", "-1", "main"], cwd=origin, env=genv) + assert "Hourly refresh" in log.stdout + show = _run(["git", "show", + "main:2099/questions-test-2099-01-01.csv"], cwd=origin, env=genv) + assert show.stdout.strip() == "id" + + +def test_entrypoint_no_changes_no_commit(tmp_path): + genv = _git_env() + origin = tmp_path / "origin.git" + _run(["git", "init", "--bare", "-b", "main", str(origin)], env=genv) + seed = tmp_path / "seed" + _run(["git", "clone", str(origin), str(seed)], env=genv) + (seed / "2099").mkdir() + (seed / "2099" / ".keep").write_text("") + _run(["git", "add", "-A"], cwd=seed, env=genv) + _run(["git", "commit", "-m", "seed"], cwd=seed, env=genv) + _run(["git", "push", "origin", "main"], cwd=seed, env=genv) + + repo_root = Path(__file__).resolve().parent.parent + env = _git_env() + env.update({ + "GIT_REPO_URL": f"file://{origin}", "GITHUB_TOKEN": "dummy", + "GIT_BRANCH": "main", "REFRESH_CMD": "true", # writes nothing + }) + _run(["bash", str(repo_root / "deploy" / "entrypoint.sh")], env=env) + + count = _run(["git", "rev-list", "--count", "main"], cwd=origin, env=genv) + assert count.stdout.strip() == "1" # still just the seed commit +``` + +- [ ] **Step 4: Run the entrypoint tests** + +Run: `uv run pytest tests/test_entrypoint.py -v` +Expected: **2 passed** — a new CSV produces one `Hourly refresh` commit; an empty run produces none. + +- [ ] **Step 5: Flip the Dockerfile CMD to the entrypoint** + +In `Dockerfile`, replace the final line: +```dockerfile +CMD ["python", "-c", "import sumo; print('image ok')"] +``` +with: +```dockerfile +ENTRYPOINT ["deploy/entrypoint.sh"] +``` + +- [ ] **Step 6: Rebuild and confirm the entrypoint is wired (fails fast without env, as designed)** + +Run: `docker run --rm aaq-scraper:dev; echo "exit=$?"` after `docker buildx build --platform linux/arm64 -t aaq-scraper:dev --load .` +Expected: exits non-zero with `set GIT_REPO_URL` (the `:?` guard) — proving the entrypoint runs. + +- [ ] **Step 7: Commit** + +```bash +git add deploy/entrypoint.sh tests/test_entrypoint.py Dockerfile +git commit -m "feat: pod entrypoint clones, refreshes, and commits CSVs back to git" +``` + +## Task A6: Move the high-water mark into the repo + +**Files:** +- Modify: `.gitignore` (remove the `/.refresh-hwm` ignore) + +- [ ] **Step 1: Un-ignore the mark** + +In `.gitignore`, delete these two lines: +``` +# Incremental-refresh high-water mark (persisted via Actions cache, not git) +/.refresh-hwm +``` + +- [ ] **Step 2: Verify it is no longer ignored** + +Run: `git check-ignore .refresh-hwm; echo "exit=$?"` +Expected: `exit=1` (not ignored). No `.refresh-hwm` file exists yet; the first CronJob run creates and commits it, and `run_refresh.py`'s 26h lookback covers its initial absence. + +- [ ] **Step 3: Commit** + +```bash +git add .gitignore +git commit -m "chore: track .refresh-hwm in git (state moves off the Actions cache)" +``` + +## Task A7: Image-build CI workflow (OIDC → ECR) + +**Files:** +- Create: `.github/workflows/aaq-scraper-image.yml` + +**Interfaces:** +- Consumes: GitHub Environment `image-aaq-scraper` (main-only) and repo variable `IMAGE_PUSH_ROLE_ARN` — the OIDC push role created in Task B1. Document this prerequisite in the workflow header. +- Produces: an immutable image `…/aaq-scraper:git-` in the shared ECR on every push to `main` that touches scraper source. + +- [ ] **Step 1: Write the workflow** + +Create `.github/workflows/aaq-scraper-image.yml` (adapt the `thundermail-ticket-spike-monitor-image.yml` external-repo-trust variant found in `platform-infrastructure/.github/workflows/`; copy its exact structure, changing names/paths): +```yaml +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +# +# Build & push the scraper image to the shared ECR via GitHub OIDC. +# One-time setup (see platform-infrastructure Task B1): +# - GitHub Environment "image-aaq-scraper" (protected to main) +# - repo variable IMAGE_PUSH_ROLE_ARN = the push-only role ARN +name: Build scraper image + +on: + push: + branches: [main] + paths: + - "**.py" + - "pyproject.toml" + - "uv.lock" + - "Dockerfile" + - "deploy/entrypoint.sh" + - ".github/workflows/aaq-scraper-image.yml" + workflow_dispatch: + inputs: + tag: + description: "Explicit image tag (default git-)" + required: false + +permissions: + contents: read + id-token: write + +jobs: + build: + runs-on: ubuntu-latest + environment: image-aaq-scraper + env: + ECR: 826971876779.dkr.ecr.us-east-1.amazonaws.com + REPO: aaq-scraper + steps: + - uses: actions/checkout@v5 + - name: Compute tag + id: tag + run: | + T="${{ github.event.inputs.tag }}" + [ -n "$T" ] || T="git-$(git rev-parse --short HEAD)" + echo "tag=$T" >> "$GITHUB_OUTPUT" + - name: Configure AWS credentials (OIDC) + uses: aws-actions/configure-aws-credentials@v4 + with: + role-to-assume: ${{ vars.IMAGE_PUSH_ROLE_ARN }} + aws-region: us-east-1 + - name: Login to ECR + uses: aws-actions/amazon-ecr-login@v2 + - name: Set up Buildx + uses: docker/setup-buildx-action@v3 + - name: Build and push + uses: docker/build-push-action@v6 + with: + context: . + platforms: linux/arm64 + push: true + tags: ${{ env.ECR }}/${{ env.REPO }}:${{ steps.tag.outputs.tag }} +``` + +- [ ] **Step 2: Lint the workflow** + +Run: `command -v actionlint >/dev/null && actionlint .github/workflows/aaq-scraper-image.yml || echo "actionlint not installed; skip (validate YAML manually)"` +Expected: no errors (or a clean skip). Also confirm valid YAML: `uv run python -c "import yaml,sys; yaml.safe_load(open('.github/workflows/aaq-scraper-image.yml')); print('yaml ok')"`. + +- [ ] **Step 3: Commit** + +```bash +git add .github/workflows/aaq-scraper-image.yml +git commit -m "ci: build and push the scraper image to ECR via OIDC" +``` + +## Task A8: Disable `schema-check.yml` with a reviewer note + +**Files:** +- Modify: `.github/workflows/schema-check.yml` + +- [ ] **Step 1: Disable the triggers, keep the file** + +In `.github/workflows/schema-check.yml`, replace the `on:` block with a manual-only trigger and add a header note. Comment out the `schedule:` so it stops firing (and stops opening spurious `api-blocked` issues once the browser path is gone), but keep `workflow_dispatch` so it can still be run by hand: +```yaml +# NOTE (2026-07-13, #28): parked pending its own migration to the k8s runner. +# check_schema.py hits the live SUMO API and is blocked by the same Fastly +# challenge as the scraper, so the daily schedule is disabled to stop spurious +# api-blocked issues. Re-enable / migrate after the egress IP is allowlisted. +on: + # schedule: + # - cron: "30 6 * * *" # disabled — see NOTE above + workflow_dispatch: +``` + +- [ ] **Step 2: Validate YAML** + +Run: `uv run python -c "import yaml; yaml.safe_load(open('.github/workflows/schema-check.yml')); print('yaml ok')"` +Expected: `yaml ok`. + +- [ ] **Step 3: Commit** + +```bash +git add .github/workflows/schema-check.yml +git commit -m "ci: park schema-check (blocked by the same challenge; #28)" +``` + +## Task A9: Documentation + +**Files:** +- Modify: `CLAUDE.md` (Automation section), `README.md` + +- [ ] **Step 1: Update CLAUDE.md** + +In the **Automation (GitHub Actions)** section of `CLAUDE.md`, add a subsection documenting: the scraper now runs as an ArgoCD CronJob on the workloads cluster (`services/aaq-scraper/` in `platform-infrastructure`); `sumo.py` uses `httpx` (Playwright dropped, #28); `.refresh-hwm` is now committed to the repo (not the Actions cache); the pod pushes commits via a fine-grained PAT from AWS Secrets Manager (ESO); `schema-check.yml` is parked; `scrape.yml` stays until cutover. Keep it to a tight paragraph in the existing style. + +- [ ] **Step 2: Update README.md** + +In `README.md`, update the "drive a real browser" framing to note the httpx client + k8s deployment, and point at the design/plan docs under `docs/superpowers/`. + +- [ ] **Step 3: Full test sweep before pushing** + +Run: `uv run pytest -v` +Expected: all tests pass (fetch_json contract + httpx mapping + entrypoint). + +- [ ] **Step 4: Commit and push the branch to PR #44** + +```bash +git add CLAUDE.md README.md +git commit -m "docs: document the k8s CronJob deployment and httpx migration" +git push +``` + +--- + +# Phase B — `platform-infrastructure` repo (separate PR) + +Executes in the `platform-infrastructure` checkout (`/home/aatchison/src/tb/platform-infrastructure`), on a new branch. These tasks use verification commands rather than unit tests (infra-as-code). Before writing manifests, **open the canonical example** `services/bamboohr-cal-sync/deploy/` and `argocd/workloads/apps/thundermail-ticket-spike-monitor.yaml` and copy their structure — especially any block this plan says to "copy verbatim". + +## Task B1: Pulumi — ECR repo + OIDC push role + +**Files:** +- Modify: `pulumi/environments/mzla-shared-services/config.prod.yaml` + +- [ ] **Step 1: Add the ECR repo and push role** + +Under `ecr.repositories`, add: +```yaml + aaq-scraper: + description: "SUMO AAQ scraper CronJob image" +``` +Under `github_oidc.roles`, add a push-only role trusting the scraper repo (mirror the existing `thundermail-ticket-spike-monitor-image` role that trusts an external repo — copy its inline ECR-push policy verbatim, changing only the repository resource ARN to `…:repository/aaq-scraper` and the trusted repo to `thunderbird/aaq-scraper`): +```yaml + aaq-scraper-image: + github_repo: thunderbird/aaq-scraper + github_environment: image-aaq-scraper + managed_policy_arns: [] + # inline policy: ecr:GetAuthorizationToken + push actions scoped to + # arn:aws:ecr:us-east-1:826971876779:repository/aaq-scraper +``` + +- [ ] **Step 2: Preview** + +Run: `cd pulumi/environments/mzla-shared-services && pulumi preview` +Expected: shows **create** for the ECR repo `aaq-scraper` and the OIDC role, no unexpected deletes/replacements. + +- [ ] **Step 3: Apply (with approval), then record outputs** + +Run: `pulumi up` (confirm). Note the created role ARN — it becomes `IMAGE_PUSH_ROLE_ARN` in the aaq-scraper repo (Task A7 prerequisite). + +- [ ] **Step 4: Commit** + +```bash +git add pulumi/environments/mzla-shared-services/config.prod.yaml +git commit -m "pulumi: add aaq-scraper ECR repo + OIDC push role" +``` + +## Task B2: Verify / ensure a stable NAT egress EIP (the IP for #27) + +The whole effort's value depends on this. Investigation + possible small change. + +**Files:** (only if a change is needed) the workloads-cluster networking Pulumi program. + +- [ ] **Step 1: Find the workloads cluster's egress path** + +Run (with the workloads AWS profile): +```bash +aws ec2 describe-nat-gateways --filter "Name=state,Values=available" \ + --query "NatGateways[].{id:NatGatewayId,eip:NatGatewayAddresses[].PublicIp,vpc:VpcId}" \ + --region eu-central-1 --output table +``` +Expected: identify the NAT gateway(s) for the `mzla-eks-workloads01` VPC and their public EIP(s). + +- [ ] **Step 2: Confirm stability** + +Confirm the NAT gateway uses an **allocated Elastic IP** (stable across restarts), not an auto-assigned address, by checking the `AllocationId` is a managed EIP in the Pulumi/networking code. If it is already a fixed EIP: **record that IP** — it is what #27 hands Mozilla. If egress is ephemeral or per-AZ with multiple IPs: add/pin a single NAT EIP (or document all egress IPs for Mozilla to allowlist) as a change to the networking Pulumi program, `pulumi preview` → `up`. + +- [ ] **Step 3: Post the egress IP to #27** + +Add a comment on issue #27 with the confirmed egress IP(s) so Roland can forward to Mozilla for the allowlist. (No code change / commit if the EIP already existed.) + +## Task B3: Kubernetes manifests + +**Files:** +- Create: `services/aaq-scraper/deploy/serviceaccount.yaml` +- Create: `services/aaq-scraper/deploy/configmap.yaml` +- Create: `services/aaq-scraper/deploy/externalsecret.yaml` +- Create: `services/aaq-scraper/deploy/cronjob.yaml` +- Create: `services/aaq-scraper/deploy/vmrule.yaml` + +**Interfaces:** +- Consumes: the image from Task A7; the SM secret `mzla/shared-services/aaq-scraper` (Task B5); the `aws-secrets-manager` ClusterSecretStore (existing on the workloads cluster). +- Produces: an hourly CronJob in namespace `aaq-scraper`. + +- [ ] **Step 1: ServiceAccount** — create `serviceaccount.yaml` (sync-wave "0"): +```yaml +# MPL header +apiVersion: v1 +kind: ServiceAccount +metadata: + name: aaq-scraper + namespace: aaq-scraper + annotations: + argocd.argoproj.io/sync-wave: "0" +automountServiceAccountToken: false +``` + +- [ ] **Step 2: ConfigMap** — create `configmap.yaml` (sync-wave "0"), non-secret env: +```yaml +# MPL header +apiVersion: v1 +kind: ConfigMap +metadata: + name: aaq-scraper-config + namespace: aaq-scraper + annotations: + argocd.argoproj.io/sync-wave: "0" +data: + GIT_REPO_URL: "https://github.com/thunderbird/aaq-scraper.git" + GIT_BRANCH: "main" + GIT_AUTHOR_NAME: "aaq-scraper-bot" + GIT_AUTHOR_EMAIL: "aaq-scraper-bot@thunderbird.net" + REFRESH_ARGS: "--soft-deadline 40 --max-429-wait 120" +``` + +- [ ] **Step 3: ExternalSecret** — create `externalsecret.yaml` (sync-wave "0"). Copy the `external-secrets.io/v1` shape from an existing app; map the PAT: +```yaml +# MPL header +# PREREQUISITE: create SM secret mzla/shared-services/aaq-scraper with key +# `githubToken` = a fine-grained PAT on thunderbird/aaq-scraper, Contents: RW. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: aaq-scraper + namespace: aaq-scraper + annotations: + argocd.argoproj.io/sync-wave: "0" +spec: + refreshInterval: 1h + secretStoreRef: + name: aws-secrets-manager-shared # cross-account store for mzla/shared-services/* + kind: ClusterSecretStore + target: + name: aaq-scraper + creationPolicy: Owner + data: + - secretKey: GITHUB_TOKEN + remoteRef: + key: mzla/shared-services/aaq-scraper + property: githubToken +``` +(Confirm the correct store name on the workloads cluster — the explore notes both `aws-secrets-manager` and `aws-secrets-manager-shared`; `mzla/shared-services/*` lives in the shared account, so use `-shared`.) + +- [ ] **Step 4: CronJob** — create `cronjob.yaml` (sync-wave "1"). Copy the hardened `securityContext` from `bamboohr-cal-sync`; add the writable emptyDir workdir: +```yaml +# MPL header +apiVersion: batch/v1 +kind: CronJob +metadata: + name: aaq-scraper + namespace: aaq-scraper + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + schedule: "0 * * * *" + timeZone: "Etc/UTC" + concurrencyPolicy: Forbid + startingDeadlineSeconds: 300 + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 1 + activeDeadlineSeconds: 3300 + ttlSecondsAfterFinished: 3600 + template: + spec: + serviceAccountName: aaq-scraper + restartPolicy: OnFailure + securityContext: + runAsNonRoot: true + runAsUser: 65532 + seccompProfile: { type: RuntimeDefault } + containers: + - name: aaq-scraper + image: 826971876779.dkr.ecr.us-east-1.amazonaws.com/aaq-scraper:REPLACE_WITH_git- + envFrom: + - configMapRef: { name: aaq-scraper-config } + - secretRef: { name: aaq-scraper } + env: + - name: HOME + value: /work # git config/creds must land on the writable volume + resources: + requests: { cpu: 100m, memory: 256Mi } + limits: { cpu: "1", memory: 1Gi } + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: [ALL] } + volumeMounts: + - name: work + mountPath: /work + volumes: + - name: work + emptyDir: {} +``` + +- [ ] **Step 5: VMRule** — create `vmrule.yaml` (sync-wave "1"), alerting on CronJob failure via kube-state-metrics, shipped **suppressed** until cutover: +```yaml +# MPL header +# NOTE: the job fails every hour until our egress IP is allowlisted (#27), so +# this alert is intentionally inert until then. At cutover, remove the always- +# false guard (`and vector(0)`) to arm it. +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMRule +metadata: + name: aaq-scraper + namespace: aaq-scraper + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + groups: + - name: aaq-scraper + rules: + - alert: AaqScraperJobFailing + expr: | + (max(kube_job_status_failed{namespace="aaq-scraper"}) > 0) and vector(0) + for: 2h + labels: { severity: warning } + annotations: + summary: "AAQ scraper CronJob has been failing" + description: "The aaq-scraper CronJob has failed for >2h. (Suppressed until IP allowlisting — #27.)" +``` + +- [ ] **Step 6: Validate all manifests** + +Run: `kubeconform -strict -ignore-missing-schemas services/aaq-scraper/deploy/*.yaml` (or `kubectl apply --dry-run=client -f services/aaq-scraper/deploy/`). +Expected: all valid (CRDs like ExternalSecret/VMRule need `-ignore-missing-schemas`). + +- [ ] **Step 7: Commit** + +```bash +git add services/aaq-scraper/deploy/ +git commit -m "feat: aaq-scraper CronJob manifests for the workloads cluster" +``` + +## Task B4: ArgoCD Application + +**Files:** +- Create: `argocd/workloads/apps/aaq-scraper.yaml` + +- [ ] **Step 1: Write the Application** + +Copy `argocd/workloads/apps/thundermail-ticket-spike-monitor.yaml` verbatim and change name/path/namespace. Keep its `destination.server` (the workloads cluster URL), the standard ESO `ignoreDifferences` block, and `syncPolicy` **exactly as in that file**: +```yaml +# MPL header +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: aaq-scraper + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "6" +spec: + project: workloads + source: + repoURL: https://github.com/thunderbird/platform-infrastructure.git + targetRevision: main + path: services/aaq-scraper/deploy + destination: + server: https://9AFFF1C78D073BA9579F882E6626DE0A.sk1.eu-central-1.eks.amazonaws.com # confirm from thundermail app + namespace: aaq-scraper + ignoreDifferences: + # COPY VERBATIM from thundermail-ticket-spike-monitor.yaml (ESO block) + [] + syncPolicy: + automated: { prune: true, selfHeal: true } + syncOptions: [ CreateNamespace=true, ServerSideApply=true, RespectIgnoreDifferences=true ] +``` + +- [ ] **Step 2: Validate** + +Run: `kubeconform -strict -ignore-missing-schemas argocd/workloads/apps/aaq-scraper.yaml` +Expected: valid. + +- [ ] **Step 3: Commit** + +```bash +git add argocd/workloads/apps/aaq-scraper.yaml +git commit -m "feat: register aaq-scraper ArgoCD app on the workloads cluster" +``` + +## Task B5: Create the AWS Secrets Manager secret (runbook) + +**Files:** none (operational). + +- [ ] **Step 1: Mint a fine-grained PAT** on `thunderbird/aaq-scraper` with **Contents: Read and write** only, no expiry-less (set a calendar reminder to rotate). + +- [ ] **Step 2: Store it in Secrets Manager** (shared account, us-east-1): +```bash +aws secretsmanager create-secret \ + --name mzla/shared-services/aaq-scraper \ + --secret-string '{"githubToken":"github_pat_..."}' \ + --region us-east-1 +``` +Expected: secret created; ESO will materialize it into the `aaq-scraper` Secret on next refresh. + +## Task B6: Push Phase B branch and open the platform-infrastructure PR + +- [ ] **Step 1: Push and open PR** + +```bash +git push -u origin aaq-scraper-cronjob +gh pr create --repo thunderbird/platform-infrastructure --base main \ + --title "Add aaq-scraper CronJob (workloads cluster)" \ + --body "Deploys the SUMO AAQ scraper as an hourly CronJob on the workloads cluster. Pairs with thunderbird/aaq-scraper#44. Bump the image tag once the first image is built. Relates to thunderbird/aaq-scraper#27, #28." +``` + +--- + +# Cutover (follow-up, after Mozilla allowlists — not in these PRs) + +- Bump the CronJob `image:` tag to the first built `git-` (platform-infra PR); confirm ArgoCD syncs and a manually-triggered Job commits real CSVs. +- Arm the VMRule (remove `and vector(0)`). +- Disable `scrape.yml` in the aaq-scraper repo. +- Decide on `schema-check.yml` migration. + +--- + +## Self-Review + +**Spec coverage:** §1 placement/egress → A (cluster consts) + B2/B4; §2 image+refactor → A2/A4; §3 image build → A4/A7/B1; §4 manifests → B3; §5 secrets → B3(externalsecret)/B5; §6 output/state → A5/A6; §7 alerting → B3(vmrule); §8 cutover → dedicated section. Out-of-scope items (schema-check disable → A8; scrape.yml stays → cutover) covered. No gaps. + +**Placeholder scan:** The only intentional placeholders are `image: …:REPLACE_WITH_git-` (bumped at release, per the human-gated-deploy constraint) and two "copy verbatim from existing file" instructions (ESO `ignoreDifferences` block, thundermail push-role policy) — deliberate, because transcribing those blindly risks transcription bugs; the plan names the exact source file. No TBD/TODO/"handle edge cases". + +**Type consistency:** `_raw_fetch` returns `{status, json, snippet, retry_after}` everywhere (A1 stub, A2 impl, A2 httpx test). Env var names match between `entrypoint.sh` (A5), `configmap.yaml` and `externalsecret.yaml` (B3): `GIT_REPO_URL`, `GIT_BRANCH`, `GIT_AUTHOR_NAME`, `GIT_AUTHOR_EMAIL`, `REFRESH_ARGS`, `GITHUB_TOKEN`. Image ref (`826971876779….amazonaws.com/aaq-scraper`) consistent across A7/B1/B3. From 498f885cfb96d76b8a71578e5380760ffc878c09 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 11:39:41 -0700 Subject: [PATCH 03/17] test: pin SumoBrowser.fetch_json retry/deferral/challenge contract Co-Authored-By: Claude Opus 4.8 (1M context) --- pyproject.toml | 5 ++ tests/__init__.py | 3 + tests/test_fetch_json.py | 69 ++++++++++++++++++++ uv.lock | 137 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 214 insertions(+) create mode 100644 tests/__init__.py create mode 100644 tests/test_fetch_json.py diff --git a/pyproject.toml b/pyproject.toml index 837ba5e..f07dbb9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,3 +6,8 @@ requires-python = ">=3.10" dependencies = [ "playwright==1.49.1", ] + +[dependency-groups] +dev = [ + "pytest>=8", +] diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..448bb86 --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1,3 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. diff --git a/tests/test_fetch_json.py b/tests/test_fetch_json.py new file mode 100644 index 0000000..72ad599 --- /dev/null +++ b/tests/test_fetch_json.py @@ -0,0 +1,69 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +"""Contract/regression tests for SumoBrowser.fetch_json. + +These stub _raw_fetch (no browser, no network), so they pin the retry / +deferral / challenge behavior independently of the transport. They must pass +both before and after the Playwright->httpx refactor (Task A2).""" +import pytest + +import sumo + + +def _resp(status, json=None, snippet="", retry_after=None): + return {"status": status, "json": json, "snippet": snippet, + "retry_after": retry_after} + + +def _browser(monkeypatch, responses, **kwargs): + """A SumoBrowser whose _raw_fetch yields the given canned responses in + order, with sleeping/jitter neutralized so tests run instantly.""" + kwargs.setdefault("max_attempts", 5) + kwargs.setdefault("backoff_base", 2.0) + kwargs.setdefault("retry_jitter_min_s", 0) + kwargs.setdefault("retry_jitter_max_s", 0) + sb = sumo.SumoBrowser(**kwargs) + seq = iter(responses) + monkeypatch.setattr(sb, "_raw_fetch", lambda url: next(seq)) + monkeypatch.setattr(sumo.time, "sleep", lambda _s: None) + monkeypatch.setattr(sumo.random, "uniform", lambda _a, _b: 0.0) + return sb + + +def test_returns_json_on_200(monkeypatch): + sb = _browser(monkeypatch, [_resp(200, {"count": 3})]) + assert sb.fetch_json("u") == {"count": 3} + + +def test_non_retryable_4xx_raises_plain_runtimeerror(monkeypatch): + sb = _browser(monkeypatch, [_resp(404, snippet="nope")]) + with pytest.raises(RuntimeError) as ei: + sb.fetch_json("u") + assert not isinstance(ei.value, sumo.ChallengeError) + assert not isinstance(ei.value, sumo.RateLimitDeferral) + + +def test_5xx_retried_then_success(monkeypatch): + sb = _browser(monkeypatch, [_resp(503), _resp(200, {"ok": 1})]) + assert sb.fetch_json("u") == {"ok": 1} + + +def test_exhausted_200_non_json_raises_challenge_error(monkeypatch): + sb = _browser(monkeypatch, [_resp(200, None, "challenge")] * 5) + with pytest.raises(sumo.ChallengeError): + sb.fetch_json("u") + + +def test_429_over_threshold_defers(monkeypatch): + sb = _browser(monkeypatch, [_resp(429, retry_after="600")], + max_429_wait_s=120) + with pytest.raises(sumo.RateLimitDeferral): + sb.fetch_json("u") + + +def test_429_under_threshold_retries_then_succeeds(monkeypatch): + sb = _browser(monkeypatch, + [_resp(429, retry_after="10"), _resp(200, {"ok": 1})], + max_429_wait_s=1000) + assert sb.fetch_json("u") == {"ok": 1} diff --git a/uv.lock b/uv.lock index e7cf921..51e59c6 100644 --- a/uv.lock +++ b/uv.lock @@ -10,9 +10,38 @@ dependencies = [ { name = "playwright" }, ] +[package.dev-dependencies] +dev = [ + { name = "pytest" }, +] + [package.metadata] requires-dist = [{ name = "playwright", specifier = "==1.49.1" }] +[package.metadata.requires-dev] +dev = [{ name = "pytest", specifier = ">=8" }] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "exceptiongroup" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598", size = 16740, upload-time = "2025-11-21T23:01:53.443Z" }, +] + [[package]] name = "greenlet" version = "3.1.1" @@ -64,6 +93,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ac/38/08cc303ddddc4b3d7c628c3039a61a3aae36c241ed01393d00c2fd663473/greenlet-3.1.1-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:411f015496fec93c1c8cd4e5238da364e1da7a124bcb293f085bf2860c32c6f6", size = 1142112, upload-time = "2024-09-20T17:09:28.753Z" }, ] +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "packaging" +version = "26.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, +] + [[package]] name = "playwright" version = "1.49.1" @@ -82,6 +129,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/71/a9/bd88ac0bd498c91aab3aba2e393d1fa59f72a7243e9265ccbf4861ca4f64/playwright-1.49.1-py3-none-win_amd64.whl", hash = "sha256:47b23cb346283278f5b4d1e1990bcb6d6302f80c0aa0ca93dd0601a1400191df", size = 34060667, upload-time = "2024-12-10T17:32:56.459Z" }, ] +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + [[package]] name = "pyee" version = "12.0.0" @@ -94,6 +150,87 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1d/0d/95993c08c721ec68892547f2117e8f9dfbcef2ca71e098533541b4a54d5f/pyee-12.0.0-py3-none-any.whl", hash = "sha256:7b14b74320600049ccc7d0e0b1becd3b4bd0a03c745758225e31a59f4095c990", size = 14831, upload-time = "2024-08-30T19:40:42.132Z" }, ] +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pytest" +version = "9.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "exceptiongroup", marker = "python_full_version < '3.11'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, + { name = "tomli", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" }, +] + +[[package]] +name = "tomli" +version = "2.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/22/de/48c59722572767841493b26183a0d1cc411d54fd759c5607c4590b6563a6/tomli-2.4.1.tar.gz", hash = "sha256:7c7e1a961a0b2f2472c1ac5b69affa0ae1132c39adcb67aba98568702b9cc23f", size = 17543, upload-time = "2026-03-25T20:22:03.828Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/11/db3d5885d8528263d8adc260bb2d28ebf1270b96e98f0e0268d32b8d9900/tomli-2.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:f8f0fc26ec2cc2b965b7a3b87cd19c5c6b8c5e5f436b984e85f486d652285c30", size = 154704, upload-time = "2026-03-25T20:21:10.473Z" }, + { url = "https://files.pythonhosted.org/packages/6d/f7/675db52c7e46064a9aa928885a9b20f4124ecb9bc2e1ce74c9106648d202/tomli-2.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4ab97e64ccda8756376892c53a72bd1f964e519c77236368527f758fbc36a53a", size = 149454, upload-time = "2026-03-25T20:21:12.036Z" }, + { url = "https://files.pythonhosted.org/packages/61/71/81c50943cf953efa35bce7646caab3cf457a7d8c030b27cfb40d7235f9ee/tomli-2.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96481a5786729fd470164b47cdb3e0e58062a496f455ee41b4403be77cb5a076", size = 237561, upload-time = "2026-03-25T20:21:13.098Z" }, + { url = "https://files.pythonhosted.org/packages/48/c1/f41d9cb618acccca7df82aaf682f9b49013c9397212cb9f53219e3abac37/tomli-2.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5a881ab208c0baf688221f8cecc5401bd291d67e38a1ac884d6736cbcd8247e9", size = 243824, upload-time = "2026-03-25T20:21:14.569Z" }, + { url = "https://files.pythonhosted.org/packages/22/e4/5a816ecdd1f8ca51fb756ef684b90f2780afc52fc67f987e3c61d800a46d/tomli-2.4.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:47149d5bd38761ac8be13a84864bf0b7b70bc051806bc3669ab1cbc56216b23c", size = 242227, upload-time = "2026-03-25T20:21:15.712Z" }, + { url = "https://files.pythonhosted.org/packages/6b/49/2b2a0ef529aa6eec245d25f0c703e020a73955ad7edf73e7f54ddc608aa5/tomli-2.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ec9bfaf3ad2df51ace80688143a6a4ebc09a248f6ff781a9945e51937008fcbc", size = 247859, upload-time = "2026-03-25T20:21:17.001Z" }, + { url = "https://files.pythonhosted.org/packages/83/bd/6c1a630eaca337e1e78c5903104f831bda934c426f9231429396ce3c3467/tomli-2.4.1-cp311-cp311-win32.whl", hash = "sha256:ff2983983d34813c1aeb0fa89091e76c3a22889ee83ab27c5eeb45100560c049", size = 97204, upload-time = "2026-03-25T20:21:18.079Z" }, + { url = "https://files.pythonhosted.org/packages/42/59/71461df1a885647e10b6bb7802d0b8e66480c61f3f43079e0dcd315b3954/tomli-2.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:5ee18d9ebdb417e384b58fe414e8d6af9f4e7a0ae761519fb50f721de398dd4e", size = 108084, upload-time = "2026-03-25T20:21:18.978Z" }, + { url = "https://files.pythonhosted.org/packages/b8/83/dceca96142499c069475b790e7913b1044c1a4337e700751f48ed723f883/tomli-2.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:c2541745709bad0264b7d4705ad453b76ccd191e64aa6f0fc66b69a293a45ece", size = 95285, upload-time = "2026-03-25T20:21:20.309Z" }, + { url = "https://files.pythonhosted.org/packages/c1/ba/42f134a3fe2b370f555f44b1d72feebb94debcab01676bf918d0cb70e9aa/tomli-2.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c742f741d58a28940ce01d58f0ab2ea3ced8b12402f162f4d534dfe18ba1cd6a", size = 155924, upload-time = "2026-03-25T20:21:21.626Z" }, + { url = "https://files.pythonhosted.org/packages/dc/c7/62d7a17c26487ade21c5422b646110f2162f1fcc95980ef7f63e73c68f14/tomli-2.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7f86fd587c4ed9dd76f318225e7d9b29cfc5a9d43de44e5754db8d1128487085", size = 150018, upload-time = "2026-03-25T20:21:23.002Z" }, + { url = "https://files.pythonhosted.org/packages/5c/05/79d13d7c15f13bdef410bdd49a6485b1c37d28968314eabee452c22a7fda/tomli-2.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ff18e6a727ee0ab0388507b89d1bc6a22b138d1e2fa56d1ad494586d61d2eae9", size = 244948, upload-time = "2026-03-25T20:21:24.04Z" }, + { url = "https://files.pythonhosted.org/packages/10/90/d62ce007a1c80d0b2c93e02cab211224756240884751b94ca72df8a875ca/tomli-2.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:136443dbd7e1dee43c68ac2694fde36b2849865fa258d39bf822c10e8068eac5", size = 253341, upload-time = "2026-03-25T20:21:25.177Z" }, + { url = "https://files.pythonhosted.org/packages/1a/7e/caf6496d60152ad4ed09282c1885cca4eea150bfd007da84aea07bcc0a3e/tomli-2.4.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5e262d41726bc187e69af7825504c933b6794dc3fbd5945e41a79bb14c31f585", size = 248159, upload-time = "2026-03-25T20:21:26.364Z" }, + { url = "https://files.pythonhosted.org/packages/99/e7/c6f69c3120de34bbd882c6fba7975f3d7a746e9218e56ab46a1bc4b42552/tomli-2.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5cb41aa38891e073ee49d55fbc7839cfdb2bc0e600add13874d048c94aadddd1", size = 253290, upload-time = "2026-03-25T20:21:27.46Z" }, + { url = "https://files.pythonhosted.org/packages/d6/2f/4a3c322f22c5c66c4b836ec58211641a4067364f5dcdd7b974b4c5da300c/tomli-2.4.1-cp312-cp312-win32.whl", hash = "sha256:da25dc3563bff5965356133435b757a795a17b17d01dbc0f42fb32447ddfd917", size = 98141, upload-time = "2026-03-25T20:21:28.492Z" }, + { url = "https://files.pythonhosted.org/packages/24/22/4daacd05391b92c55759d55eaee21e1dfaea86ce5c571f10083360adf534/tomli-2.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:52c8ef851d9a240f11a88c003eacb03c31fc1c9c4ec64a99a0f922b93874fda9", size = 108847, upload-time = "2026-03-25T20:21:29.386Z" }, + { url = "https://files.pythonhosted.org/packages/68/fd/70e768887666ddd9e9f5d85129e84910f2db2796f9096aa02b721a53098d/tomli-2.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:f758f1b9299d059cc3f6546ae2af89670cb1c4d48ea29c3cacc4fe7de3058257", size = 95088, upload-time = "2026-03-25T20:21:30.677Z" }, + { url = "https://files.pythonhosted.org/packages/07/06/b823a7e818c756d9a7123ba2cda7d07bc2dd32835648d1a7b7b7a05d848d/tomli-2.4.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:36d2bd2ad5fb9eaddba5226aa02c8ec3fa4f192631e347b3ed28186d43be6b54", size = 155866, upload-time = "2026-03-25T20:21:31.65Z" }, + { url = "https://files.pythonhosted.org/packages/14/6f/12645cf7f08e1a20c7eb8c297c6f11d31c1b50f316a7e7e1e1de6e2e7b7e/tomli-2.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:eb0dc4e38e6a1fd579e5d50369aa2e10acfc9cace504579b2faabb478e76941a", size = 149887, upload-time = "2026-03-25T20:21:33.028Z" }, + { url = "https://files.pythonhosted.org/packages/5c/e0/90637574e5e7212c09099c67ad349b04ec4d6020324539297b634a0192b0/tomli-2.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c7f2c7f2b9ca6bdeef8f0fa897f8e05085923eb091721675170254cbc5b02897", size = 243704, upload-time = "2026-03-25T20:21:34.51Z" }, + { url = "https://files.pythonhosted.org/packages/10/8f/d3ddb16c5a4befdf31a23307f72828686ab2096f068eaf56631e136c1fdd/tomli-2.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f3c6818a1a86dd6dca7ddcaaf76947d5ba31aecc28cb1b67009a5877c9a64f3f", size = 251628, upload-time = "2026-03-25T20:21:36.012Z" }, + { url = "https://files.pythonhosted.org/packages/e3/f1/dbeeb9116715abee2485bf0a12d07a8f31af94d71608c171c45f64c0469d/tomli-2.4.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d312ef37c91508b0ab2cee7da26ec0b3ed2f03ce12bd87a588d771ae15dcf82d", size = 247180, upload-time = "2026-03-25T20:21:37.136Z" }, + { url = "https://files.pythonhosted.org/packages/d3/74/16336ffd19ed4da28a70959f92f506233bd7cfc2332b20bdb01591e8b1d1/tomli-2.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:51529d40e3ca50046d7606fa99ce3956a617f9b36380da3b7f0dd3dd28e68cb5", size = 251674, upload-time = "2026-03-25T20:21:38.298Z" }, + { url = "https://files.pythonhosted.org/packages/16/f9/229fa3434c590ddf6c0aa9af64d3af4b752540686cace29e6281e3458469/tomli-2.4.1-cp313-cp313-win32.whl", hash = "sha256:2190f2e9dd7508d2a90ded5ed369255980a1bcdd58e52f7fe24b8162bf9fedbd", size = 97976, upload-time = "2026-03-25T20:21:39.316Z" }, + { url = "https://files.pythonhosted.org/packages/6a/1e/71dfd96bcc1c775420cb8befe7a9d35f2e5b1309798f009dca17b7708c1e/tomli-2.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:8d65a2fbf9d2f8352685bc1364177ee3923d6baf5e7f43ea4959d7d8bc326a36", size = 108755, upload-time = "2026-03-25T20:21:40.248Z" }, + { url = "https://files.pythonhosted.org/packages/83/7a/d34f422a021d62420b78f5c538e5b102f62bea616d1d75a13f0a88acb04a/tomli-2.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:4b605484e43cdc43f0954ddae319fb75f04cc10dd80d830540060ee7cd0243cd", size = 95265, upload-time = "2026-03-25T20:21:41.219Z" }, + { url = "https://files.pythonhosted.org/packages/3c/fb/9a5c8d27dbab540869f7c1f8eb0abb3244189ce780ba9cd73f3770662072/tomli-2.4.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:fd0409a3653af6c147209d267a0e4243f0ae46b011aa978b1080359fddc9b6cf", size = 155726, upload-time = "2026-03-25T20:21:42.23Z" }, + { url = "https://files.pythonhosted.org/packages/62/05/d2f816630cc771ad836af54f5001f47a6f611d2d39535364f148b6a92d6b/tomli-2.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:a120733b01c45e9a0c34aeef92bf0cf1d56cfe81ed9d47d562f9ed591a9828ac", size = 149859, upload-time = "2026-03-25T20:21:43.386Z" }, + { url = "https://files.pythonhosted.org/packages/ce/48/66341bdb858ad9bd0ceab5a86f90eddab127cf8b046418009f2125630ecb/tomli-2.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:559db847dc486944896521f68d8190be1c9e719fced785720d2216fe7022b662", size = 244713, upload-time = "2026-03-25T20:21:44.474Z" }, + { url = "https://files.pythonhosted.org/packages/df/6d/c5fad00d82b3c7a3ab6189bd4b10e60466f22cfe8a08a9394185c8a8111c/tomli-2.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:01f520d4f53ef97964a240a035ec2a869fe1a37dde002b57ebc4417a27ccd853", size = 252084, upload-time = "2026-03-25T20:21:45.62Z" }, + { url = "https://files.pythonhosted.org/packages/00/71/3a69e86f3eafe8c7a59d008d245888051005bd657760e96d5fbfb0b740c2/tomli-2.4.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7f94b27a62cfad8496c8d2513e1a222dd446f095fca8987fceef261225538a15", size = 247973, upload-time = "2026-03-25T20:21:46.937Z" }, + { url = "https://files.pythonhosted.org/packages/67/50/361e986652847fec4bd5e4a0208752fbe64689c603c7ae5ea7cb16b1c0ca/tomli-2.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ede3e6487c5ef5d28634ba3f31f989030ad6af71edfb0055cbbd14189ff240ba", size = 256223, upload-time = "2026-03-25T20:21:48.467Z" }, + { url = "https://files.pythonhosted.org/packages/8c/9a/b4173689a9203472e5467217e0154b00e260621caa227b6fa01feab16998/tomli-2.4.1-cp314-cp314-win32.whl", hash = "sha256:3d48a93ee1c9b79c04bb38772ee1b64dcf18ff43085896ea460ca8dec96f35f6", size = 98973, upload-time = "2026-03-25T20:21:49.526Z" }, + { url = "https://files.pythonhosted.org/packages/14/58/640ac93bf230cd27d002462c9af0d837779f8773bc03dee06b5835208214/tomli-2.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:88dceee75c2c63af144e456745e10101eb67361050196b0b6af5d717254dddf7", size = 109082, upload-time = "2026-03-25T20:21:50.506Z" }, + { url = "https://files.pythonhosted.org/packages/d5/2f/702d5e05b227401c1068f0d386d79a589bb12bf64c3d2c72ce0631e3bc49/tomli-2.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:b8c198f8c1805dc42708689ed6864951fd2494f924149d3e4bce7710f8eb5232", size = 96490, upload-time = "2026-03-25T20:21:51.474Z" }, + { url = "https://files.pythonhosted.org/packages/45/4b/b877b05c8ba62927d9865dd980e34a755de541eb65fffba52b4cc495d4d2/tomli-2.4.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:d4d8fe59808a54658fcc0160ecfb1b30f9089906c50b23bcb4c69eddc19ec2b4", size = 164263, upload-time = "2026-03-25T20:21:52.543Z" }, + { url = "https://files.pythonhosted.org/packages/24/79/6ab420d37a270b89f7195dec5448f79400d9e9c1826df982f3f8e97b24fd/tomli-2.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7008df2e7655c495dd12d2a4ad038ff878d4ca4b81fccaf82b714e07eae4402c", size = 160736, upload-time = "2026-03-25T20:21:53.674Z" }, + { url = "https://files.pythonhosted.org/packages/02/e0/3630057d8eb170310785723ed5adcdfb7d50cb7e6455f85ba8a3deed642b/tomli-2.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1d8591993e228b0c930c4bb0db464bdad97b3289fb981255d6c9a41aedc84b2d", size = 270717, upload-time = "2026-03-25T20:21:55.129Z" }, + { url = "https://files.pythonhosted.org/packages/7a/b4/1613716072e544d1a7891f548d8f9ec6ce2faf42ca65acae01d76ea06bb0/tomli-2.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:734e20b57ba95624ecf1841e72b53f6e186355e216e5412de414e3c51e5e3c41", size = 278461, upload-time = "2026-03-25T20:21:56.228Z" }, + { url = "https://files.pythonhosted.org/packages/05/38/30f541baf6a3f6df77b3df16b01ba319221389e2da59427e221ef417ac0c/tomli-2.4.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8a650c2dbafa08d42e51ba0b62740dae4ecb9338eefa093aa5c78ceb546fcd5c", size = 274855, upload-time = "2026-03-25T20:21:57.653Z" }, + { url = "https://files.pythonhosted.org/packages/77/a3/ec9dd4fd2c38e98de34223b995a3b34813e6bdadf86c75314c928350ed14/tomli-2.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:504aa796fe0569bb43171066009ead363de03675276d2d121ac1a4572397870f", size = 283144, upload-time = "2026-03-25T20:21:59.089Z" }, + { url = "https://files.pythonhosted.org/packages/ef/be/605a6261cac79fba2ec0c9827e986e00323a1945700969b8ee0b30d85453/tomli-2.4.1-cp314-cp314t-win32.whl", hash = "sha256:b1d22e6e9387bf4739fbe23bfa80e93f6b0373a7f1b96c6227c32bef95a4d7a8", size = 108683, upload-time = "2026-03-25T20:22:00.214Z" }, + { url = "https://files.pythonhosted.org/packages/12/64/da524626d3b9cc40c168a13da8335fe1c51be12c0a63685cc6db7308daae/tomli-2.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:2c1c351919aca02858f740c6d33adea0c5deea37f9ecca1cc1ef9e884a619d26", size = 121196, upload-time = "2026-03-25T20:22:01.169Z" }, + { url = "https://files.pythonhosted.org/packages/5a/cd/e80b62269fc78fc36c9af5a6b89c835baa8af28ff5ad28c7028d60860320/tomli-2.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:eab21f45c7f66c13f2a9e0e1535309cee140182a9cdae1e041d02e47291e8396", size = 100393, upload-time = "2026-03-25T20:22:02.137Z" }, + { url = "https://files.pythonhosted.org/packages/7b/61/cceae43728b7de99d9b847560c262873a1f6c98202171fd5ed62640b494b/tomli-2.4.1-py3-none-any.whl", hash = "sha256:0d85819802132122da43cb86656f8d1f8c6587d54ae7dcaf30e90533028b49fe", size = 14583, upload-time = "2026-03-25T20:22:03.012Z" }, +] + [[package]] name = "typing-extensions" version = "4.15.0" From d1e507b9eb7bb433fd1fd372f071739367661981 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 11:44:20 -0700 Subject: [PATCH 04/17] chore: pin Python to 3.12 so uv sync builds greenlet The sandbox default is Python 3.14, against which playwright's greenlet dependency has no wheel and fails to build, breaking `uv sync`/`uv run`. Pin to 3.12 (within requires-python >=3.10) so the mandated `uv run pytest` works. (greenlet/playwright are removed in the next task, but the pin keeps the toolchain reproducible regardless of the host default.) Co-Authored-By: Claude Opus 4.8 (1M context) --- .python-version | 1 + 1 file changed, 1 insertion(+) create mode 100644 .python-version diff --git a/.python-version b/.python-version new file mode 100644 index 0000000..e4fba21 --- /dev/null +++ b/.python-version @@ -0,0 +1 @@ +3.12 From a9d69b9f87c7795e2aae0ba851b56ec6ee158a98 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 11:46:27 -0700 Subject: [PATCH 05/17] refactor(sumo): replace Playwright with httpx behind the frozen SumoBrowser API --- pyproject.toml | 2 +- sumo.py | 122 ++++++++++++++------------------- tests/test_fetch_json.py | 28 ++++++++ uv.lock | 144 ++++++++++++++++++--------------------- 4 files changed, 148 insertions(+), 148 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index f07dbb9..14fab65 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ version = "0.1.0" description = "Scrape the SUMO (support.mozilla.org) AAQ API by driving a real browser past its JavaScript challenge." requires-python = ">=3.10" dependencies = [ - "playwright==1.49.1", + "httpx==0.28.1", ] [dependency-groups] diff --git a/sumo.py b/sumo.py index 2bb52e1..5621495 100644 --- a/sumo.py +++ b/sumo.py @@ -2,20 +2,22 @@ # License, v. 2.0. If a copy of the MPL was not distributed with this # file, You can obtain one at https://mozilla.org/MPL/2.0/. """ -Shared SUMO (support.mozilla.org) browser helper. +Shared SUMO (support.mozilla.org) API client. -Drives a real Chromium (Playwright) so the JavaScript challenge that fronts the -API is satisfied, then lets callers fetch JSON API pages from inside the -browser's authenticated context. Proven in Bucket 0 (poc.py) to work both -headed and headless. +Historically drove a real Chromium (Playwright) to pass the Fastly JS/WAF +challenge; that path is now fingerprinted and blocked (issue #28), so this uses +a plain HTTP client (httpx). Once our egress IP is allowlisted by Mozilla +(issue #27) the API is reachable directly. The public API (SumoBrowser, +fetch_json, paginate) is unchanged so callers did not have to change. """ import csv +import json import random import sys import time -from playwright.sync_api import sync_playwright, TimeoutError as PWTimeoutError +import httpx def _max_csv_field_size(): @@ -73,12 +75,13 @@ class RateLimitDeferral(RuntimeError): class SumoBrowser: - """A browser session that has passed the SUMO JS challenge. + """A SUMO API HTTP session. - Use as a context manager: + Named `SumoBrowser` for backwards compatibility with existing call sites; + it no longer drives a browser. Use as a context manager: with SumoBrowser(headless=True) as sumo: - data = sumo.fetch_json("https://support.mozilla.org/api/2/question/?...") + data = sumo.fetch_json(sumo_url) for page in sumo.paginate(first_url): ... """ @@ -87,8 +90,9 @@ def __init__(self, headless=False, settle_ms=6000, max_attempts=5, backoff_base=2.0, retry_jitter_min_s=60, retry_jitter_max_s=300, max_429_wait_s=None): - self.headless = headless - self.settle_ms = settle_ms + # `headless` and `settle_ms` are accepted for backwards-compatible call + # sites but ignored: there is no browser to run headed/headless and + # nothing to "settle" for a plain HTTP client. self.max_attempts = max_attempts self.backoff_base = backoff_base # If set, a 429 whose required wait (Retry-After / backoff) exceeds this @@ -100,72 +104,52 @@ def __init__(self, headless=False, settle_ms=6000, # always retry strictly AFTER SUMO's Retry-After window and desync retries. self.retry_jitter_min_s = retry_jitter_min_s self.retry_jitter_max_s = retry_jitter_max_s - self._pw = None - self._browser = None - self._page = None + self._client = None def __enter__(self): - self._pw = sync_playwright().start() - # --disable-quic: SUMO's CDN advertises HTTP/3, and Chromium can stall - # indefinitely on the QUIC/UDP path on some networks (while curl over - # TCP is fine); forcing HTTP over TCP avoids that hang. - self._browser = self._pw.chromium.launch( - headless=self.headless, args=["--disable-quic"]) - context = self._browser.new_context( - user_agent=USER_AGENT, - locale="en-US", - viewport={"width": 1280, "height": 800}, + # A persistent client keeps a cookie jar across calls, so any edge + # cookie handed out on the first request is reused (mirrors the old + # browser context's cookie reuse). follow_redirects matches a browser. + self._client = httpx.Client( + headers={ + "User-Agent": USER_AGENT, + "Accept-Language": "en-US,en;q=0.9", + }, + timeout=60.0, + follow_redirects=True, ) - # Acquire challenge cookies by loading the site once. This load can - # transiently stall (SUMO intermittently tarpits automated clients), so - # retry with a fresh page + backoff rather than letting one timeout kill - # the whole per-day scrape — a sustained stall window otherwise fails - # every day in a backfill (the "cascade" failures seen in practice). - last_err = None - for attempt in range(1, self.max_attempts + 1): - self._page = context.new_page() - try: - self._page.goto(HOME_URL, wait_until="domcontentloaded", - timeout=60000) - self._page.wait_for_timeout(self.settle_ms) - return self - except PWTimeoutError as e: - last_err = e - print(f"home load timed out " - f"(attempt {attempt}/{self.max_attempts})", - file=sys.stderr, flush=True) - self._page.close() - if attempt < self.max_attempts: - time.sleep(self.backoff_base ** attempt) - raise last_err + # Warm up by loading the home page once so any cookie is captured before + # the API call. Best-effort: a failure here is not fatal — a genuine + # block surfaces later as ChallengeError from fetch_json. + try: + self._client.get(HOME_URL) + except httpx.HTTPError as e: + print(f"home warm-up failed (continuing): {e}", + file=sys.stderr, flush=True) + return self def __exit__(self, exc_type, exc, tb): - if self._browser is not None: - self._browser.close() - if self._pw is not None: - self._pw.stop() + if self._client is not None: + self._client.close() return False def _raw_fetch(self, url): - """Do one in-page fetch(); return {status, json, snippet, retry_after}.""" - return self._page.evaluate( - """async (url) => { - const res = await fetch(url, { - headers: { 'Accept': 'application/json' }, - credentials: 'include', - }); - const text = await res.text(); - let json = null; - try { json = JSON.parse(text); } catch (e) {} - return { - status: res.status, - json, - snippet: text.slice(0, 300), - retry_after: res.headers.get('retry-after'), - }; - }""", - url, - ) + """Do one HTTP GET; return {status, json, snippet, retry_after}. + + Parses the body as JSON the same way the old in-page fetch did (json is + None when the body is not valid JSON — e.g. an HTML challenge page).""" + res = self._client.get(url, headers={"Accept": "application/json"}) + text = res.text + try: + body = json.loads(text) + except ValueError: + body = None + return { + "status": res.status_code, + "json": body, + "snippet": text[:300], + "retry_after": res.headers.get("retry-after"), + } def fetch_json(self, url): """Fetch `url` via an in-page fetch(), retrying transient failures. diff --git a/tests/test_fetch_json.py b/tests/test_fetch_json.py index 72ad599..cbce8cf 100644 --- a/tests/test_fetch_json.py +++ b/tests/test_fetch_json.py @@ -67,3 +67,31 @@ def test_429_under_threshold_retries_then_succeeds(monkeypatch): [_resp(429, retry_after="10"), _resp(200, {"ok": 1})], max_429_wait_s=1000) assert sb.fetch_json("u") == {"ok": 1} + + +def test_raw_fetch_maps_httpx_response(): + """_raw_fetch turns an httpx response into the dict fetch_json expects.""" + import httpx + + def handler(request): + return httpx.Response(429, headers={"retry-after": "30"}, + text='{"a": 1}') + + sb = sumo.SumoBrowser() + sb._client = httpx.Client(transport=httpx.MockTransport(handler)) + out = sb._raw_fetch("https://support.mozilla.org/api/2/question/") + assert out == {"status": 429, "json": {"a": 1}, + "snippet": '{"a": 1}', "retry_after": "30"} + + +def test_raw_fetch_non_json_body_sets_json_none(): + import httpx + + def handler(request): + return httpx.Response(200, text="challenge") + + sb = sumo.SumoBrowser() + sb._client = httpx.Client(transport=httpx.MockTransport(handler)) + out = sb._raw_fetch("https://support.mozilla.org/api/2/question/") + assert out["status"] == 200 and out["json"] is None + assert out["snippet"] == "challenge" diff --git a/uv.lock b/uv.lock index 51e59c6..9067adc 100644 --- a/uv.lock +++ b/uv.lock @@ -7,7 +7,7 @@ name = "aaq-scraper" version = "0.1.0" source = { virtual = "." } dependencies = [ - { name = "playwright" }, + { name = "httpx" }, ] [package.dev-dependencies] @@ -16,11 +16,34 @@ dev = [ ] [package.metadata] -requires-dist = [{ name = "playwright", specifier = "==1.49.1" }] +requires-dist = [{ name = "httpx", specifier = "==0.28.1" }] [package.metadata.requires-dev] dev = [{ name = "pytest", specifier = ">=8" }] +[[package]] +name = "anyio" +version = "4.14.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "exceptiongroup", marker = "python_full_version < '3.11'" }, + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, +] + +[[package]] +name = "certifi" +version = "2026.6.17" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c9/c7/424b75da314c1045981bd9777432fad05a9e0c69daa4ed7e308bbaffe405/certifi-2026.6.17.tar.gz", hash = "sha256:024c88eeec92ca068db80f02b8b07c9cef7b9fe261d1d535abfd5abd6f6af432", size = 134594, upload-time = "2026-06-17T10:31:07.894Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ef/2f/c5464532e965badff2f4c4c1a3a83f5697f0d7c407ed0cda44aaa99bb451/certifi-2026.6.17-py3-none-any.whl", hash = "sha256:2227dcbaafe0d2f59279d1762ddddc37783ed4354594f194ffc31d20f41fc3db", size = 133289, upload-time = "2026-06-17T10:31:06.348Z" }, +] + [[package]] name = "colorama" version = "0.4.6" @@ -43,54 +66,49 @@ wheels = [ ] [[package]] -name = "greenlet" -version = "3.1.1" +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/2f/ff/df5fede753cc10f6a5be0931204ea30c35fa2f2ea7a35b25bdaf4fe40e46/greenlet-3.1.1.tar.gz", hash = "sha256:4ce3ac6cdb6adf7946475d7ef31777c26d94bccc377e070a7986bd2d5c515467", size = 186022, upload-time = "2024-09-20T18:21:04.506Z" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/25/90/5234a78dc0ef6496a6eb97b67a42a8e96742a56f7dc808cb954a85390448/greenlet-3.1.1-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:0bbae94a29c9e5c7e4a2b7f0aae5c17e8e90acbfd3bf6270eeba60c39fce3563", size = 271235, upload-time = "2024-09-20T17:07:18.761Z" }, - { url = "https://files.pythonhosted.org/packages/7c/16/cd631fa0ab7d06ef06387135b7549fdcc77d8d859ed770a0d28e47b20972/greenlet-3.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0fde093fb93f35ca72a556cf72c92ea3ebfda3d79fc35bb19fbe685853869a83", size = 637168, upload-time = "2024-09-20T17:36:43.774Z" }, - { url = "https://files.pythonhosted.org/packages/2f/b1/aed39043a6fec33c284a2c9abd63ce191f4f1a07319340ffc04d2ed3256f/greenlet-3.1.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:36b89d13c49216cadb828db8dfa6ce86bbbc476a82d3a6c397f0efae0525bdd0", size = 648826, upload-time = "2024-09-20T17:39:16.921Z" }, - { url = "https://files.pythonhosted.org/packages/76/25/40e0112f7f3ebe54e8e8ed91b2b9f970805143efef16d043dfc15e70f44b/greenlet-3.1.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:94b6150a85e1b33b40b1464a3f9988dcc5251d6ed06842abff82e42632fac120", size = 644443, upload-time = "2024-09-20T17:44:21.896Z" }, - { url = "https://files.pythonhosted.org/packages/fb/2f/3850b867a9af519794784a7eeed1dd5bc68ffbcc5b28cef703711025fd0a/greenlet-3.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:93147c513fac16385d1036b7e5b102c7fbbdb163d556b791f0f11eada7ba65dc", size = 643295, upload-time = "2024-09-20T17:08:37.951Z" }, - { url = "https://files.pythonhosted.org/packages/cf/69/79e4d63b9387b48939096e25115b8af7cd8a90397a304f92436bcb21f5b2/greenlet-3.1.1-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:da7a9bff22ce038e19bf62c4dd1ec8391062878710ded0a845bcf47cc0200617", size = 599544, upload-time = "2024-09-20T17:08:27.894Z" }, - { url = "https://files.pythonhosted.org/packages/46/1d/44dbcb0e6c323bd6f71b8c2f4233766a5faf4b8948873225d34a0b7efa71/greenlet-3.1.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:b2795058c23988728eec1f36a4e5e4ebad22f8320c85f3587b539b9ac84128d7", size = 1125456, upload-time = "2024-09-20T17:44:11.755Z" }, - { url = "https://files.pythonhosted.org/packages/e0/1d/a305dce121838d0278cee39d5bb268c657f10a5363ae4b726848f833f1bb/greenlet-3.1.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:ed10eac5830befbdd0c32f83e8aa6288361597550ba669b04c48f0f9a2c843c6", size = 1149111, upload-time = "2024-09-20T17:09:22.104Z" }, - { url = "https://files.pythonhosted.org/packages/96/28/d62835fb33fb5652f2e98d34c44ad1a0feacc8b1d3f1aecab035f51f267d/greenlet-3.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:77c386de38a60d1dfb8e55b8c1101d68c79dfdd25c7095d51fec2dd800892b80", size = 298392, upload-time = "2024-09-20T17:28:51.988Z" }, - { url = "https://files.pythonhosted.org/packages/28/62/1c2665558618553c42922ed47a4e6d6527e2fa3516a8256c2f431c5d0441/greenlet-3.1.1-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:e4d333e558953648ca09d64f13e6d8f0523fa705f51cae3f03b5983489958c70", size = 272479, upload-time = "2024-09-20T17:07:22.332Z" }, - { url = "https://files.pythonhosted.org/packages/76/9d/421e2d5f07285b6e4e3a676b016ca781f63cfe4a0cd8eaecf3fd6f7a71ae/greenlet-3.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:09fc016b73c94e98e29af67ab7b9a879c307c6731a2c9da0db5a7d9b7edd1159", size = 640404, upload-time = "2024-09-20T17:36:45.588Z" }, - { url = "https://files.pythonhosted.org/packages/e5/de/6e05f5c59262a584e502dd3d261bbdd2c97ab5416cc9c0b91ea38932a901/greenlet-3.1.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d5e975ca70269d66d17dd995dafc06f1b06e8cb1ec1e9ed54c1d1e4a7c4cf26e", size = 652813, upload-time = "2024-09-20T17:39:19.052Z" }, - { url = "https://files.pythonhosted.org/packages/49/93/d5f93c84241acdea15a8fd329362c2c71c79e1a507c3f142a5d67ea435ae/greenlet-3.1.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3b2813dc3de8c1ee3f924e4d4227999285fd335d1bcc0d2be6dc3f1f6a318ec1", size = 648517, upload-time = "2024-09-20T17:44:24.101Z" }, - { url = "https://files.pythonhosted.org/packages/15/85/72f77fc02d00470c86a5c982b8daafdf65d38aefbbe441cebff3bf7037fc/greenlet-3.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e347b3bfcf985a05e8c0b7d462ba6f15b1ee1c909e2dcad795e49e91b152c383", size = 647831, upload-time = "2024-09-20T17:08:40.577Z" }, - { url = "https://files.pythonhosted.org/packages/f7/4b/1c9695aa24f808e156c8f4813f685d975ca73c000c2a5056c514c64980f6/greenlet-3.1.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9e8f8c9cb53cdac7ba9793c276acd90168f416b9ce36799b9b885790f8ad6c0a", size = 602413, upload-time = "2024-09-20T17:08:31.728Z" }, - { url = "https://files.pythonhosted.org/packages/76/70/ad6e5b31ef330f03b12559d19fda2606a522d3849cde46b24f223d6d1619/greenlet-3.1.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:62ee94988d6b4722ce0028644418d93a52429e977d742ca2ccbe1c4f4a792511", size = 1129619, upload-time = "2024-09-20T17:44:14.222Z" }, - { url = "https://files.pythonhosted.org/packages/f4/fb/201e1b932e584066e0f0658b538e73c459b34d44b4bd4034f682423bc801/greenlet-3.1.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:1776fd7f989fc6b8d8c8cb8da1f6b82c5814957264d1f6cf818d475ec2bf6395", size = 1155198, upload-time = "2024-09-20T17:09:23.903Z" }, - { url = "https://files.pythonhosted.org/packages/12/da/b9ed5e310bb8b89661b80cbcd4db5a067903bbcd7fc854923f5ebb4144f0/greenlet-3.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:48ca08c771c268a768087b408658e216133aecd835c0ded47ce955381105ba39", size = 298930, upload-time = "2024-09-20T17:25:18.656Z" }, - { url = "https://files.pythonhosted.org/packages/7d/ec/bad1ac26764d26aa1353216fcbfa4670050f66d445448aafa227f8b16e80/greenlet-3.1.1-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:4afe7ea89de619adc868e087b4d2359282058479d7cfb94970adf4b55284574d", size = 274260, upload-time = "2024-09-20T17:08:07.301Z" }, - { url = "https://files.pythonhosted.org/packages/66/d4/c8c04958870f482459ab5956c2942c4ec35cac7fe245527f1039837c17a9/greenlet-3.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f406b22b7c9a9b4f8aa9d2ab13d6ae0ac3e85c9a809bd590ad53fed2bf70dc79", size = 649064, upload-time = "2024-09-20T17:36:47.628Z" }, - { url = "https://files.pythonhosted.org/packages/51/41/467b12a8c7c1303d20abcca145db2be4e6cd50a951fa30af48b6ec607581/greenlet-3.1.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c3a701fe5a9695b238503ce5bbe8218e03c3bcccf7e204e455e7462d770268aa", size = 663420, upload-time = "2024-09-20T17:39:21.258Z" }, - { url = "https://files.pythonhosted.org/packages/27/8f/2a93cd9b1e7107d5c7b3b7816eeadcac2ebcaf6d6513df9abaf0334777f6/greenlet-3.1.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2846930c65b47d70b9d178e89c7e1a69c95c1f68ea5aa0a58646b7a96df12441", size = 658035, upload-time = "2024-09-20T17:44:26.501Z" }, - { url = "https://files.pythonhosted.org/packages/57/5c/7c6f50cb12be092e1dccb2599be5a942c3416dbcfb76efcf54b3f8be4d8d/greenlet-3.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:99cfaa2110534e2cf3ba31a7abcac9d328d1d9f1b95beede58294a60348fba36", size = 660105, upload-time = "2024-09-20T17:08:42.048Z" }, - { url = "https://files.pythonhosted.org/packages/f1/66/033e58a50fd9ec9df00a8671c74f1f3a320564c6415a4ed82a1c651654ba/greenlet-3.1.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1443279c19fca463fc33e65ef2a935a5b09bb90f978beab37729e1c3c6c25fe9", size = 613077, upload-time = "2024-09-20T17:08:33.707Z" }, - { url = "https://files.pythonhosted.org/packages/19/c5/36384a06f748044d06bdd8776e231fadf92fc896bd12cb1c9f5a1bda9578/greenlet-3.1.1-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:b7cede291382a78f7bb5f04a529cb18e068dd29e0fb27376074b6d0317bf4dd0", size = 1135975, upload-time = "2024-09-20T17:44:15.989Z" }, - { url = "https://files.pythonhosted.org/packages/38/f9/c0a0eb61bdf808d23266ecf1d63309f0e1471f284300ce6dac0ae1231881/greenlet-3.1.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:23f20bb60ae298d7d8656c6ec6db134bca379ecefadb0b19ce6f19d1f232a942", size = 1163955, upload-time = "2024-09-20T17:09:25.539Z" }, - { url = "https://files.pythonhosted.org/packages/43/21/a5d9df1d21514883333fc86584c07c2b49ba7c602e670b174bd73cfc9c7f/greenlet-3.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:7124e16b4c55d417577c2077be379514321916d5790fa287c9ed6f23bd2ffd01", size = 299655, upload-time = "2024-09-20T17:21:22.427Z" }, - { url = "https://files.pythonhosted.org/packages/f3/57/0db4940cd7bb461365ca8d6fd53e68254c9dbbcc2b452e69d0d41f10a85e/greenlet-3.1.1-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:05175c27cb459dcfc05d026c4232f9de8913ed006d42713cb8a5137bd49375f1", size = 272990, upload-time = "2024-09-20T17:08:26.312Z" }, - { url = "https://files.pythonhosted.org/packages/1c/ec/423d113c9f74e5e402e175b157203e9102feeb7088cee844d735b28ef963/greenlet-3.1.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:935e943ec47c4afab8965954bf49bfa639c05d4ccf9ef6e924188f762145c0ff", size = 649175, upload-time = "2024-09-20T17:36:48.983Z" }, - { url = "https://files.pythonhosted.org/packages/a9/46/ddbd2db9ff209186b7b7c621d1432e2f21714adc988703dbdd0e65155c77/greenlet-3.1.1-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:667a9706c970cb552ede35aee17339a18e8f2a87a51fba2ed39ceeeb1004798a", size = 663425, upload-time = "2024-09-20T17:39:22.705Z" }, - { url = "https://files.pythonhosted.org/packages/bc/f9/9c82d6b2b04aa37e38e74f0c429aece5eeb02bab6e3b98e7db89b23d94c6/greenlet-3.1.1-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b8a678974d1f3aa55f6cc34dc480169d58f2e6d8958895d68845fa4ab566509e", size = 657736, upload-time = "2024-09-20T17:44:28.544Z" }, - { url = "https://files.pythonhosted.org/packages/d9/42/b87bc2a81e3a62c3de2b0d550bf91a86939442b7ff85abb94eec3fc0e6aa/greenlet-3.1.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:efc0f674aa41b92da8c49e0346318c6075d734994c3c4e4430b1c3f853e498e4", size = 660347, upload-time = "2024-09-20T17:08:45.56Z" }, - { url = "https://files.pythonhosted.org/packages/37/fa/71599c3fd06336cdc3eac52e6871cfebab4d9d70674a9a9e7a482c318e99/greenlet-3.1.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0153404a4bb921f0ff1abeb5ce8a5131da56b953eda6e14b88dc6bbc04d2049e", size = 615583, upload-time = "2024-09-20T17:08:36.85Z" }, - { url = "https://files.pythonhosted.org/packages/4e/96/e9ef85de031703ee7a4483489b40cf307f93c1824a02e903106f2ea315fe/greenlet-3.1.1-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:275f72decf9932639c1c6dd1013a1bc266438eb32710016a1c742df5da6e60a1", size = 1133039, upload-time = "2024-09-20T17:44:18.287Z" }, - { url = "https://files.pythonhosted.org/packages/87/76/b2b6362accd69f2d1889db61a18c94bc743e961e3cab344c2effaa4b4a25/greenlet-3.1.1-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:c4aab7f6381f38a4b42f269057aee279ab0fc7bf2e929e3d4abfae97b682a12c", size = 1160716, upload-time = "2024-09-20T17:09:27.112Z" }, - { url = "https://files.pythonhosted.org/packages/1f/1b/54336d876186920e185066d8c3024ad55f21d7cc3683c856127ddb7b13ce/greenlet-3.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:b42703b1cf69f2aa1df7d1030b9d77d3e584a70755674d60e710f0af570f3761", size = 299490, upload-time = "2024-09-20T17:17:09.501Z" }, - { url = "https://files.pythonhosted.org/packages/5f/17/bea55bf36990e1638a2af5ba10c1640273ef20f627962cf97107f1e5d637/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f1695e76146579f8c06c1509c7ce4dfe0706f49c6831a817ac04eebb2fd02011", size = 643731, upload-time = "2024-09-20T17:36:50.376Z" }, - { url = "https://files.pythonhosted.org/packages/78/d2/aa3d2157f9ab742a08e0fd8f77d4699f37c22adfbfeb0c610a186b5f75e0/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7876452af029456b3f3549b696bb36a06db7c90747740c5302f74a9e9fa14b13", size = 649304, upload-time = "2024-09-20T17:39:24.55Z" }, - { url = "https://files.pythonhosted.org/packages/f1/8e/d0aeffe69e53ccff5a28fa86f07ad1d2d2d6537a9506229431a2a02e2f15/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4ead44c85f8ab905852d3de8d86f6f8baf77109f9da589cb4fa142bd3b57b475", size = 646537, upload-time = "2024-09-20T17:44:31.102Z" }, - { url = "https://files.pythonhosted.org/packages/05/79/e15408220bbb989469c8871062c97c6c9136770657ba779711b90870d867/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8320f64b777d00dd7ccdade271eaf0cad6636343293a25074cc5566160e4de7b", size = 642506, upload-time = "2024-09-20T17:08:47.852Z" }, - { url = "https://files.pythonhosted.org/packages/18/87/470e01a940307796f1d25f8167b551a968540fbe0551c0ebb853cb527dd6/greenlet-3.1.1-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6510bf84a6b643dabba74d3049ead221257603a253d0a9873f55f6a59a65f822", size = 602753, upload-time = "2024-09-20T17:08:38.079Z" }, - { url = "https://files.pythonhosted.org/packages/e2/72/576815ba674eddc3c25028238f74d7b8068902b3968cbe456771b166455e/greenlet-3.1.1-cp313-cp313t-musllinux_1_1_aarch64.whl", hash = "sha256:04b013dc07c96f83134b1e99888e7a79979f1a247e2a9f59697fa14b5862ed01", size = 1122731, upload-time = "2024-09-20T17:44:20.556Z" }, - { url = "https://files.pythonhosted.org/packages/ac/38/08cc303ddddc4b3d7c628c3039a61a3aae36c241ed01393d00c2fd663473/greenlet-3.1.1-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:411f015496fec93c1c8cd4e5238da364e1da7a124bcb293f085bf2860c32c6f6", size = 1142112, upload-time = "2024-09-20T17:09:28.753Z" }, + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "idna" +version = "3.18" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/63/9496c57188a2ee585e0f1db071d75089a11e98aa86eb99d9d7618fc1edce/idna-3.18.tar.gz", hash = "sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848", size = 196711, upload-time = "2026-06-02T14:34:07.794Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/5e/d4e9f1a599fb8e573b7b87160658329fbf28d19eac2718f51fc3def3aa5a/idna-3.18-py3-none-any.whl", hash = "sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2", size = 65455, upload-time = "2026-06-02T14:34:06.319Z" }, ] [[package]] @@ -111,24 +129,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, ] -[[package]] -name = "playwright" -version = "1.49.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "greenlet" }, - { name = "pyee" }, -] -wheels = [ - { url = "https://files.pythonhosted.org/packages/ab/be/01025581052e43eb698092c4328d7497ca62bcb5c83f15a611d4a71b4b92/playwright-1.49.1-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:1041ffb45a0d0bc44d698d3a5aa3ac4b67c9bd03540da43a0b70616ad52592b8", size = 39559859, upload-time = "2024-12-10T17:32:14.907Z" }, - { url = "https://files.pythonhosted.org/packages/79/25/ef1010a42cc7d576282015d983c5451d73e369b198b6eb32a177fae281f8/playwright-1.49.1-py3-none-macosx_11_0_arm64.whl", hash = "sha256:9f38ed3d0c1f4e0a6d1c92e73dd9a61f8855133249d6f0cec28648d38a7137be", size = 38808973, upload-time = "2024-12-10T17:32:22.516Z" }, - { url = "https://files.pythonhosted.org/packages/70/4b/3930cf10f303a10d493a382e4448aaff898b4065698b3b8d92f902e53e08/playwright-1.49.1-py3-none-macosx_11_0_universal2.whl", hash = "sha256:3be48c6d26dc819ca0a26567c1ae36a980a0303dcd4249feb6f59e115aaddfb8", size = 39559863, upload-time = "2024-12-10T17:32:29.12Z" }, - { url = "https://files.pythonhosted.org/packages/9a/c1/ea765e72a746dc7ec2ce155ffea29d454e7171db78f3c09185e888387246/playwright-1.49.1-py3-none-manylinux1_x86_64.whl", hash = "sha256:753ca90ee31b4b03d165cfd36e477309ebf2b4381953f2a982ff612d85b147d2", size = 44163300, upload-time = "2024-12-10T17:32:35.647Z" }, - { url = "https://files.pythonhosted.org/packages/5a/52/95efac704bf36b770a2522d88a6dee298042845d10bfb35f7ca0fcc36d91/playwright-1.49.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cd9bc8dab37aa25198a01f555f0a2e2c3813fe200fef018ac34dfe86b34994b9", size = 43744353, upload-time = "2024-12-10T17:32:43.189Z" }, - { url = "https://files.pythonhosted.org/packages/f9/97/a3fccc9aaa6da83890772e9980703b0ea6b1e1ad42042fb50df3aef6c641/playwright-1.49.1-py3-none-win32.whl", hash = "sha256:43b304be67f096058e587dac453ece550eff87b8fbed28de30f4f022cc1745bb", size = 34060663, upload-time = "2024-12-10T17:32:49.904Z" }, - { url = "https://files.pythonhosted.org/packages/71/a9/bd88ac0bd498c91aab3aba2e393d1fa59f72a7243e9265ccbf4861ca4f64/playwright-1.49.1-py3-none-win_amd64.whl", hash = "sha256:47b23cb346283278f5b4d1e1990bcb6d6302f80c0aa0ca93dd0601a1400191df", size = 34060667, upload-time = "2024-12-10T17:32:56.459Z" }, -] - [[package]] name = "pluggy" version = "1.6.0" @@ -138,18 +138,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, ] -[[package]] -name = "pyee" -version = "12.0.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/d2/a7/8faaa62a488a2a1e0d56969757f087cbd2729e9bcfa508c230299f366b4c/pyee-12.0.0.tar.gz", hash = "sha256:c480603f4aa2927d4766eb41fa82793fe60a82cbfdb8d688e0d08c55a534e145", size = 29675, upload-time = "2024-08-30T19:40:43.555Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/1d/0d/95993c08c721ec68892547f2117e8f9dfbcef2ca71e098533541b4a54d5f/pyee-12.0.0-py3-none-any.whl", hash = "sha256:7b14b74320600049ccc7d0e0b1becd3b4bd0a03c745758225e31a59f4095c990", size = 14831, upload-time = "2024-08-30T19:40:42.132Z" }, -] - [[package]] name = "pygments" version = "2.20.0" From fb04c94fdd7b0ec5e025be46dd7958838c855309 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 11:53:40 -0700 Subject: [PATCH 06/17] build: browser-free arm64 image for the scraper CronJob --- .dockerignore | 8 ++++++++ Dockerfile | 31 +++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 .dockerignore create mode 100644 Dockerfile diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..a9147ce --- /dev/null +++ b/.dockerignore @@ -0,0 +1,8 @@ +.git +.venv +__pycache__ +*.pyc +20*/ +docs/ +backfill-reports/ +tests/ diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..32936b8 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,31 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. + +# Browser-free image: the old Playwright/Chromium approach is dead (issue #28); +# this runs the plain-httpx scraper. git + a shell are needed at runtime because +# the entrypoint clones aaq-scraper, runs the refresh, and commits CSVs back. +FROM python:3.12-slim + +# uv for dependency management (matches local/CI: uv sync / uv run). +COPY --from=ghcr.io/astral-sh/uv:0.5.11 /uv /usr/local/bin/uv + +RUN apt-get update \ + && apt-get install -y --no-install-recommends git ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app +COPY pyproject.toml uv.lock ./ +RUN uv sync --frozen --no-dev + +COPY . . + +# Activate venv by adding it to PATH so `python` resolves to the venv's interpreter. +ENV PATH="/app/.venv/bin:$PATH" + +# Non-root. The clone/commit workdir is a writable emptyDir mounted at runtime +# (the root FS is read-only in the pod), so this user only needs to read /app. +RUN useradd --create-home --uid 65532 appuser +USER 65532 + +CMD ["python", "-c", "import sumo; print('image ok')"] From 2b03e7f9520900eb3da9eba08e0a45ce14f6a268 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 11:56:22 -0700 Subject: [PATCH 07/17] chore: add MPL header to .dockerignore (review follow-up) Co-Authored-By: Claude Opus 4.8 (1M context) --- .dockerignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.dockerignore b/.dockerignore index a9147ce..226dab4 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,3 +1,6 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. .git .venv __pycache__ From 917ff8231357e79d5c92abd783dec9ea1f838078 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 11:58:23 -0700 Subject: [PATCH 08/17] feat: pod entrypoint clones, refreshes, and commits CSVs back to git --- Dockerfile | 2 +- deploy/entrypoint.sh | 63 +++++++++++++++++++++++++++++++ tests/test_entrypoint.py | 81 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 145 insertions(+), 1 deletion(-) create mode 100755 deploy/entrypoint.sh create mode 100644 tests/test_entrypoint.py diff --git a/Dockerfile b/Dockerfile index 32936b8..f4fd557 100644 --- a/Dockerfile +++ b/Dockerfile @@ -28,4 +28,4 @@ ENV PATH="/app/.venv/bin:$PATH" RUN useradd --create-home --uid 65532 appuser USER 65532 -CMD ["python", "-c", "import sumo; print('image ok')"] +ENTRYPOINT ["deploy/entrypoint.sh"] diff --git a/deploy/entrypoint.sh b/deploy/entrypoint.sh new file mode 100755 index 0000000..4825cba --- /dev/null +++ b/deploy/entrypoint.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +# +# Pod entrypoint: clone aaq-scraper, run the incremental refresh, commit changed +# CSVs (and the high-water mark) back to the branch. Mirrors the logic that used +# to live in .github/workflows/scrape.yml (checkout + run_refresh + commit with +# rebase-onto-latest-main retry). The pod is stateless; the repo IS the state. +set -euo pipefail + +: "${GIT_REPO_URL:?set GIT_REPO_URL}" +: "${GITHUB_TOKEN:?set GITHUB_TOKEN}" +GIT_BRANCH="${GIT_BRANCH:-main}" +GIT_AUTHOR_NAME="${GIT_AUTHOR_NAME:-aaq-scraper-bot}" +GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-aaq-scraper-bot@thunderbird.net}" +REFRESH_ARGS="${REFRESH_ARGS:---soft-deadline 40 --max-429-wait 120}" + +WORKDIR="$(mktemp -d)" +# Embed the token in the clone URL (https://x-access-token:TOKEN@github.com/...). +AUTH_URL="$(printf '%s' "$GIT_REPO_URL" | sed -E "s#https://#https://x-access-token:${GITHUB_TOKEN}@#")" + +git clone --branch "$GIT_BRANCH" "$AUTH_URL" "$WORKDIR/repo" +cd "$WORKDIR/repo" +git config user.name "$GIT_AUTHOR_NAME" +git config user.email "$GIT_AUTHOR_EMAIL" + +# Run the refresh. REFRESH_CMD lets tests inject a fake; default is the real one. +# Preserve the deferral exit code (75) as a non-error signal. +set +e +if [ -n "${REFRESH_CMD:-}" ]; then + eval "$REFRESH_CMD" +else + uv run python run_refresh.py $REFRESH_ARGS +fi +rc=$? +set -e +if [ "$rc" -eq 75 ]; then + echo "refresh deferred (exit 75); committing whatever completed" +elif [ "$rc" -ne 0 ]; then + echo "refresh failed (exit $rc)"; exit "$rc" +fi + +git add 20*/ .refresh-hwm 2>/dev/null || git add 20*/ +if git diff --cached --quiet; then + echo "No changes to commit." + exit 0 +fi +git commit -m "Hourly refresh $(date -u +%Y-%m-%dT%H:%MZ)" + +# main also receives pushes from manual backfills, so rebase onto latest and +# retry with backoff rather than failing on a rejected push. +for attempt in 1 2 3 4 5; do + if ! git pull --rebase origin "$GIT_BRANCH"; then + git rebase --abort || true + echo "rebase failed (attempt $attempt/5); retrying"; sleep $((attempt * 5)); continue + fi + if git push origin "$GIT_BRANCH"; then + echo "pushed on attempt $attempt"; exit 0 + fi + echo "push rejected (attempt $attempt/5); retrying"; sleep $((attempt * 5)) +done +echo "ERROR: could not push after 5 attempts"; exit 1 diff --git a/tests/test_entrypoint.py b/tests/test_entrypoint.py new file mode 100644 index 0000000..26127a4 --- /dev/null +++ b/tests/test_entrypoint.py @@ -0,0 +1,81 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +"""Integration test for deploy/entrypoint.sh using a local bare repo. + +Exercises the clone -> (fake) refresh -> commit -> push loop without a network +or a real token. The token is stripped from a file:// URL by git, so a dummy +value is fine.""" +import os +import subprocess +from pathlib import Path + + +def _run(cmd, cwd=None, env=None): + return subprocess.run(cmd, cwd=cwd, env=env, check=True, + capture_output=True, text=True) + + +def _git_env(): + env = dict(os.environ) + env.update({ + "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@e", + "GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@e", + }) + return env + + +def test_entrypoint_commits_new_csv(tmp_path): + genv = _git_env() + # A bare "origin" with one commit on main containing an empty data dir marker. + origin = tmp_path / "origin.git" + _run(["git", "init", "--bare", "-b", "main", str(origin)], env=genv) + seed = tmp_path / "seed" + _run(["git", "clone", str(origin), str(seed)], env=genv) + (seed / "2099").mkdir() + (seed / "2099" / ".keep").write_text("") + _run(["git", "add", "-A"], cwd=seed, env=genv) + _run(["git", "commit", "-m", "seed"], cwd=seed, env=genv) + _run(["git", "push", "origin", "main"], cwd=seed, env=genv) + + repo_root = Path(__file__).resolve().parent.parent + env = _git_env() + env.update({ + "GIT_REPO_URL": f"file://{origin}", + "GITHUB_TOKEN": "dummy", + "GIT_BRANCH": "main", + # Fake refresh: write a CSV under 2099/ so there is something to commit. + "REFRESH_CMD": "echo id > 2099/questions-test-2099-01-01.csv", + }) + _run(["bash", str(repo_root / "deploy" / "entrypoint.sh")], env=env) + + # The bare origin now has the new file on main. + log = _run(["git", "log", "--oneline", "-1", "main"], cwd=origin, env=genv) + assert "Hourly refresh" in log.stdout + show = _run(["git", "show", + "main:2099/questions-test-2099-01-01.csv"], cwd=origin, env=genv) + assert show.stdout.strip() == "id" + + +def test_entrypoint_no_changes_no_commit(tmp_path): + genv = _git_env() + origin = tmp_path / "origin.git" + _run(["git", "init", "--bare", "-b", "main", str(origin)], env=genv) + seed = tmp_path / "seed" + _run(["git", "clone", str(origin), str(seed)], env=genv) + (seed / "2099").mkdir() + (seed / "2099" / ".keep").write_text("") + _run(["git", "add", "-A"], cwd=seed, env=genv) + _run(["git", "commit", "-m", "seed"], cwd=seed, env=genv) + _run(["git", "push", "origin", "main"], cwd=seed, env=genv) + + repo_root = Path(__file__).resolve().parent.parent + env = _git_env() + env.update({ + "GIT_REPO_URL": f"file://{origin}", "GITHUB_TOKEN": "dummy", + "GIT_BRANCH": "main", "REFRESH_CMD": "true", # writes nothing + }) + _run(["bash", str(repo_root / "deploy" / "entrypoint.sh")], env=env) + + count = _run(["git", "rev-list", "--count", "main"], cwd=origin, env=genv) + assert count.stdout.strip() == "1" # still just the seed commit From 63d64c8b50b9d7b207532dec5afc480dfa2e21f6 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:06:30 -0700 Subject: [PATCH 09/17] fix(entrypoint): supply PAT via credential helper (not argv); test rebase-retry path --- deploy/entrypoint.sh | 13 ++++++++--- tests/test_entrypoint.py | 50 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 3 deletions(-) diff --git a/deploy/entrypoint.sh b/deploy/entrypoint.sh index 4825cba..d58831e 100755 --- a/deploy/entrypoint.sh +++ b/deploy/entrypoint.sh @@ -17,11 +17,18 @@ GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-aaq-scraper-bot@thunderbird.net}" REFRESH_ARGS="${REFRESH_ARGS:---soft-deadline 40 --max-429-wait 120}" WORKDIR="$(mktemp -d)" -# Embed the token in the clone URL (https://x-access-token:TOKEN@github.com/...). -AUTH_URL="$(printf '%s' "$GIT_REPO_URL" | sed -E "s#https://#https://x-access-token:${GITHUB_TOKEN}@#")" +trap 'rm -rf "$WORKDIR"' EXIT -git clone --branch "$GIT_BRANCH" "$AUTH_URL" "$WORKDIR/repo" +# Supply the PAT via a credential helper that reads $GITHUB_TOKEN from the +# environment at call time, so only the variable NAME (never its value) ever +# appears in argv (visible via ps / /proc//cmdline). The clone/pull/push +# all use the plain GIT_REPO_URL -- no token spliced into the URL. +export GIT_TERMINAL_PROMPT=0 +CRED_HELPER='!f() { echo "username=x-access-token"; echo "password=${GITHUB_TOKEN}"; }; f' + +git -c credential.helper="$CRED_HELPER" clone --branch "$GIT_BRANCH" "$GIT_REPO_URL" "$WORKDIR/repo" cd "$WORKDIR/repo" +git config credential.helper "$CRED_HELPER" git config user.name "$GIT_AUTHOR_NAME" git config user.email "$GIT_AUTHOR_EMAIL" diff --git a/tests/test_entrypoint.py b/tests/test_entrypoint.py index 26127a4..7505308 100644 --- a/tests/test_entrypoint.py +++ b/tests/test_entrypoint.py @@ -79,3 +79,53 @@ def test_entrypoint_no_changes_no_commit(tmp_path): count = _run(["git", "rev-list", "--count", "main"], cwd=origin, env=genv) assert count.stdout.strip() == "1" # still just the seed commit + + +def test_entrypoint_rebase_retries_past_competing_push(tmp_path): + """A competing commit lands on origin after the entrypoint clones but + before it pushes, so the entrypoint's first push is rejected. Verify the + rebase-retry loop recovers and both commits end up on origin/main.""" + genv = _git_env() + origin = tmp_path / "origin.git" + _run(["git", "init", "--bare", "-b", "main", str(origin)], env=genv) + seed = tmp_path / "seed" + _run(["git", "clone", str(origin), str(seed)], env=genv) + (seed / "2099").mkdir() + (seed / "2099" / ".keep").write_text("") + _run(["git", "add", "-A"], cwd=seed, env=genv) + _run(["git", "commit", "-m", "seed"], cwd=seed, env=genv) + _run(["git", "push", "origin", "main"], cwd=seed, env=genv) + + repo_root = Path(__file__).resolve().parent.parent + comp = tmp_path / "competitor" + # REFRESH_CMD runs from inside the entrypoint's own clone (after it has + # already cloned origin), so pushing a competing commit here lands on + # origin *after* the entrypoint's clone but *before* its own push -- + # forcing the entrypoint's first push to be rejected. + refresh_cmd = ( + f"git clone {origin} {comp} && " + f"(cd {comp} && echo x > other.csv && git add -A && " + f"git commit -m competitor && git push origin main) && " + f"echo id > 2099/questions-test-2099-01-01.csv" + ) + env = _git_env() + env.update({ + "GIT_REPO_URL": f"file://{origin}", + "GITHUB_TOKEN": "dummy", + "GIT_BRANCH": "main", + "REFRESH_CMD": refresh_cmd, + }) + _run(["bash", str(repo_root / "deploy" / "entrypoint.sh")], env=env) + + # Both the competitor's file and the entrypoint's own commit/CSV must be + # present on origin/main -- proving rebase-retry reconciled them instead + # of dropping either side. + log = _run(["git", "log", "--oneline", "-5", "main"], cwd=origin, env=genv) + assert "Hourly refresh" in log.stdout + assert "competitor" in log.stdout + + other = _run(["git", "show", "main:other.csv"], cwd=origin, env=genv) + assert other.stdout.strip() == "x" + show = _run(["git", "show", + "main:2099/questions-test-2099-01-01.csv"], cwd=origin, env=genv) + assert show.stdout.strip() == "id" From 2a61859ca7aa9cbda9e63b0af0a9097671816bd2 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:10:25 -0700 Subject: [PATCH 10/17] chore: track .refresh-hwm in git (state moves off the Actions cache) The CronJob commits the high-water mark to the repo each active run instead of the (unavailable) GitHub Actions cache. run_refresh.py's 26h lookback covers its initial absence, so no seed file is needed. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 3 --- 1 file changed, 3 deletions(-) diff --git a/.gitignore b/.gitignore index c056185..731b29c 100644 --- a/.gitignore +++ b/.gitignore @@ -11,9 +11,6 @@ poc-sample.json /answers*.csv /*.json -# Incremental-refresh high-water mark (persisted via Actions cache, not git) -/.refresh-hwm - # Month-backfill failure log (local record of dates to redo) /backfill-failures.txt From 7f812839e23ae1ed82fc9df532c10636e08fb139 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:11:15 -0700 Subject: [PATCH 11/17] ci: build and push the scraper image to ECR via OIDC --- .github/workflows/aaq-scraper-image.yml | 61 +++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 .github/workflows/aaq-scraper-image.yml diff --git a/.github/workflows/aaq-scraper-image.yml b/.github/workflows/aaq-scraper-image.yml new file mode 100644 index 0000000..f98dcc5 --- /dev/null +++ b/.github/workflows/aaq-scraper-image.yml @@ -0,0 +1,61 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at https://mozilla.org/MPL/2.0/. +# +# Build & push the scraper image to the shared ECR via GitHub OIDC. +# One-time setup (see platform-infrastructure Task B1): +# - GitHub Environment "image-aaq-scraper" (protected to main) +# - repo variable IMAGE_PUSH_ROLE_ARN = the push-only role ARN +name: Build scraper image + +on: + push: + branches: [main] + paths: + - "**.py" + - "pyproject.toml" + - "uv.lock" + - "Dockerfile" + - "deploy/entrypoint.sh" + - ".github/workflows/aaq-scraper-image.yml" + workflow_dispatch: + inputs: + tag: + description: "Explicit image tag (default git-)" + required: false + +permissions: + contents: read + id-token: write + +jobs: + build: + runs-on: ubuntu-latest + environment: image-aaq-scraper + env: + ECR: 826971876779.dkr.ecr.us-east-1.amazonaws.com + REPO: aaq-scraper + steps: + - uses: actions/checkout@v5 + - name: Compute tag + id: tag + run: | + T="${{ github.event.inputs.tag }}" + [ -n "$T" ] || T="git-$(git rev-parse --short HEAD)" + echo "tag=$T" >> "$GITHUB_OUTPUT" + - name: Configure AWS credentials (OIDC) + uses: aws-actions/configure-aws-credentials@v4 + with: + role-to-assume: ${{ vars.IMAGE_PUSH_ROLE_ARN }} + aws-region: us-east-1 + - name: Login to ECR + uses: aws-actions/amazon-ecr-login@v2 + - name: Set up Buildx + uses: docker/setup-buildx-action@v3 + - name: Build and push + uses: docker/build-push-action@v6 + with: + context: . + platforms: linux/arm64 + push: true + tags: ${{ env.ECR }}/${{ env.REPO }}:${{ steps.tag.outputs.tag }} From 8ce1b8973a5412a30156b74264abd80c38922523 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:12:12 -0700 Subject: [PATCH 12/17] ci: park schema-check (blocked by the same challenge; #28) --- .github/workflows/schema-check.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/schema-check.yml b/.github/workflows/schema-check.yml index a11705f..ecbc38a 100644 --- a/.github/workflows/schema-check.yml +++ b/.github/workflows/schema-check.yml @@ -3,9 +3,13 @@ # file, You can obtain one at https://mozilla.org/MPL/2.0/. name: API schema check +# NOTE (2026-07-13, #28): parked pending its own migration to the k8s runner. +# check_schema.py hits the live SUMO API and is blocked by the same Fastly +# challenge as the scraper, so the daily schedule is disabled to stop spurious +# api-blocked issues. Re-enable / migrate after the egress IP is allowlisted. on: - schedule: - - cron: "30 6 * * *" # daily at 06:30 UTC + # schedule: + # - cron: "30 6 * * *" # disabled — see NOTE above workflow_dispatch: permissions: From 4aef51902e0044d7a68095e0d3f5b4114b18bb1a Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:14:38 -0700 Subject: [PATCH 13/17] ci(security): harden image workflow against tag script-injection Pass the workflow_dispatch tag input via an env var instead of interpolating github.event.* into the run script, and validate its charset. Flagged HIGH (ci-cd-script-injection) by review; the job holds ECR-push OIDC creds. Behavior unchanged for the default git- path. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/aaq-scraper-image.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/aaq-scraper-image.yml b/.github/workflows/aaq-scraper-image.yml index f98dcc5..9331d81 100644 --- a/.github/workflows/aaq-scraper-image.yml +++ b/.github/workflows/aaq-scraper-image.yml @@ -39,10 +39,20 @@ jobs: - uses: actions/checkout@v5 - name: Compute tag id: tag + env: + # Pass the workflow_dispatch input through an env var rather than + # interpolating ${{ github.event.* }} directly into the run script + # (that is a shell-injection vector, since this job holds ECR-push + # OIDC creds). Then validate the tag charset before use. + TAG_INPUT: ${{ github.event.inputs.tag }} run: | - T="${{ github.event.inputs.tag }}" + T="$TAG_INPUT" [ -n "$T" ] || T="git-$(git rev-parse --short HEAD)" - echo "tag=$T" >> "$GITHUB_OUTPUT" + case "$T" in + ""|*[!A-Za-z0-9._-]*) + echo "invalid tag: $T" >&2; exit 1 ;; + esac + printf 'tag=%s\n' "$T" >> "$GITHUB_OUTPUT" - name: Configure AWS credentials (OIDC) uses: aws-actions/configure-aws-credentials@v4 with: From 9f69aa6835d1370f5e640fcb306222c8131a0c89 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:18:12 -0700 Subject: [PATCH 14/17] docs: document the k8s CronJob deployment and httpx migration --- CLAUDE.md | 23 +++++++++++++++++++++++ README.md | 18 +++++++++++++----- pyproject.toml | 2 +- tests/test_entrypoint.py | 5 +++-- 4 files changed, 40 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ddcd7e1..e4df6ff 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -184,6 +184,29 @@ vary 2–10s (`--min-delay`/`--max-delay`). Use `--headless` for CI parity. `run_refresh` instead. - Actions are pinned to Node-24 versions: `actions/checkout@v5`, `astral-sh/setup-uv@v8.2.0`. +- **k8s CronJob deployment (Phase B, in progress; ready-for-cutover, not live):** + the scraper is moving off GitHub Actions onto an ArgoCD-managed **Kubernetes + CronJob** on the workloads EKS cluster so it egresses via a stable IP Mozilla + can allowlist (issue #27) — the k8s manifests/Pulumi/ArgoCD app live in the + separate `platform-infrastructure` repo; this repo only builds the image + (`Dockerfile`, `.github/workflows/aaq-scraper-image.yml` → shared ECR via + OIDC) and ships `deploy/entrypoint.sh` (clone → `run_refresh.py` → commit). + Two prerequisites already landed here: (1) `sumo.py` **dropped Playwright** + entirely — the Chromium challenge-bypass is itself now fingerprinted and + blocked (issue #28) — and drives the same `SumoBrowser`/`fetch_json` public + API over a plain **`httpx`** client instead; (2) **`.refresh-hwm` is now + tracked in git** (removed from `.gitignore`, no longer the Actions cache) + since the pod is stateless and the repo is the durable state. The pod pushes + its commits authenticated with a **fine-grained GitHub PAT** sourced from AWS + Secrets Manager (synced in by External Secrets Operator) via a **git + credential helper** — the token is read from the environment at call time and + never appears in argv or a URL. Because the API is still blocked pending the + egress-IP allowlist, `.github/workflows/scrape.yml` **stays live** as the + production refresh until cutover, and `schema-check.yml`'s daily schedule is + **parked** (`workflow_dispatch` only) to stop spurious `api-blocked` issues + against the same block. Full design: + `docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md` + and `docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md`. ## Notes diff --git a/README.md b/README.md index 3cfff40..7525392 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,19 @@ # aaq-scraper Until we get an official API that has a way to prevent DOS-ing, we scrape the SUMO -(support.mozilla.org) API by driving a real browser. Since ~June 2026 the API sits -behind a JavaScript challenge that blocks headless HTTP (see -[thunderbird/github-action-thunderbird-aaq#34](https://github.com/thunderbird/github-action-thunderbird-aaq/issues/34)); -a real browser passes the challenge, then we call the JSON API from inside the -browser's authenticated context. +(support.mozilla.org) API. Since ~June 2026 the API sits behind a JavaScript +challenge that blocks headless HTTP (see +[thunderbird/github-action-thunderbird-aaq#34](https://github.com/thunderbird/github-action-thunderbird-aaq/issues/34)). +A real-browser (Playwright/Chromium) workaround passed the challenge for a +while but is now itself fingerprinted and blocked (issue #28), so the scraper +is a plain **`httpx`** client instead, and is being redeployed as a +**Kubernetes CronJob** on a cluster with a stable egress IP that Mozilla can +allowlist (issue #27) — see the design and plan docs under +[`docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md`](docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md) +and +[`docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md`](docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md). +Until that IP is allowlisted, live API calls remain blocked and the hourly +GitHub Actions refresh (`scrape.yml`) stays the production path. ## Proof of concept (Bucket 0) diff --git a/pyproject.toml b/pyproject.toml index 14fab65..7ca893a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "aaq-scraper" version = "0.1.0" -description = "Scrape the SUMO (support.mozilla.org) AAQ API by driving a real browser past its JavaScript challenge." +description = "Scrape the SUMO (support.mozilla.org) AAQ API over HTTP, deployed as a Kubernetes CronJob." requires-python = ">=3.10" dependencies = [ "httpx==0.28.1", diff --git a/tests/test_entrypoint.py b/tests/test_entrypoint.py index 7505308..96eab1f 100644 --- a/tests/test_entrypoint.py +++ b/tests/test_entrypoint.py @@ -4,8 +4,9 @@ """Integration test for deploy/entrypoint.sh using a local bare repo. Exercises the clone -> (fake) refresh -> commit -> push loop without a network -or a real token. The token is stripped from a file:// URL by git, so a dummy -value is fine.""" +or a real token. The entrypoint authenticates via a git credential helper +rather than splicing the token into the URL, and a file:// origin needs no +auth at all, so the dummy token value is never actually used.""" import os import subprocess from pathlib import Path From 3569ce1fd82ccf1340b03f8562bc8ab29c926b36 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:28:22 -0700 Subject: [PATCH 15/17] fix: address final-review findings (QEMU for arm64 build, doc staleness, read-only-FS workdir) --- .github/workflows/aaq-scraper-image.yml | 4 ++++ CLAUDE.md | 17 +++++++++-------- README.md | 3 +++ deploy/entrypoint.sh | 5 ++++- poc.py | 6 ++++++ sumo.py | 12 ++++++------ 6 files changed, 32 insertions(+), 15 deletions(-) diff --git a/.github/workflows/aaq-scraper-image.yml b/.github/workflows/aaq-scraper-image.yml index 9331d81..ffc4c24 100644 --- a/.github/workflows/aaq-scraper-image.yml +++ b/.github/workflows/aaq-scraper-image.yml @@ -60,6 +60,10 @@ jobs: aws-region: us-east-1 - name: Login to ECR uses: aws-actions/amazon-ecr-login@v2 + - name: Set up QEMU + uses: docker/setup-qemu-action@v3 + with: + platforms: arm64 - name: Set up Buildx uses: docker/setup-buildx-action@v3 - name: Build and push diff --git a/CLAUDE.md b/CLAUDE.md index e4df6ff..313a966 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,17 +8,18 @@ Scrape the Mozilla SUMO (support.mozilla.org) "Ask a Question" API for Thunderbi Desktop and Thunderbird Android, producing CSVs compatible with the legacy Ruby reports in `thunderbird/github-action-thunderbird-aaq`. Since ~June 2026 the API sits behind a JavaScript challenge that blocks headless HTTP (issue -thunderbird/github-action-thunderbird-aaq#34), so we drive a **real browser** to -pass the challenge and call the JSON API from inside the browser's authenticated -context. +thunderbird/github-action-thunderbird-aaq#34); a real-browser (Playwright/ +Chromium) workaround passed the challenge for a while but is now itself +fingerprinted and blocked (issue #28), so the scraper instead calls the API +over a plain **`httpx`** HTTP client, and is moving to a Kubernetes CronJob +deployment with a stable, allowlistable egress IP (issue #27). ## Core architecture & crucial decisions -- **Browser-passes-challenge approach** (`sumo.py`): `SumoBrowser` launches - Chromium (Playwright), loads the site once to acquire challenge cookies, then - `fetch_json()` does an **in-page `fetch()`** (`page.evaluate`) so the request - reuses the page's cookies/origin. Headless works (locally **and in GitHub - Actions** — this is what resolves #34). +- **HTTP-client approach** (`sumo.py`): `SumoBrowser` (name kept for + compatibility) drives a plain `httpx` client; historically it launched + Chromium via Playwright to pass the JS challenge, but that is now + fingerprinted/blocked (#28). - **Stack: Python + Playwright, managed with `uv`** — use `uv sync` / `uv run`, never pip or raw venv. Deps in `pyproject.toml`. - **`fetch_json` retries** transient failures with exponential backoff: HTTP 429 diff --git a/README.md b/README.md index 7525392..d8be2f5 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,9 @@ GitHub Actions refresh (`scrape.yml`) stays the production path. ## Proof of concept (Bucket 0) +Historical Bucket-0 script; `playwright` is no longer a project dependency, +so install it separately (`uv run pip install playwright`) before running. + ```sh uv sync uv run playwright install chromium diff --git a/deploy/entrypoint.sh b/deploy/entrypoint.sh index d58831e..1f4415d 100755 --- a/deploy/entrypoint.sh +++ b/deploy/entrypoint.sh @@ -16,7 +16,10 @@ GIT_AUTHOR_NAME="${GIT_AUTHOR_NAME:-aaq-scraper-bot}" GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-aaq-scraper-bot@thunderbird.net}" REFRESH_ARGS="${REFRESH_ARGS:---soft-deadline 40 --max-429-wait 120}" -WORKDIR="$(mktemp -d)" +# $HOME is the writable emptyDir mounted in the pod (Phase B sets HOME=/work); +# /tmp is on the read-only root filesystem (readOnlyRootFilesystem: true), so +# mktemp must be pointed at $HOME rather than its default /tmp base. +WORKDIR="$(mktemp -d "${HOME:-/tmp}/aaq-scraper.XXXXXX")" trap 'rm -rf "$WORKDIR"' EXIT # Supply the PAT via a credential helper that reads $GITHUB_TOKEN from the diff --git a/poc.py b/poc.py index 2eb6597..55603ed 100644 --- a/poc.py +++ b/poc.py @@ -5,6 +5,12 @@ """ aaq-scraper — Bucket 0 proof of concept. +Historical: this was the original browser-based proof of concept; the +scraper has since moved to a plain `httpx` client (`sumo.py`), and +`playwright` is no longer a project dependency (`pyproject.toml`). To run +this script, install playwright separately (`uv run pip install playwright` +or `pip install playwright`, then `playwright install chromium`). + Goal: prove a real browser can pass the JavaScript challenge that SUMO (support.mozilla.org) now puts in front of its API, then read one page of /api/2/question/ as JSON from inside the browser's authenticated context. diff --git a/sumo.py b/sumo.py index 5621495..557e8bf 100644 --- a/sumo.py +++ b/sumo.py @@ -136,8 +136,8 @@ def __exit__(self, exc_type, exc, tb): def _raw_fetch(self, url): """Do one HTTP GET; return {status, json, snippet, retry_after}. - Parses the body as JSON the same way the old in-page fetch did (json is - None when the body is not valid JSON — e.g. an HTML challenge page).""" + Parses the body as JSON (json is None when the body is not valid + JSON — e.g. an HTML challenge page).""" res = self._client.get(url, headers={"Accept": "application/json"}) text = res.text try: @@ -152,9 +152,9 @@ def _raw_fetch(self, url): } def fetch_json(self, url): - """Fetch `url` via an in-page fetch(), retrying transient failures. + """Fetch `url` via the HTTP client, retrying transient failures. - Reuses the page's cookies/origin so the JS challenge stays satisfied. + Reuses the client's cookie jar so the JS challenge stays satisfied. Retries with exponential backoff on HTTP 429 and 5xx (honouring Retry-After for 429). A 200 that isn't JSON is treated as a challenge hiccup and retried too. Other 4xx fail immediately. Raises RuntimeError @@ -215,8 +215,8 @@ def fetch_json(self, url): raise exc( f"Gave up after {self.max_attempts} attempts; last status " f"{last['status']} for {url}\nFirst 300 chars: {last['snippet']!r}\n" - "If this is a 200 with HTML, the browser may not have passed the " - "challenge." + "If this is a 200 with HTML, the client may not have passed the " + "challenge / the edge served the challenge page." ) def paginate(self, first_url, on_page=None): From d200004ca5a59adab7a4a34e6ed6f3164716d80e Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:29:43 -0700 Subject: [PATCH 16/17] docs: finish CLAUDE.md httpx de-staling (Stack bullet + commands) Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 313a966..327fcb2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,8 +20,9 @@ deployment with a stable, allowlistable egress IP (issue #27). compatibility) drives a plain `httpx` client; historically it launched Chromium via Playwright to pass the JS challenge, but that is now fingerprinted/blocked (#28). -- **Stack: Python + Playwright, managed with `uv`** — use `uv sync` / `uv run`, - never pip or raw venv. Deps in `pyproject.toml`. +- **Stack: Python + `httpx`, managed with `uv`** — use `uv sync` / `uv run`, + never pip or raw venv. Deps in `pyproject.toml`. (`poc.py` still uses Playwright, + which is no longer a project dependency — install it separately to run the PoC.) - **`fetch_json` retries** transient failures with exponential backoff: HTTP 429 (honours `Retry-After`), 5xx, and a 200-but-non-JSON challenge hiccup. Non-retryable 4xx fail fast. @@ -112,8 +113,7 @@ deployment with a stable, allowlistable egress IP (issue #27). ## Commands ```sh -uv sync -uv run playwright install chromium # one-time +uv sync # httpx client; no browser install needed # Questions (single day = same date twice). Add --product thunderbird-android for Android. uv run python scrape_questions.py 2026 6 10 2026 6 10 --headless From 8f01f6d52d4c116c957ce88a04cc18785d326600 Mon Sep 17 00:00:00 2001 From: Arron Atchison Date: Mon, 13 Jul 2026 12:30:08 -0700 Subject: [PATCH 17/17] docs(plan): add Phase B pre-cutover checklist for read-only-FS entrypoint verification Co-Authored-By: Claude Opus 4.8 (1M context) --- .../plans/2026-07-13-k8s-argocd-scraper-deployment.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md b/docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md index 3b10c54..8a72195 100644 --- a/docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md +++ b/docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md @@ -1130,6 +1130,16 @@ gh pr create --repo thunderbird/platform-infrastructure --base main \ # Cutover (follow-up, after Mozilla allowlists — not in these PRs) +- **Pre-cutover: verify the entrypoint runs end-to-end in the container against a + read-only root FS** (final-review Issue 2). The pod uses + `readOnlyRootFilesystem: true` with a writable emptyDir at `$HOME=/work`. The + entrypoint now creates its workdir under `$HOME` (survives a read-only `/tmp`), + but two things still need real-image verification before the Job can succeed: + (1) set `TMPDIR=/work` (or mount an emptyDir at `/tmp`) if any tool writes to + `/tmp`; (2) `run_refresh.py` shells out to `uv run python scrape_*.py` inside + the *clone* dir — confirm `uv run` reuses the image's baked venv (or set + `UV_PROJECT_ENVIRONMENT`/run the baked interpreter directly) so it does **not** + try to build a fresh `.venv` needing PyPI network + a writable clone at runtime. - Bump the CronJob `image:` tag to the first built `git-` (platform-infra PR); confirm ArgoCD syncs and a manually-triggered Job commits real CSVs. - Arm the VMRule (remove `and vector(0)`). - Disable `scrape.yml` in the aaq-scraper repo.