Move scraper to a self-hosted runner with a static egress IP (drop Playwright)

[aatchison]

Summary

The browser-based scraper is now fully blocked on GitHub Actions. As of ~2026-07-09 the Fastly bot-challenge fingerprints the automated browser itself — it blocks headless and headed, Playwright's Chromium and real Chrome, with navigator.webdriver=false + stealth scripts, even a persistent real-Chrome profile. Only a genuine human-driven browser passes, so this is not solvable on our side within Actions (see #27, #26, and thunderbird/github-action-thunderbird-aaq#34).

We have no path forward on GitHub Actions. We need to move off it.

Proposed path forward

1. Stand up a self-hosted runner with a static public egress IP — a VPS or Thunderbird infra.
2. Give Mozilla/SUMO that egress IP to allowlist at the edge (bot-challenge exemption). Request is tracked in ACTION (Roland): post API-access request to mozilla/sumo#3124 + bitergia-deploy#50 #27 → API: /api/2/question/ intermittently returns HTTP 500 with ordering=updated + updated__gt mozilla/sumo#3124.
3. Switch the scraper to a plain HTTP client (drop Playwright) — once the IP is allowlisted, /api/2/ is reachable directly over HTTP; no browser needed.
4. Re-enable .github/workflows/scrape.yml to run on the self-hosted runner.

An API token that bypasses the challenge would be an acceptable alternative to the IP allowlist — either mechanism unblocks us.

Open questions / decisions

• VPS vs. Thunderbird infra for the runner (cost, ownership, maintenance).
• Whether Mozilla grants the IP allowlist or an API token.
• Scope of the Playwright → HTTP-client refactor in sumo.py (challenge-cookie logic can be removed once the IP is exempt).

Context

• rtanglao, ACTION (Roland): post API-access request to mozilla/sumo#3124 + bitergia-deploy#50 #27 (2026-07-10): "the scraper started failing again yesterday and there is no path forward on github actions. We have no choice now. Must move to a VPS or Thunderbird infra."
• Blocked upstream on the API-access request in ACTION (Roland): post API-access request to mozilla/sumo#3124 + bitergia-deploy#50 #27.

cc @rtanglao @mzeier