feat: add Cloudflare browser backend and bound Puppeteer resource usage

[nonumpa]

Summary

This PR makes Puppeteer execution configurable and resource-bounded:

• Adds local and cloudflare browser backends behind BROWSER_BACKEND.
• Limits concurrent Puppeteer scrapes across all gRPC requests.
• Blocks image, media, and font requests by default to reduce browser memory and network usage.
• Renames the internal scrap terminology to scrape.

The URL resolver gRPC response shape remains unchanged.

Browser backends

Local

BROWSER_BACKEND=local remains the default.

The resolver launches one local Chromium browser per process and opens one page per concurrent scrape() call. Local Chromium is started eagerly and relaunched immediately after an unexpected disconnect.

Cloudflare Browser Rendering

BROWSER_BACKEND=cloudflare connects to Cloudflare Browser Rendering over CDP using puppeteer-core.

BROWSER_BACKEND=cloudflare
CLOUDFLARE_ACCOUNT_ID=...
CLOUDFLARE_API_TOKEN=...
CLOUDFLARE_KEEP_ALIVE_MS=60000

Backend selection is explicit. A Cloudflare connection failure does not automatically fall back to local Chromium; operators can switch back by setting BROWSER_BACKEND=local.

Cloudflare sessions are acquired lazily:

• No session is created when the resolver process starts.
• The first scrape() call opens the session.
• When Cloudflare closes an idle session, the browser handle is cleared.
• The next scrape() reconnects on demand.
• A failed connection does not leave the resolver pinned to a rejected promise.

This avoids repeatedly acquiring billed browser sessions while the resolver is idle.

Resource bounds

Global scrape concurrency

SCRAPE_MAX_CONCURRENCY=3

A module-level p-limit semaphore limits concurrent Puppeteer scrape() calls across all in-flight gRPC requests.

Only the Puppeteer fallback is limited. normalize, unshorten, and parseMeta retain their existing concurrency.

Lower values reduce Chromium memory pressure; higher values improve tail latency when sufficient memory is available.

Resource interception

SCRAPE_BLOCK_RESOURCES=image,media,font

Puppeteer aborts these resource types before they are downloaded. Documents, stylesheets, scripts, XHR, and fetch requests remain enabled so client-rendered pages can still build their DOM.

Set the variable to an empty string to disable interception:

SCRAPE_BLOCK_RESOURCES=

request.abort() and request.continue() rejections caused by a page closing mid-request are handled to avoid unhandled rejections on Node.js 24.

Image-selection trade-off

Open Graph image URLs are read from HTML metadata and do not require the image itself to be downloaded.

When no Open Graph image is available, scrape() falls back to selecting the largest rendered <img>. With image requests blocked, image dimensions may be 0x0, so this fallback may select the first image rather than the largest one.

Operators that require the previous rendered-image behavior can set:

SCRAPE_BLOCK_RESOURCES=

Compatibility

• BROWSER_BACKEND=local remains the default.
• The Docker image still includes Chromium, allowing operators to switch from Cloudflare back to local without rebuilding the image.
• The gRPC wire format and response fields are unchanged.
• UNKNOWN_SCRAP_ERROR was renamed to UNKNOWN_SCRAPE_ERROR, retaining protobuf numeric value 6.
• Default runtime behavior changes by limiting concurrent scrapes to 3 and blocking image, media, and font resources.

Verification

• npm run compile
• npm test: 7 suites, 35 tests, 83 snapshots
• npm run lint
• Docker build
• Local browser smoke test
• Cloudflare connection and lifecycle unit tests
• Concurrency-limit unit test
• Resource-interception unit tests

[gemini-code-assist]

Code Review

This pull request upgrades the project to Node 24, migrates from the deprecated grpc package to @grpc/grpc-js, and introduces support for Cloudflare Browser Rendering as an alternative browser backend. It also adds resource limits for scraping, such as blocking specific resource types and capping concurrency using p-limit. However, several critical issues must be addressed: the gRPC server will not accept requests because server.start() was omitted from the new asynchronous binding callback; unhandled promise rejections from browser launching and request interception could crash the Node 24 process; using new Function for evaluating Readability.js will fail on sites with strict Content Security Policies; and wrapping the entire resolution pipeline in limit unnecessarily throttles fast, non-Puppeteer operations.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[MrOrz]

LGTM!