Reduce Puppeteer usage with static article extraction

[nonumpa]

Base: feat/cloudflare-browser-rendering — merge that PR first.

Why

Launching Chromium for every URL adds roughly 500 MB of browser overhead and is much slower than a plain HTTP fetch. For SSR pages, the article body is already present in the HTML even when parseMeta cannot produce a complete result.

This PR adds a lightweight static extraction step before the Puppeteer fallback, and avoids rendering responses that a GET request has confirmed are terminal HTTP errors.

Pipeline

1. unshorten — follows redirect chains (up to 5 hops)
2. parseMeta — unchanged
3. extractStatic — node-fetch + linkedom + @mozilla/readability
4. scrape — Puppeteer fallback

Each layer fills the result or falls through. Static extraction errors, non-HTML responses, and DOM parse failures always fall through to Puppeteer.

extractStatic

• Fetches at most 5 MiB of HTML with a 5-second timeout.
• Parses the document with linkedom and extracts article text with Readability.
• Preserves canonical URL, Open Graph image, status, and raw HTML.
• Returns null for unsupported responses or parse failures.
• Maps known network failures to the existing ResolveError surface.
• Drains response bodies on early returns so sockets can be reused.

Terminal HTTP errors

Puppeteer is skipped for 5xx, 401, 403, 410, and 451 only when extractStatic's GET request observed that status.

The status returned by unshorten is based primarily on HEAD and is not used for this decision: many CDNs and anti-bot systems reject HEAD while serving GET normally. If static extraction throws or returns null, the pipeline still tries Puppeteer.

Tests cover both directions:

• HEAD 403, static GET 200 → use the static result.
• HEAD 200, static GET 500 → skip Puppeteer.
• HEAD 403, static extraction throws or returns null → fall through to Puppeteer.

Why linkedom instead of jsdom

Readability needs DOM tree operations but not script execution, CSS layout, subresource fetching, or a cookie jar. linkedom provides the required DOM surface with substantially lower startup and per-document cost.

The comparison harness lives on experiment/linkedom-vs-jsdom, directly on top of this PR. It uses committed HTML fixtures so both backends parse identical input.

Results on the 11-URL fixture set

Metric	linkedom vs jsdom
Output parity	41/44 fields exact; 3 whitespace-only; 0 content mismatches
Parse time, median of per-URL medians	114.5 ms vs 300.3 ms (2.6×)
Typical per-URL speedup, median ratio	3.1×
Heap delta, median across URLs	18 MB vs 54 MB (3×)
Cold-start RSS above baseline	14.4 MB vs 103.5 MB (7.2×)
Require time	79 ms vs 533 ms (6.7×)

Measured on Node 24 / Ubuntu; absolute timing and RSS vary by machine, while the same-input comparison is the relevant signal.

Both backends produced a structurally complete ScrapeResult for 8/11 fixtures. This measures the existing isIncomplete heuristic, not content quality—for example, the Threads login wall satisfies the heuristic. Production impact should be measured again on a representative Cofacts URL sample.

Detailed methodology, per-URL timing, parity classification, and memory caveats are documented in bench/README.md on the experiment branch.

Reproduce

git checkout experiment/linkedom-vs-jsdom
npm install

node bench/fetch.js
node --expose-gc bench/parse.js --runs 10
node bench/parity.js bench/results/parse-<timestamp>
node bench/cold-start.js --runs 10

bench/e2e.js --runs 3 is also available for the noisier live-network comparison.

Other changes

Multi-hop unshorten

• Follows up to 5 redirects with cycle detection.
• Resolves relative Location headers.
• Retries HEAD with GET on 405/501.
• Sends the Cofacts bot user agent.
• Drains response bodies for socket reuse.
• Returns { url, status }.

Dependencies

"@mozilla/readability": "^0.6.0",
"linkedom": "^0.18.12"

package-lock.json was regenerated with npm install --before=2026-05-09, following the repository's 30-day supply-chain convention.

[gemini-code-assist]

Code Review

This pull request introduces a fast-path static extraction mechanism (extractStatic) using jsdom and @mozilla/readability to avoid booting Puppeteer for SSR-rendered pages, alongside a benchmarking suite to compare performance. It also refactors the unshorten utility to follow redirect chains up to a maximum number of hops with HEAD/GET fallback. The code review feedback highlights critical opportunities to prevent memory and socket leaks, specifically by explicitly closing the JSDOM instance, consuming unread response bodies in node-fetch early-exit paths, and wrapping the static extraction call in a try-catch block to ensure robust fallback to Puppeteer if a network error occurs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To prevent memory leaks in a long-running server process, the JSDOM instance should be explicitly closed using dom.window.close(). Additionally, if og:url is relative, resolving canonical against finalUrl ensures it is always an absolute URL, preventing errors when resolving topImageUrl.

  let dom;
  try {
    dom = new JSDOM(html, { url: finalUrl });
  } catch (e) {
    return null;
  }

  try {
    const document = dom.window.document;

    const canonicalEl = document.querySelector('link[rel=canonical]');
    let canonical =
      (canonicalEl && canonicalEl.href) ||
      pickMeta(document, 'meta[property="og:url"]') ||
      finalUrl;

    try {
      canonical = new URL(canonical, finalUrl).href;
    } catch (e) {
      canonical = finalUrl;
    }

    let topImageUrl = '';
    const ogImage = pickMeta(
      document,
      'meta[property="og:image"]',
      'meta[property="og:image:url"]',
      'meta[name="twitter:image"]'
    );
    if (ogImage) {
      try {
        topImageUrl = new URL(ogImage, canonical).href;
      } catch (e) {
        topImageUrl = '';
      }
    }

    let article = null;
    try {
      article = new Readability(document).parse();
    } catch (e) {
      article = null;
    }

    const title =
      (article && article.title && article.title.trim()) ||
      pickMeta(document, 'meta[property="og:title"]') ||
      (document.title || '').trim();

    const summary =
      (article && article.textContent && article.textContent.trim()) ||
      pickMeta(
        document,
        'meta[property="og:description"]',
        'meta[name=description]'
      );

    return new ScrapResult({
      canonical,
      title: title || undefined,
      summary: summary || undefined,
      topImageUrl: topImageUrl || undefined,
      html,
      status,
    });
  } finally {
    dom.window.close();
  }

[gemini-code-assist]

When returning early from extractStatic (either due to a non-OK status or a non-HTML content type), the response body stream must be resumed/consumed to release the underlying socket and prevent connection/memory leaks in node-fetch.

Suggested change

	if (!res.ok) {
	return new ScrapResult({ canonical: finalUrl, status });
	}
	const ct = (res.headers.get('content-type') || '').toLowerCase();
	if (!ct.startsWith('text/html') && !ct.startsWith('application/xhtml')) {
	return null;
	}
	if (!res.ok) {
	if (res.body) res.body.resume();
	return new ScrapResult({ canonical: finalUrl, status });
	}
	const ct = (res.headers.get('content-type') || '').toLowerCase();
	if (!ct.startsWith('text/html') && !ct.startsWith('application/xhtml')) {
	if (res.body) res.body.resume();
	return null;
	}

[gemini-code-assist]

When falling back to a GET request in unshorten, the response body is never consumed. In node-fetch, leaving the response body unconsumed can leak sockets and memory. Resuming the body stream ensures resources are freed.

Suggested change

	let res;
	try {
	res = await fetch(current, { method: 'HEAD', ...FETCH_OPTS });
	if (res.status === 405 || res.status === 501) {
	res = await fetch(current, { method: 'GET', ...FETCH_OPTS });
	}
	} catch (e) {
	let res;
	try {
	res = await fetch(current, { method: 'HEAD', ...FETCH_OPTS });
	if (res.status === 405 || res.status === 501) {
	res = await fetch(current, { method: 'GET', ...FETCH_OPTS });
	}
	if (res.body) res.body.resume();
	} catch (e) {

[gemini-code-assist]

If extractStatic throws a ResolveError (e.g., due to a timeout or temporary network issue), the entire URL resolution will fail immediately. Wrapping it in a try-catch block allows the system to gracefully fall back to the more robust scrap (puppeteer) layer.

        if (fetchResult.isIncomplete) {
          if (isTerminalHttpError(finalStatus)) {
            fetchResult.status = finalStatus;
          } else {
            let staticResult = null;
            try {
              staticResult = await extractStatic(unshortened);
            } catch (e) {
              // eslint-disable-next-line no-console
              console.error('[extractStatic]', unshortened, e);
            }
            if (staticResult) fetchResult.merge(staticResult);

            if (fetchResult.isIncomplete) {
              fetchResult.merge(await limit(() => scrap(unshortened)));
            }
          }
        }