feat: implement per-domain request throttling (ThrottlingRequestManager) - #3741

Merged

janbuchar merged 33 commits into

harryautomazione:v4

Aug 10, 2026

Commits on Aug 5, 2026

feat: implement per-domain request throttling (ThrottlingRequestManager)

harryautomazione
authored and
janbuchar
committed
0a1035c
test(basic-crawler): cover the `requestManager` constructor option
janbuchar
committed
f88413a
fix(core): reopen per-domain sub-queues instead of discovering them lazily
janbuchar
committed
7c84d14
fix(core): don't park a concurrency slot waiting out a throttled domain
janbuchar
committed
acf8aab
fix(core): delegate batched adds instead of reimplementing them
janbuchar
committed
c176431
fix(core): warn that `requestsFromUrl` sources are not domain-routed
janbuchar
committed
cad6eaa
fix(core): advance the 429 backoff per rate-limit event, not per response
janbuchar
committed
d7bbd41
fix(crawlers): don't spend a retry or the session on a throttled 429
janbuchar
committed
0785300
fix(crawlers): warn only when a robots.txt crawl-delay is actually dropped
janbuchar
committed
a1269e5
refactor(crawlers): fold the 429 interception into one typed helper
janbuchar
committed
61e8135
fix(core): tighten `Retry-After` parsing
janbuchar
committed
260b907
docs(core): document ThrottlingRequestManager and drop leftovers
janbuchar
committed
6388def
docs: cover per-domain throttling in the guides and upgrading notes
janbuchar
committed
b3d26b4
chore: regenerate public API snapshots
janbuchar
committed
157682f

Commits on Aug 6, 2026

Commits on Aug 8, 2026

Commits on Aug 9, 2026

refactor(core): tidy up the throttling manager's public surface
janbuchar
committed
430cc3b
refactor(browser-crawler): declare `headers()` on `BaseResponse`
janbuchar
committed
ffe2ef1
docs(core): note that `isEmpty` may disagree with `getPendingCount`
janbuchar
committed
3255a77
fix: Ensure that scraping is paced on a per-domain basis, not per-task
janbuchar
committed
337baf8
fix: Normalize hostnames
janbuchar
committed
5159245
fix: Do not give up on stalled domains when keepAlive is enabled
janbuchar
committed
0002e6d
fix: Fix validation rules
janbuchar
committed
acd50ac
fix: Improve rate limit tracking
janbuchar
committed
2abc412
chore: Random cleanup
janbuchar
committed
9d1cf48
fix: Improve domain stall detection
janbuchar
committed
d80e4a9
fix: Prevent race conditions in batched request addition
janbuchar
committed
a2b408c
fix: Consume http response body streams of throttled requests
janbuchar
committed
d18d72c