feat: Add opt-in per-domain request throttling for HTTP 429 backoff - #1762

Merged

vdusek merged 29 commits into

apify:masterfrom

MrAliHasan:fix/request-throttler-429-backoff

May 6, 2026

Commits on Feb 20, 2026

fix: add per-domain RequestThrottler for 429 backoff (#1437)
MrAliHasan
committed
64f7247

Commits on Feb 23, 2026

refactor: replace RequestThrottler with ThrottlingRequestManager
MrAliHasan
committed
62ab3b8

Commits on Feb 25, 2026

refactor: reimplement `ThrottlingRequestManager` with per-domain sub-queues and update its integration across crawlers.
MrAliHasan
committed
1065e9b

Commits on Feb 26, 2026

test: fix typing and linting checks in ThrottlingRequestManager tests
MrAliHasan
committed
138fd67

Commits on Mar 2, 2026

feat: Add explicit domain routing and management to the request throttling manager.
MrAliHasan
committed
abdf51c

Commits on Mar 4, 2026

feat: Implement recreate_purged for ThrottlingRequestManager and refactor domain state management and sub-queue handling.
MrAliHasan
committed
dd99d9d

Commits on Mar 5, 2026

deps: Pin ty to version 0.0.18 and update uv.lock to include Python 3.14 wheels for brotlicffi.
MrAliHasan
committed
497b782

Commits on Mar 6, 2026

fix: Ensure ThrottlingRequestManager.add_request explicitly handles None returns and updates the ty dependency constraint.
MrAliHasan
committed
902f885

Commits on Mar 11, 2026

refactor: Address reviewer feedback on ThrottlingRequestManager and revert uv.lock
MrAliHasan
committed
e02dd68

Commits on Mar 13, 2026

Commits on Mar 26, 2026

refactor: Address review feedback — proper typing, recreate_purged guard, crawl-delay caching, revert uv.lock
MrAliHasan
committed
412df15

Commits on Apr 3, 2026

refactor: Add request_manager_opener callback, move record_success into mark_request_as_handled, fix docs
MrAliHasan
committed
a249a23

Commits on May 5, 2026

Merge branch 'master' into fix/request-throttler-429-backoff
vdusek
committed
4be7b2d

Commits on May 6, 2026

Make `ThrottlingRequestManager` generic over inner manager type
vdusek
committed
a3f5c7c
Cache crawl-delay configuration per origin in `BasicCrawler`
vdusek
committed
bebf2db
Reflow docstrings to full 120-char width and fix backticks
vdusek
committed
e5fe554
Simplify
vdusek
committed
6dbe696
Fix flaky test
vdusek
committed
8f10c10
warn when capping Retry-After or backoff at max_delay
vdusek
committed
ed0dcc0
make ThrottlingRequestManager sub-managers honor the generic inner type
vdusek
committed
f2f47a5
wake ThrottlingRequestManager fetch loop when new work arrives during throttle wait
vdusek
committed
546d7ac
Match throttled domains case-insensitively
vdusek
committed
10481c4
address review nits on ThrottlingRequestManager
vdusek
committed
fac6367
Polishment & ordering
vdusek
committed
8acbb43
Final
vdusek
committed
0f1574a
Address review comments
vdusek
committed
b60c720
Add warnings
vdusek
committed
851d044