Skip to content

ThrottlingRequestManager computes the wrong backoff for concurrent 429s and for Retry-After: 0 #2125

Description

@janbuchar

Two bugs in record_domain_delay, both from #1762:

  • consecutive_429_count advances once per response, not per rate-limit event. A burst of N in-flight requests all come back 429 and jump the exponent N steps at once — 8 concurrent requests take the delay from 2s to past max_delay in one go. The TS port added a guard for this in feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741, though it broke crawl-delay doing it, so that one isn't worth copying verbatim.
  • Retry-After: 0 parses to timedelta(0) and takes priority over the exponential backoff, so the domain ends up with no delay at all and the backoff never engages. test_parse_retry_after_zero_seconds currently asserts the broken behaviour. Treating a zero delay as "no usable header" and falling through to the backoff seems right.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions