You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
consecutive_429_count advances once per response, not per rate-limit event. A burst of N in-flight requests all come back 429 and jump the exponent N steps at once — 8 concurrent requests take the delay from 2s to past max_delay in one go. The TS port added a guard for this in feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741, though it broke crawl-delay doing it, so that one isn't worth copying verbatim.
Retry-After: 0 parses to timedelta(0) and takes priority over the exponential backoff, so the domain ends up with no delay at all and the backoff never engages. test_parse_retry_after_zero_seconds currently asserts the broken behaviour. Treating a zero delay as "no usable header" and falling through to the backoff seems right.
Two bugs in
record_domain_delay, both from #1762:consecutive_429_countadvances once per response, not per rate-limit event. A burst of N in-flight requests all come back 429 and jump the exponent N steps at once — 8 concurrent requests take the delay from 2s to pastmax_delayin one go. The TS port added a guard for this in feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741, though it broke crawl-delay doing it, so that one isn't worth copying verbatim.Retry-After: 0parses totimedelta(0)and takes priority over the exponential backoff, so the domain ends up with no delay at all and the backoff never engages.test_parse_retry_after_zero_secondscurrently asserts the broken behaviour. Treating a zero delay as "no usable header" and falling through to the backoff seems right.