fix(crawler): treat HTTP 429 as rate-limit condition with cooldown backoff

[SoloDevAbu]

Fix: HTTP 429 treated as rate-limit condition with cooldown backoff

Closes #3623

Changes

• Introduced RateLimitError to represent retriable rate-limit failures with optional delay metadata
• Added rateLimitCooldownSecs (BasicCrawler option, default 60) as fallback wait time for 429 without Retry-After
• Updated retry handling in BasicCrawler to:
  • sleep for RateLimitError.delayMillis (or fallback cooldown)
  • reclaim the request after the delay
  • avoid session.markBad() for rate-limit retries
• Updated HTTP and Browser crawler response handling to throw RateLimitError on 429 and parse Retry-After
• Removed 429 from default blocked status codes so session blocking remains focused on true block/auth statuses (401, 403)
• Updated related docs/default annotations accordingly

Test Coverage

Test updates/additions validate:

• 429 with Retry-After delays retry appropriately
• 429 without Retry-After uses fallback cooldown
• blocked-status session retirement expectations exclude 429
• browser crawler coverage includes 429 delayed-retry behavior

[barjin]

Thank you for your contribution @SoloDevAbu .

I believe we should match the implementation of this feature with the similar feature in crawlee-python. Please see the PR here.

Below, I added a few comments (those might be irrelevant to the Python-inspired implementation).

[barjin]

Note that for backward compatibility, we should retry immediately. This change in default behaviour might make some users' workflows run much longer.

[barjin]

What happens if delayMillis is longer than internalTimeoutMillis? Won't the request fail anyway with a timeout?

[barjin]

As described in the previous review, we want to implement something like the new Python ThrottlingRequestManager to keep both versions aligned.

Also, the implementation in this PR would likely not work well with Crawlee's autoscaling. We do not want the waiting requests to block the execution of other requests - these should preferably "skip the line". With the current impl, we might just get stuck waiting for the delay to elapse, with other (valid) requests just waiting idle in the RequestQueue.

For these reasons, I'll be closing this PR now. Feel free to open a new one following apify/crawlee-python#1762 if you feel like it.

Thank you!