Skip to content

Reimplement sameDomainDelaySecs on top of ThrottlingRequestManager #3997

Description

@janbuchar

sameDomainDelaySecs and the ThrottlingRequestManager added in #3741 are now two per-domain delay mechanisms that don't know about each other — delayRequest keys on eTLD+1 via tldts.getDomain, the manager on exact hostname, and nothing warns if both are set. Reimplementing the former on top of the latter is the obvious fix and would close out #3148, but it isn't cheap:

  • The manager opens one RequestQueue per configured domain, eagerly. Fine for a hand-written domains list, hopeless for something that applies to every domain a crawl encounters — and the sub-queue-per-domain design is exactly what lets fetchNextRequest skip a throttled domain without rescanning, so it can't simply be dropped.
  • Sub-queues are alias-keyed and outlive the process, so the manager reopens every configured domain at startup, or a restart strands their requests. A domain set discovered at runtime has nothing to reopen unless the list is persisted as well.
  • Subdomains would stop sharing a clock, so existing sameDomainDelaySecs crawls end up more parallel rather than less.

Rough shape: make domains optional, create sub-queues lazily on first sighting, cap them with an LRU and fall back to unpaced inner beyond the cap, and persist the live domain list so startup can reopen it.

Metadata

Metadata

Assignees

Labels

debtCode quality improvement or decrease of technical debt.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions