sameDomainDelaySecs and the ThrottlingRequestManager added in #3741 are now two per-domain delay mechanisms that don't know about each other — delayRequest keys on eTLD+1 via tldts.getDomain, the manager on exact hostname, and nothing warns if both are set. Reimplementing the former on top of the latter is the obvious fix and would close out #3148, but it isn't cheap:
- The manager opens one
RequestQueue per configured domain, eagerly. Fine for a hand-written domains list, hopeless for something that applies to every domain a crawl encounters — and the sub-queue-per-domain design is exactly what lets fetchNextRequest skip a throttled domain without rescanning, so it can't simply be dropped.
- Sub-queues are alias-keyed and outlive the process, so the manager reopens every configured domain at startup, or a restart strands their requests. A domain set discovered at runtime has nothing to reopen unless the list is persisted as well.
- Subdomains would stop sharing a clock, so existing
sameDomainDelaySecs crawls end up more parallel rather than less.
Rough shape: make domains optional, create sub-queues lazily on first sighting, cap them with an LRU and fall back to unpaced inner beyond the cap, and persist the live domain list so startup can reopen it.
sameDomainDelaySecsand theThrottlingRequestManageradded in #3741 are now two per-domain delay mechanisms that don't know about each other —delayRequestkeys on eTLD+1 viatldts.getDomain, the manager on exact hostname, and nothing warns if both are set. Reimplementing the former on top of the latter is the obvious fix and would close out #3148, but it isn't cheap:RequestQueueper configured domain, eagerly. Fine for a hand-writtendomainslist, hopeless for something that applies to every domain a crawl encounters — and the sub-queue-per-domain design is exactly what letsfetchNextRequestskip a throttled domain without rescanning, so it can't simply be dropped.sameDomainDelaySecscrawls end up more parallel rather than less.Rough shape: make
domainsoptional, create sub-queues lazily on first sighting, cap them with an LRU and fall back to unpacedinnerbeyond the cap, and persist the live domain list so startup can reopen it.