Currently we scrape the target URL directly, but some hosts might detect the incoming IP in the access log, and block it after a while.
To prevent this, we want to allow the host to specify a list of proxies to use.
We can implement TinyProxy to cycle through the proxies when scraping a site.
Currently we scrape the target URL directly, but some hosts might detect the incoming IP in the access log, and block it after a while.
To prevent this, we want to allow the host to specify a list of proxies to use.
We can implement TinyProxy to cycle through the proxies when scraping a site.