Feature request: parallel iteration

[noamross]

I'd like to be able to perform multiple iterative request chains in parallel.

My current use-case is having a large set of multi-step requests (POST a query, receive a query ID URL, poll the query ID URL for query status, on completion, download file from the provided URL). I'd like to be able to run this list using a parallel queue that handles throttling and retrying well.

I'm not sure if req_perform_spider() (#584) will ultimately be flexible enough for this.

[hadley]

Yeah, I think req_perform_spider() would be able to handle that (or is at least a prerequisite).