When running evals with Docker, the only rate limit is when the containers are started. For the matrix I'm running, this means that around 400 containers are running in parallel, even if their starts are slightly staggered.
This should be capped with a configurable number, probably defaulting to os.cpus().length, so that machines running evals via Docker don't become overloaded from running so many experiments at the same time.
When running evals with Docker, the only rate limit is when the containers are started. For the matrix I'm running, this means that around 400 containers are running in parallel, even if their starts are slightly staggered.
This should be capped with a configurable number, probably defaulting to
os.cpus().length, so that machines running evals via Docker don't become overloaded from running so many experiments at the same time.