[DNM] Try two worker processes per host

[ntabris]

I'd like to see how this impacts perf of various benchmarks.

[fjetter]

is this on prod now?

[fjetter]

IIUC a fair assessment would require us to double the machine sizes and reduce cluster sizes by half. Otherwise the same-host workers have much less memory to work with and are much more likely to run into spilling and OOM

[fjetter]

I kicked off an A/B test with this https://github.com/coiled/benchmarks/actions/runs/5667683049

[fjetter]

Results of this A/B test are interesting but likely require a bit further analysis

We can see a couple of tests that regress significantly while others are getting much worse.

My best read without digging deeper is that most/almost all of our tests require a bit of spilling and that all test cases that are spilling behave really poorly, likely because disk is just busy?

[ntabris]

@fjetter you want to try run with 1/2 the number of m6i.xlarge machines? want me to push that change to this branch?

[fjetter]

@ntabris #917 is what I ran for the A/B tests. Clusters were using only half the number of VMs, VMs were twice as large, i.e. same total number of CPUs, same total RAM, half the number of VMs with two workers each