[DNM] Try using placement group (faster worker <-> worker network)

[ntabris]

I'm especially curious if this makes an impact on something like the vorticity test now that (in upstream dask) comms aren't slowed down by compression.

[ntabris]

No improvement in transfer time on test_double_diff, but test_vorticity showed transfer time of 276.0s compared to ~285–310s on previous runs (on compressible data but with compression disabled).

So... not clear that there's improvement, but seems like there might be.

@crusaderky what do you think about doing A/B test with placement groups?

For context, placement groups let you tell Amazon you'd like your instances physically close to each other, which can improve network perf between those instances. There's a not-yet-advertised Coiled option to use a placement group for your cluster. (We don't use it by default because we've yet to see definite improvement, and it does have some impact on availability and/or time to provision cluster.)

[crusaderky]

@crusaderky what do you think about doing A/B test with placement groups?

Sure, go on :)