Develop a benchmark set (or sets) for realistic comparison of performance?

[mrshirts]

There's been some discussion, if we are going to compare performance, that we have a community organized set of simulations for benchmarks. This would probably actually be a paper developed by authors, but the journal editorial board might need to be involved to help it along.

[davidlmobley]

We should start with the density benchmark set from John Chodera and Kyle Beauchamp. I believe Kyle has also done some work on modernizing it/making it easy to repeat with protocol modifications, etc. As it is, they ran it with GAFF and then I ran it with SMIRNOFF, so there will be at least multiple force fields on the same set. Would be good to extend out for other codes and force fields. I can get Kyle to update us on where things stand if it's of interest.

This is on roughly 42 different pure solvents at a range of temperatures and pressures, IIRC. The data is from ThermoML. I can give a reference if needed, but not at the moment as I'm headed to a meeting.

[mrshirts]

I think there are actually multiple benchmarks. At least two: one for correctness, one for speed.