Define some statistical method for the comparison of two experimental datasets

[arm61]

We need a way to compare "identical" datasets.

[arm61]

WIP here https://github.com/reflectivity/reproducibility/blob/master/comparing_datasets.md

[jfkcooper]

As detailed in https://arxiv.org/pdf/2103.08973.pdf Section 3.1 if the points are indeed at the same place (a rebin or similar required here though). Then the Hotelling t^2 test can give the probability that the datasets are different (and the significance of that difference). The inverse way of thinking about the problem, but having looked into this quite alot, I couldn't find anything better.

[arm61]

What about if the q-ranges are different? I guess we only use overlapping regions then.

[jfkcooper]

Pretty much all you can do I think.

[jfkcooper]

Have made a thing (mostly stolen from James Durant's code), it will do this and is in the main repo as am not sure where sripts should live

[jfkcooper]

I think I need to do some more work on this still as am not sure it is correct yet...

[arm61]

I quite liked the use of a Hotelling r² test, what is wrong with it?

[jfkcooper]

I think the principle might be okay, but my implementation isn't quite right. The current "solution" works in count space (because we need to somehow include the error bars of the points), however this makes something with a scalefactor of a half which has been counted twice as long equal to a dataset of unity scalefactor counted half as long, i.e. not what we want really.
I had thought it was working but recently revisited it and found it wasn't. The Hotelling test might be helpful still, but I think it needs a different way to include the error bars of the datapoints

[wpotrzebowski]

Not sure if it is useful for your case but here is a method applied to Small Angle Scattering data "for assessing differences between one-dimensional spectra independently of explicit error estimates" https://pubmed.ncbi.nlm.nih.gov/25849637/. There is an open-source implementation of the method in freesas (https://github.com/kif/freesas/blob/master/freesas/app/cormap.py)