Weighting schemes

[cdowd]

Should be able to incorporate observation weights with ease. This has been requested by several users.

[cdowd]

Okay, so weights on observations are a bit funky actually. For the ECDF the implication is easy -- it merely adjusts heights. I actually build that weight vector anyhow, and it would be easy to adjust.

But the weights may also affect the resampling probabilities (i.e. chances I observed this individual affect the sampling variation associated with drawing them, while the portion of the larger population they represent affects the ecdf height -- survey designs are capable of tying or splitting those values). Typically these weights are related (though they don't need to be), so you need to build in two different weight options, set a reasonable default, and explain which is which in a clear and consistent (and correct) manner.

The good news (I think) is that only the ECDF height weights need to be passed down to the C++, the other weights only affect the sampling level.

[cdowd]

Worse. Seems likely that sampling weights break exchangeability assumption.

Easy enough to build each ECDF. Much harder to figure out what the null about the ECDFs are.

E.g. suppose samples from two different surveys... what then?

[luisvalenzuelar]

But if we are talking about comparing within a sample, for example, male vs female for employment data, then exchangeability assumption ok?

Maybe can implement weights and let individuals know that it only works when weights come from same sampling framework?

[cdowd]

But if we are talking about comparing within a sample, for example, male vs female for employment data, then exchangeability assumption ok?

Seems to have taken me a while, but I think this is right. E.g. imagine a stratified random sample from a population. Some people are chosen 10% of the time (weight = 10), some are chosen 1/100th of the time (weight=100).
As a starting point, assume the strata and sampling probs are the same in both samples. Then we shouldn't have any real issues? if the samples are the same, we could draw from both.

If the strata or sampling probabilities are different, maybe you wind up with issues caused by having different amounts of high weight observations in each sample -- and thus different 'discreteness', which is itself fine, until you start doing permutations. This feels closer to a power problem than a validity one, but I'd want to do some experimentation to be sure.