mask in mapsets

[zatkins2]

We've discussed this informally before, but just wanted to formalize it here. The question is whether this is an issue -- I am raising it so I think so :p, but basically while masks look like maps and behave like them in code, they are abstractly different things. In practice, people make lots of mask versions -- often more than one mask for a given map product, and often the same mask applied to many different map-like products including derived maps. So unless it's difficult to implement, I'd push for masks being their own product (and they can inherit whatever low-level code from some base maps thing if needed for functionality).

Said another way: only from the point of view of client code are masks like maps. From a data management perspective, they are not maps.

[JBorrow]

That's interesting as I have not seen any instances on LAMBDA where the number of masks outnumbers the number of maps. But I could see how this would be of interest.

What I will say is that what we would probably want to do in this case is create a collection, with one MapSet containing e.g. the map and its inverse variance, and a number of other MapSet items that only contain masks. Would that work? This has the benefit of allowing people to bundle masks when appropriate (e.g. with a compton-y map or something) without the need to create a redundant collection.

[zatkins2]

Your slack message reminded me that this exists :p

That would work. Would it be clunky to have a mapset with only a mask in it? I’ll leave up to you, but there could also be a Mask class. I think that would mirror closest how users would expect to interact with the data, i.e. “I want to load my mask! Let me call up a mask-specific mapset” might feel unnatural vs. a Mask class.

[zatkins2]

(As to the philosophical question, I feel pretty strongly that we should lean on the flexibility of collections to associate data products. The one exception being unless two data products are unambiguously associated in all cases, as in the case of an inverse variance map with the actual map. Masks don’t rise to that level in my opinion. I am wary of “hard-coded-association” creep.)

[JBorrow]

In the case of the recent BOSSN maps there was a mask-per-map approach taken. See https://hippo.simonsobservatory.org/web/collections/686bfcda5897eba23c7e73e2

[zatkins2]

Yes, but that’s because it was one person doing one analysis (a single-shot analysis). In that case, you don’t need to worry about associations, there is just one analysis.

In the future, someone will want to do something similar on the same maps but with many different masks. I don’t think it’s the right thing to have to update the map version just to accommodate a new mask. Likewise, someone will want to use the same mask on a different map. I don’t think it’s the right thing that they will have to retrieve that mask from an unrelated map, and then hard-code the new association.

This discussion isn’t limited to masks. Beams, beam leakages, passbands, and calibration factors are also associated with specific maps. In fact, I’d argue they are more strongly associated with specific maps than a mask, since you really can't use a beam from one map on another. Wouldn’t you agree a collection is the way to go for these product associations?

[JBorrow]

The current system allows for both of these setups. You can create a mask in the top-level or on a per-product level depending on the type of analysis. You could add another MapSet only containing a mask in that collection (e.g. the i6 products that only have a 'map' associated with them).

[zatkins2]

Right, I guess I don't see the logic of why masks get special treatment in that sense (compared to beams, beam leakages, passbands, and calibration factors, etc). I think it will be confusing as a user to remember that (sometimes) masks live in the MapSet, sometimes not, while other, more map-specific products do not. If you feel strongly that masks should be an attribute of a MapSet, I'd say that beams, beam leakages, passbands, and calibration factors should be too, and that this is the default way to associate these "low-level products," while collections are reserved for more "abstract"/"higher-level" associations if that makes sense?

[JBorrow]

All a MapSet encodes is an arbitrary set of map-shaped items (i.e. fits files that have a geometry on the sky). You can stick ones that make sense together into a collective MapSet (e.g. a sky map, a point source map, an inverse variance map) for colocated data or have them be separate products.

So it makes complete sense to me that you could have:

Collection/
  MapSetA/
    map
    ivar
  MapSetB/
    map
    ivar
    mask
  MapSetC/
    mask

Am I missing something here? There is no requirement currently to provide a specific set of items for a MapSet. Maybe the name is confusing?

[zatkins2]

I agree that product structure makes sense within the schema of "an arbitrary set of map-shaped items." But I fear the following situation leading to a proliferation of mask confusion in the future:

User 1 wants to add a mask into this schema for them to use on MapSetB.map. This mask is not exactly specific to "what MapSetB.map is," say it's some cluster_mask_v1, so they add a new MapSetD.mask product and perhaps add a new Collection within the higher-level Collection here already that includes MapSetB and MapSetD, with a note describing their use of MapSetD.mask on MapSetB.map.

Meanwhile, User 2 thinks that masks should live within the MapSet they are used, and they want to add a similar, somewhat generic cluster_mask_v2 mask into this schema for use on MapSetB.map. They are not worried about overwriting the current MapSetB.mask, because when they do, a new version will be tracked.

So now we have:

Collection/
  MapSetA/
    map
    ivar
  MapSetB/
    map
    ivar
    mask (v1.0.0 of MapSetB)
    cluster_mask_v2 (v1.0.1 of MapSetB)
  MapSetC/
    mask
  MapSetD/
    cluster_mask_v1 (v1.0.0 of MapSetD)

Finally, User 3 wants to compare v1 and v2 of the cluster mask. They must first go to MapSetB.v1.0.1.mask and then to MapSetD.v1.0.0.mask to do so, because User 1 and User 2 had different "mask association conventions" in mind when deciding where to add their mask to hippo. I think that would be a confusing experience for User 3.

[JBorrow]

That's not possible, you can only have one mask per MapSet, so they would need to use MapSetD

[JBorrow]

You can't have arbitrary 'slugs' - there is a pre-defined set (see hippometa/*).

[zatkins2]

Can't you update the mask in MapSetB with a new mask?

[JBorrow]

You could, but it has a description, and it would be very weird to swap it for a specific cluster mask. I'm not sure how this problem is solved by not allowing masks to be a part of MapSets. In that case, you could still have this confusion problem where people make new copies of data that should really be versioned.

[zatkins2]

It does have a description, but it's "locked-in" from the first time someone added that mask slug to that MapSet. If someone had happened to add the cluster mask to the BOSSN maps first, you couldn't have added Merry's mask later, even if everyone else thinks Merry's "should" belong in that MapSet more, because it's more closely associated to that specific map (or more likely to be used by more people) or something.

This is avoided if masks are their own product and associated with maps via a Collection. A new association can just be made via a new Collection (as with any other product).

[zatkins2]

So (assuming people respect descriptions), the toy model issue would be:

Collection/
  MapSetA/
    map
    ivar
  MapSetB/
    map
    ivar
    cluster_mask_v1 (v1.0.0 of MapSetB)
    cluster_mask_v2 (v1.0.1 of MapSetB)
  MapSetC/
    mask
  MapSetD/
    mask_that_everyone_agrees_is_supreme_default_for_MapSetB_but_was_made_a_week_later (v1.0.0 of MapSetD)