Remove diaSrc objects from association based on a set of flags

[rearmstr]

It would be useful to potentially filter out some diaSrc objects when running association based on a set of flags. This could help to remove some of the junk sources in the diaSrc catalogs.

[wmwv]

@shuliu2017 What's your recommended set of flags?

[shuliu2017]

@wmwv I am using base_PixelFlags_flag_bad, base_PixelFlags_flag_saturatedCenter, base_PixelFlags_flag_saturated, base_PixelFlags_flag_edge, base_PixelFlags_flag_offimage.

[shuliu2017]

Below are all the PixelFlags I got from the source table, I am not sure whether we should also include base_PixelFlags_flag_cr and base_PixelFlags_flag_crCenter.

base_PixelFlags_flag
base_PixelFlags_flag_offimage
base_PixelFlags_flag_edge
base_PixelFlags_flag_interpolated
base_PixelFlags_flag_saturated
base_PixelFlags_flag_cr
base_PixelFlags_flag_bad
base_PixelFlags_flag_suspect
base_PixelFlags_flag_interpolatedCenter
base_PixelFlags_flag_saturatedCenter
base_PixelFlags_flag_crCenter
base_PixelFlags_flag_suspectCenter

[wmwv]

Yes, I support adding flag_cr and flat_crCenter to the rejection flags.

[shuliu2017]

Here are all the flags I got from the diaSrc table using the code:

dia_astropy = dia_src.asAstropy()
for col in dia_astropy.columns:
    if re.search('flag', col, re.IGNORECASE):
        print('`'+col+'`')

flags_negative
base_NaiveCentroid_flag
base_NaiveCentroid_flag_noCounts
base_NaiveCentroid_flag_edge
base_NaiveCentroid_flag_resetToPeak
base_PeakCentroid_flag
base_SdssCentroid_flag
base_SdssCentroid_flag_edge
base_SdssCentroid_flag_noSecondDerivative
base_SdssCentroid_flag_almostNoSecondDerivative
base_SdssCentroid_flag_notAtMaximum
base_SdssCentroid_flag_resetToPeak
base_SdssCentroid_flag_badError
ip_diffim_NaiveDipoleCentroid_flag
base_CircularApertureFlux_flag_badCentroid
base_GaussianFlux_flag_badCentroid
base_NaiveCentroid_flag_badInitialCentroid
base_PeakLikelihoodFlux_flag_badCentroid
base_PsfFlux_flag_badCentroid
base_SdssCentroid_flag_badInitialCentroid
base_SdssShape_flag_badCentroid
slot_Centroid_flag
ip_diffim_NaiveDipoleCentroid_pos_flag
slot_Centroid_pos_flag
ip_diffim_NaiveDipoleCentroid_neg_flag
slot_Centroid_neg_flag
base_SdssShape_flag
base_GaussianFlux_flag_badShape
slot_Shape_flag
base_SdssShape_flag_unweightedBad
base_GaussianFlux_flag_badShape_unweightedBad
slot_Shape_flag_unweightedBad
base_SdssShape_flag_unweighted
base_GaussianFlux_flag_badShape_unweighted
slot_Shape_flag_unweighted
base_SdssShape_flag_shift
base_GaussianFlux_flag_badShape_shift
slot_Shape_flag_shift
base_SdssShape_flag_maxIter
base_GaussianFlux_flag_badShape_maxIter
slot_Shape_flag_maxIter
base_SdssShape_flag_psf
base_GaussianFlux_flag_badShape_psf
slot_Shape_flag_psf
base_CircularApertureFlux_3_0_flag
base_CircularApertureFlux_3_0_flag_apertureTruncated
base_CircularApertureFlux_3_0_flag_sincCoeffsTruncated
base_CircularApertureFlux_4_5_flag
base_CircularApertureFlux_4_5_flag_apertureTruncated
base_CircularApertureFlux_4_5_flag_sincCoeffsTruncated
base_CircularApertureFlux_6_0_flag
base_CircularApertureFlux_6_0_flag_apertureTruncated
base_CircularApertureFlux_6_0_flag_sincCoeffsTruncated
base_CircularApertureFlux_9_0_flag
base_CircularApertureFlux_9_0_flag_apertureTruncated
base_CircularApertureFlux_9_0_flag_sincCoeffsTruncated
base_CircularApertureFlux_12_0_flag
slot_ApFlux_flag
base_CircularApertureFlux_12_0_flag_apertureTruncated
slot_ApFlux_flag_apertureTruncated
base_CircularApertureFlux_17_0_flag
base_CircularApertureFlux_17_0_flag_apertureTruncated
base_CircularApertureFlux_25_0_flag
base_CircularApertureFlux_25_0_flag_apertureTruncated
base_CircularApertureFlux_35_0_flag
base_CircularApertureFlux_35_0_flag_apertureTruncated
base_CircularApertureFlux_50_0_flag
base_CircularApertureFlux_50_0_flag_apertureTruncated
base_CircularApertureFlux_70_0_flag
base_CircularApertureFlux_70_0_flag_apertureTruncated
base_GaussianFlux_flag
base_PeakLikelihoodFlux_flag
base_PixelFlags_flag
base_PixelFlags_flag_offimage
base_PixelFlags_flag_edge
base_PixelFlags_flag_interpolated
base_PixelFlags_flag_saturated
base_PixelFlags_flag_cr
base_PixelFlags_flag_bad
base_PixelFlags_flag_suspect
base_PixelFlags_flag_interpolatedCenter
base_PixelFlags_flag_saturatedCenter
base_PixelFlags_flag_crCenter
base_PixelFlags_flag_suspectCenter
base_PsfFlux_flag
slot_PsfFlux_flag
base_PsfFlux_flag_noGoodPixels
slot_PsfFlux_flag_noGoodPixels
base_PsfFlux_flag_edge
slot_PsfFlux_flag_edge
ip_diffim_NaiveDipoleFlux_flag
ip_diffim_NaiveDipoleFlux_pos_flag
ip_diffim_NaiveDipoleFlux_neg_flag
ip_diffim_PsfDipoleFlux_flag
ip_diffim_PsfDipoleFlux_pos_flag
ip_diffim_PsfDipoleFlux_neg_flag
ip_diffim_ClassificationDipole_flag
ip_diffim_DipoleFit_flag_classification
ip_diffim_DipoleFit_flag_classificationAttempted
ip_diffim_DipoleFit_flag
ip_diffim_DipoleFit_flag_edge
ip_diffim_forced_PsfFlux_flag
ip_diffim_forced_PsfFlux_flag_noGoodPixels
ip_diffim_forced_PsfFlux_flag_edge

[wmwv]

I think we should think about flag_edge a bit more. To the degree that it refers to having an edge of a CCD go into the coadd template, but where we have 20 images making up that template, we may not clearly want to throw out such flag_edge objects.

[rearmstr]

I'm wondering if flag_cr is needed. Since we interpolate over cosmic rays, I can see a case where having a cosmic ray in the outer regions of the footprint may have little effect on the measurement.

Another question I have is do you care about tracking what gets rejected by the flag cut? Will there be a case where someone can't find a particular object in an association because it was rejected by a flag. We will have no knowledge of why it was missing unless we know which flags were used by the algorithm which should in principle be available, but can also be hard to find.

[rearmstr]

@wmwv isn't the problem with flag_edge that the object will have missing pixels?

[wmwv]

@wmwv isn't the problem with flag_edge that the object will have missing pixels?

No. flag_edge can be set because the pixels in the template were on the edge of a contributing CCD. But there could be 8 other CCDs that did contribute pixels.

flag_edge as propagated from the calexp indeed represents sources that should be rejected.

[wmwv]

Another question I have is do you care about tracking what gets rejected by the flag cut?

Yes, in the long term, but I don't have a clear vision for how to do this efficiently.

Will there be a case where someone can't find a particular object in an association because it was rejected by a flag.

Yes. But many of those cases it may be in fact correct to reject the object. If, e.g., the center pixel is saturated, we should not count it and that's a legitimate loss of efficiency.

We will have no knowledge of why it was missing unless we know which flags were used by the algorithm which should in principle be available, but can also be hard to find.

But trying to ask (1) "what happened to each fake I put down" is a different question than (2) "what did I recover from the simulations". Question (1) inherently requires a different form of analysis than (2).

[rearmstr]

No. flag_edge can be set because the pixels in the template were on the edge of a contributing CCD. But there could be 8 other CCDs that did contribute pixels.

I'm confused. flag_edge refers to being on the edge of the image. Are you talking about sensor_edge? This indicates pixels on the coadd that had an edge from one of the input images.

[rearmstr]

Yes. But many of those cases it may be in fact correct to reject the object. If, e.g., the center pixel is saturated, we should not count it and that's a legitimate loss of efficiency.

Yes, I'm sure most of these we want to ignore, I was thinking more about the edge cases.

But trying to ask (1) "what happened to each fake I put down" is a different question than (2) "what did I recover from the simulations". Question (1) inherently requires a different form of analysis than (2).

I agree. I'm happy not to worry about it for now. I've just seen many problems that took a long time to track down that were a result of similar choices.

[wmwv]

I'm confused. flag_edge refers to being on the edge of the image. Are you talking about sensor_edge? This indicates pixels on the coadd that had an edge from one of the input images.

Yes, I meant what you describe as sensor_edge. I must have had the flag names wrong in my head. Thank you.