Fringe scaling exception when OTA is extremely saturated

[agopu]

Heidi/Wilson reported QR failures with certain input images within PPA. Mike & I looked into this. We discovered that persistency part of the job finishes despite the job status indicating it's still going on. QR was indeed erroring out of processing the 1st image.

We tried removing the 1st image in case it had a bad FITS file, and discovered the same error cropped up with another image. We suspect this has to do an extremely saturated field in the top left OTA - see ImageX thumbnail view screenshot of a collection of the raw images from that PROPID, in particular watch OTA 56/top left - which in turn is causing QR to error out with this error:

2019-01-11 17:45:36,905 Process-2:83 parallel_collectcells CRITICAL 

=========== EXCEPTION ==============
etype: <type 'exceptions.IndexError'>
error: too many indices for array
stackpos: <traceback object at 0x7fa044530ea8>
---

Traceback (most recent call last):
  File "/usr/local/quickreduce/production//podi_collectcells.py", line 1818, in parallel_collect_reduce_ota
    data_products = collect_reduce_ota(filename, options=options)
  File "/usr/local/quickreduce/production//podi_collectcells.py", line 1244, in collect_reduce_ota
    good_scalings = three_sigma_clip(fringe_scaling[:,6], [0, 1e9])
IndexError: too many indices for array

--- end

We could execute the job without selecting fringe correction but odds are the PI wants fringe correction to be applied. Mike also found that there's a QR commandline argument which could be used to bypass this issue... by calling collectcells with an additional argument of "-skipota=56". However, ideally QR would handle the input data as-is without erroring out so I am filing this bug report (per our IM conversation.) Thanks for looking into this!

[rkotulla]

I've just pushed commit 9007b43 - by default, OTAs used for guiding are now excluded when calculating the global fringe scaling factor. In theory that should fix this problem. I've tried this with some frames (e.g. o20190826T015434.3) and seems to do the trick. If you can dig out what frames were causing this problem and re-test with this latest version that would be awesome.

[agopu]

Thanks @rkotulla - we will test this out asap.