QR erroring out silently with exit code 0 if autophotflat is selected

[agopu]

We've discovered another instance of the pipeline erroring out silently without producing output data and with exit code 0 -- this time when the user selected the autophotflat option.

Running the same job without that option helps produce output data, albeit without that correction.

Reference job on ODI-PPA: https://portal.odi.iu.edu/process/oldjob?jid=8897. Note that this option is not 'selected' by default on the portal but the user requested it; perhaps we should remove the option altogether, we will discuss this with Wilson, et al.

Additional information from Mike Young:

• "It’s a secondary correction. It looks like the QR FITS output is created, then when the autophotflat is applied it deletes the original output but fails to write out the corrected version."
• "Found this in the log when that option is turned on"
  2020-10-20 12:18:32,718 Process-2:23 20190510T205339.1 - OTA21.SCI WARNING Something went wrong with the sky-calculation: (0,)

[youngmd]

Likely a similar fix is needed as was implemented for Swarp in this PR (to skip OTA21 for images taken after it died): #15

[agopu]

@rkotulla Any chance we could get this fixed?

[rkotulla]

I'm trying to find out what's happening here - I've been running the autophotflat option locally a lot without any issues.

[agopu]

@rkotulla I recall vaguely another similar issue that only happened on our compute nodes. Nonetheless, given @youngmd pointed you at a solution with the QR code, I am wondering if you can push through a solution for the part of the code affected given it impacts us for sure? Alternately, we could help you recreate the issue on our nodes but given it's not a consistent bug it's going to be kinda hard.

[rkotulla]

Ah, so the autophotflat problem is related to the missing OTA21? In that case we are all set - I've added to the focalplane layout that OTA21 is simply ignored altogether for all data after Feb 2019. So no OTA to process, no problem with sky estimate, no issue with autophotflat, and all are happy ;-)