Special instrument tests or recipes that are not science data should be clearly and uniquely identified and properly catalogued in the database. These special data have been problematic in the past because the pipeline had no way to identify them and often lead to failures. They should not be part of the normal data processing. Some of these data can be discarded completely, while engineering data may need to be advanced to the level 1 or 2, but not released to the public.
Below I summarize what Ben, Mike, and myself agreed on how to best handle this.
-
Test data that are not useful and should be discarded
Add a new keyword "TEST" which is set to true "T" if the data acquired are bad data, for example taken to test some new software. These data will be ignored by the pipeline. For all other data: science, calibration, and engineering the keyword is set to false "F"
-
Polarimetric calibration
These data are already uniquely identified - no change needed
-
Distortion grid calibration
These data do not need to be processed to the level because the existing code runs on raw data (but this may change if this becomes part of the pipeline). The unique identifier for these kind of data will be the keyword OBS_PLAN in the primary header. The keyword "grid"will be added to the secondary header with values "in" or "out" as for other calibration hardware. The datatype will be "flat" since they are taken with the diffuser in
-
Focus test and radiometric calibration
These data do not need be processed to the level 1 because current software to analyze the data works on the raw data and the pipeline would fail on these data that image the disk of the Sun, not the corona. The unique identifier for these kind of data will be the keyword OBS_PLAN in the primary header. A unique identifier already exists in the extension header "O1ND". When the 'O1ND' is set to "in" the datatype will be "engineering"
-Scans to find the line center
These data need to be processed to the level 1. The unique identifier for these kind of data will be the keyword OBS_PLAN in the primary header. The data type can be left to 'science"
Special instrument tests or recipes that are not science data should be clearly and uniquely identified and properly catalogued in the database. These special data have been problematic in the past because the pipeline had no way to identify them and often lead to failures. They should not be part of the normal data processing. Some of these data can be discarded completely, while engineering data may need to be advanced to the level 1 or 2, but not released to the public.
Below I summarize what Ben, Mike, and myself agreed on how to best handle this.
Test data that are not useful and should be discarded
Add a new keyword "TEST" which is set to true "T" if the data acquired are bad data, for example taken to test some new software. These data will be ignored by the pipeline. For all other data: science, calibration, and engineering the keyword is set to false "F"
Polarimetric calibration
These data are already uniquely identified - no change needed
Distortion grid calibration
These data do not need to be processed to the level because the existing code runs on raw data (but this may change if this becomes part of the pipeline). The unique identifier for these kind of data will be the keyword OBS_PLAN in the primary header. The keyword "grid"will be added to the secondary header with values "in" or "out" as for other calibration hardware. The datatype will be "flat" since they are taken with the diffuser in
Focus test and radiometric calibration
These data do not need be processed to the level 1 because current software to analyze the data works on the raw data and the pipeline would fail on these data that image the disk of the Sun, not the corona. The unique identifier for these kind of data will be the keyword OBS_PLAN in the primary header. A unique identifier already exists in the extension header "O1ND". When the 'O1ND' is set to "in" the datatype will be "engineering"
-Scans to find the line center
These data need to be processed to the level 1. The unique identifier for these kind of data will be the keyword OBS_PLAN in the primary header. The data type can be left to 'science"