readme

!$Id:$sidb

Version 11.10. Date:  March 15, 2024

The readme contains the following sections to setup and configure MIRS:
--------------------------------------------------------------------

A) Directory Structure
B) Configuring the MIRS Installation Paths
C) Selecting a Linux Fortran Compiler
D) Note About the Linux g95/gfortran Compilers
E) Selecting an IBM AIX Fortran Compiler
F) Selecting a C and C++ Compiler
G) Selecting the IDL installation location
H) How to Run MIRS Control Panel (MCP)
I) How to Compile the MIRS Source Code
J) MIRS Modes of Operation
K) Important Note on MIRS levels of interaction
L) Potential Problems
M) Testing MIRS for Successful Installation
N) MIRS Operation Scenarios
O) Special Operating Environments (NDE)
P) Quality Control Functionality
Q) More Information


For Detailed Information:
-----------------------

The Microwave Integrated Retrieval System (MIRS) package comes with a 
set of documents that give a wealth of information about the system: 
(1)  physical basis, 
(2)  software architecture, 
(3)  how to compile, 
(4)  how to run, 
(5)  different levels of interaction, 
(6)  resources required, 
(7)  data flow, 
(8)  files description, 
(9)  results and timing efficiency, etc.
(10) performances of all officially delivered parameters, for all sensors considered.

These documents could be found under the /doc directory.
For more information about the details of MIRS, the user is referred 
to these documents, including:

(1) User's Manual,
(2) Interface Control Document,
(3) System Description Document.



A) Directory Structure:
--------------------

The following is a brief description of MIRS directory structure

- doc/ contains all MIRS documents.
- bin/ contains all executables. 
  For those sensor-specific applications, the prefix will distinguish them. 
  Examples: tdr2sdr (sensor-independent), or 
  fm_n18 (footprint matching for NOAA-18 AMSU/MHS)
- data/ contains all data: static data (such as CRTM Coefficients files, 
  covariance matrix, etc.), semi-static data (such as the bias files 
  and regression coefficients),testbed data (both outputs and intermediary files),
  etc. (see more details in the MIRS documents)
- logs/ contains all log files generated by MIRS.
- gui/ contains the GUI-based MIRS Control Panel (MCP) that will make and/or 
  execute any part of (or the whole of) the MIRS process.
- scripts/ contains all scripts (called sequence control scripts or SCS). 
  These may be generated by the MCP.  Samples are provided for testing the system
  and comparisons to benchmark test data.  Newly generated SCS files from the MCP 
  could act as templates by users, to schedule operational running.
- setup/ contains a number of setup control files. 
  Some of them could be controlled by the MCP (the Paths and Configuration 
  Files or PCFs) and others need to be updated manually for root paths ('paths' file).
- src/ all source files are contained under this consolidated directory: 
  Fortran-95, IDL, BASH, JAVA, etc. 



B) Configuring the MIRS Installation Paths:
----------------------------------------

- Go to setup/ directory
- Edit the file 'paths' used by the makefiles across the package.
- Change MIRS_ROOT to the absolute path of your MIRS root level directory
- Change the part of the root path that relates to CRTM_PATH, LIB_PATH and
  EXEC_PATH if they are not located within the MIRS root (default).
- Configure paths to HDF4, HDF5, HDFEOS, NetCDF4, SZIP and ZLIB libraries (needed for
  processing NPP ATMS, FY3, and AMSR-E data only).
- Configure the JAVA_HOME path to local JAVA library (required for MCP only)
- Adopt '-convert big_endian' for 'convEndOption' if using 
  little-endian machines (such as linux-based).
- Adopt '' for 'convEndOption' if using IBM or SGI, fendian for g95, etc
- Edit the file 'paths_idl.pro' to point to the MIRS root path for each MIRS IDL
  library file listed.



C) Selecting a Linux Fortran Compiler:
-----------------------------------

- Usually the make detects automatically which compiler to use. But if you have more than one
  fortran compiler (ifort, g95 and gfortran, etc), you need specify which one to use here.
- By default, MIRS will compile Fortran 90/95 codes using Intel compiler (ifort). If you are
  using Intel, then nothing further is required.
- For using other compilers on Linux, the user needs to point to the right compiler
  by editing the 'make.macros' file located under src/crtm/REL_2.1.1/.
  Search for the string "Define the default Linux flags", there are 5 definitons for macro
  Linux_FLAGS. The default one is Linux_FLAGS_Intel. Currently tested are: Intel, g95 and gfortran:
  
  a)If you are using Linux g95 compiler:
  Linux_FLAGS = $(Linux_FLAGS_g95), be sure to comment the other 4
 
  b)If you are using Linux gfortran compiler:
  Linux_FLAGS = $(Linux_FLAGS_gfortran), be sure to comment the other 4

- If you are using Linux g95/gfortran compiler, ensure that in the generated ${satId}_scs.bash,
  located in scripts/, has the accessStr and formStr set as follows:
	accessStr='stream'
	formStr='unformatted'



D) Note About the Linux g95/gfortran Compilers:
--------------------------------------------

- If running Linux using g95, and the following error occurs:

Error occurred in step: rdr2tdr: calib_f16_ssmis
*** glibc detected *** free(): invalid next size (normal): 0x080cafd0 ***
Aborted

Please set the environment variable MALLOC_CHECK_ to 1, to keep glibc from terminating the process:

export MALLOC_CHECK_=1

- If running Linux gfortran, make sure gfortran supports access='stream' specifier when opening a file.
  Only latest gfortran supports this specifier, otherwise, DMSP(F16/F18) can not be run using gfortran.



E) Selecting an IBM AIX Fortran Compiler:
--------------------------------------

- By default, the IBM AIX compiler is xlf95. No configuration is needed to use xlf95 beyond commenting/
  uncommenting compiler options in the 'setup/paths' file (i.e. the 'make.macros' file does not need
  to be modified to compile on IBM AIX).



F) Selecting a C and C++ Compiler:
-----------------------

- Codes to process some satellite level 1b data are written in C.  The C and C++ compilers
   may be selected in the 'setup/paths' file. 
- For Linux, uncomment CXX and CXX_FLAGS under "Uncomment the following for Linux g++ (C++ 
  Compiler)" and CC and CC_FLAGS under "Uncomment the following for Linux gcc (C compiler)"
- For IBM AIX, uncomment CXX and CXX_FLAGS under "Uncomment the following for IBM AIX xlc++
  (C++ Compiler)" and CC and CC_FLAGS under "Uncomment the following for IBM AIX xlc (C compiler)"
- Comment out the compiler for the platform not used.

G) Selecting the IDL installation location
-----------------------

- Some of the MiRS post-processing steps (e.g. product images, quality control) 
  require the use of IDL codes. The location of the IDL installation is 
  machine dependent. Modify the variable IDL under the major root paths
  section in the PCF to match the IDL installation on your machine. Note
  that IDL is not required to run any steps that generate the MiRS retrieval
  products.

H) How to Run MIRS Control Panel (MCP):
------------------------------------

To run the MIRS Control Panel (MCP), go to gui/ directory and type: 
> make
> make run

The second command will launch the MCP GUI window.



I) How to Compile the MIRS Source Code:
------------------------------------
- Currently, the CRTM library must be compiled before the MIRS source code is compiled
  and linked.  To compile CRTM:
  - go to src/crtm/REL-2.1.1/configure
  - source the configuration for desired compiler to set enviroment variables (e.g. 'source ifort.setup')
    this should match the compiler seleted in make.macros
  - go to src/crtm/REL-2.1.1
  - type 'make'
  - type 'make install'
  - verify that both the include/ and /lib directories contain files

- If you run the MIRS through the MCP, you can simply go to the Preferences tab and 
  turn ON the 'make' option. In this case, the compilation will be done (when necessary) 
  by the MCP itself every time you run some or all MIRS programs.

- You can instead do the compilation manually by going to the src/ 
  directory and typing 'make'. This will compile all source code and create all executables. 
  If you want to make a particular application, you can go to the particular 
  application/program directory and type 'make' there. 
  The required libraries will be made automatically.



J) MIRS Modes of Operation:
------------------------

MIRS has two modes of operation:

(1) Daily mode: 
In this mode, all (or some of the) files contained in a directory (typically named
to correspond to a date e.g. 2006-02-01) will be processed. In this mode, the MIRS script 
could be run either (a) with no argument, in which case, the location of the data will be 
assumed in the PCF 'rdrSensor1Path' path; or it could run (b) with a single argument, 
in which case the argument must be the directory name where the iput data exists. If the
directory name alone is specified, that directory must exist in 'rdrSensor1Path'.  If the absolute
path is given, then 'rdrSensor1Path' will be ignored.

(2) Orbital mode: 
In this mode, only one orbit at a time is processed. 
The MIRS script in this case runs with one single argument only. 
And it needs to be the name of the level 1b orbit file name (with no paths).
For MIRS processing with multiple sensors each with 1evel 1b files, only one
level 1b file name needs to be given to the script (e.g. NOAA-18 AMSU/MHS).
The path to the data is specified in the PCF under 'rdrOrbitPath'.
This is a requirement from the OPUS NOAA operational machine.

Both modes could be controlled with different levels of interactions 
(MCP or SCS) as explained in the 'How to Run' section.



K) Important Note on MIRS levels of interaction:
---------------------------------------------

1) MIRS could be run through the MIRS Control Panel (MCP) tool.  This
is the highest level of interaction (user-friendly).  This is a
GUI-based tool that should make running MIRS with all different
options, an easy task. This mode automatically generates sequence
control scripts (SCS). These are BASH-based scripts.  Note that two
types of scripts are delivered as part of the package and these
correspond to orbital mode processing (more uited to operational
users) and daily mode processing (more fit to Research and
Applications Center STAR users).  It also automatically generates
Paths and Configuration Files (PCFs).  These are files containing
important paths and setup parameters needed by the SCS to run
properly. In this mode, all options are controlled by a click of the
mouse. Including the control of the mode of operation (daily or
orbital). 

PLEASE NOTE that, except for the NDE-specific files, the SCS
and PCFs that are located under the scripts and setup directories are
examples only, and users should generate their own SCS and PCF using
the interactive MCP tool.

2) MIRS could be run through the SCS directly. You could either use the 
template SCS and PCF or you could generate one from the MCP.
You can simply run MIRS by invoquing 'n18_scs arg' or simply 'n18_scs' 
depending on the mode. The SCS will use the configuration as defined in 
the PCF, so it is critical to control the options, modes, etc through manually 
editing the PCF file. Make sure in this case to edit the PCF and change the 
root paths and any other paths. By default, the paths are set to point to the 
delivered MIRS package (relative paths).

3) The third level of interaction -lowest level- in MIRS is performed by 
invoquing directly the executable located under the bin/directory with a 
namelist directed to it. 
The namelists are generated automatically by the script (SCS). 
An example could be: 'bin/fm_n18 <data/ControlData/n18_amsua_mhs_fm_2011-04-01.in'



L) Potential Problems:
-------------------

If you have a 'memory allocation' problem at the NWP step (happened only under IBM). 
It is likely due to memory stack limitation on your machine. 
Please try the following to remedy the problem: 
%ulimit -d unlimited  ( on IBM or Linux ) or,	
%limit datasize unlimited ( on other platforms )



M) Testing MIRS for Successful Installation:
-----------------------------------------

Once the setup/ 'paths' and 'paths_idl.pro' files have been configured,
and the 'make.macros' file has been modified if using a Fortran compiler 
other than Intel, follow these steps to compile, run MIRS, and compare 
the testing outputs to benchmark data located in data/BenchmarkData.

- Go to src/ and type 'make'.  All source code should be compiled without 
  errors. If any errors were encountered, please verify 'paths' file 
  settings and compiler options.
- Verify executables are created by querying the bin/ directory.
- Modify the 'rootPath' variable in the sample PCF file in setup/ that 
  corresponds to the sensor and resolution you want to test. This should 
  match the MIRS_ROOT path of the 'paths' file set to the MIRS install
  directory.
- Run a sample benchmark driver script located in scripts/ corresponding 
  to the PCF modified in the previous step. A script (SCS) and PCF is 
  provided for each operational sensor.  A sample orbit data file for each 
  sensor is located in data/ExternalData/rdr/OrbitalMode. This filename 
  will be passed as the script argument. Note: For NOAA-18, NOAA-19 and
  Metop-A, the AMSU-A file (AMAX) and not the MHS file (MHSX) file may be
  used.  For NPP ATMS, the TATMS file may be used.

For example, to process a test orbit for NOAA-18 AMSU/MHS high resolution:
- Change the 'rootPath' in setup/n18_pcf_benchmark_hr.bash to local MIRS 
  directory
- From the scripts/ directory, run:
	./n18_scs_benchmark_hr.bash NSS.AMAX.NN.D14255.S0051.E0246.B4798283.GC
- Allow MIRS processing to complete.

Similarly, to process a test orbit for DMSP F18 SSMIS high resolution:
- Change the 'rootPath' in setup/f18_pcf_benchmark_hr.bash to local MIRS 
  directory
- From the scripts/ directory, run:
	./f18_scs_benchmark_hr.bash NPR.TDRN.SC.D14255.S0001.E0159.B2526566.NS
- Allow MIRS processing to complete.

To compare new output to the benchmark data:
- Go to the src/qcDelivery.
- Run the IDL program benchmark.pro
- The user will be prompted for input.  Select the number corresponding to 
  the SCS sensor run. A series of PNG image files will be created.
- Exit IDL and view the scatter plots (e.g. N18_tskin.png) and verify that 
  data points fall on the 1-to-1 line. (Note that compiler differences may
  result in a very small number of points that don't agree with 
  the benchmark data.)

- If the user installed MIRS on an IBM AIX platform without IDL, then the 
  user should log into a Linux platform which has IDL. The 'file1' and 'file2' 
  variables in benchmark.pro may have to be modified to absolute 
  paths in order for the program to find the benchmark and newly generated files.



N) MIRS Operation Scenarios:
-------------------------

Once you have installed and tested the MIRS package and verified that
the system is running properly, you will likely customize the
configuration for your specific research and applications
requirements. The large number of path variables, switches, and flags
that control the behavior of the MIRS are critical, but can be
confusing for first-time users. Here we provide several potential
operating scenarios that users may find useful and similar to the
types of scenarios they may be running themselves.  These scenarios
are not intended to be exhaustive, but to indicate some likely
processing sequences, which may help give the user a better idea of
how MIRS functions. For the purposes of this discussion, we assume
that the user has:

(1) already installed and run the GUI-based tool and the MIRS Control
Panel (MCP) to create the initial PCF (Paths and Configuration File)
contained in the setup/ directory, and SCS (Sequence Control Script)
contained in the scripts/ directory, and

(2) uses this PCF as the baseline for subsequent changes which will
correspond to each desired operating scenario. The SCS residing under
the scripts/ directory need not be changed unless a new and distinct
PCF is created for each scenario. In this case, then the section

#-----Include libraries and setup Info
. ../setup/n18_pcf_daily.bash # file name may be different

should be changed in the SCS to reflect the proper PCF file name for
that scenario. In either case, the MIRS SCS is then executed directly
from the command line. We note that it is also certainly possible to
make these modifications and run MIRS strictly at the level of the
GUI, but it is our experience that once the baseline SCS and PCF are
generated for each sensor, it is simpler to edit and run these files
directly.

For each operating scenario, we indicate the combination of switches
and flags to be set in the PCF to obtain the desired operation of
MIRS. We use the term "switches" for those variables which determine
whether or not a particular processing step (e.g. footprint matching
or 1dvar retrieval) is run. The term "flags" is used for those
variables which mostly determine how a particular processing step is
run (e.g. processMode=0 or 1 for orbit or daily processing,
respectively). For these scenarios we assume the sensors to be the 2
cross-track sounders (AMSU and MHS) of NOAA-18/NOAA-19
(i.e. SensorId=1 or SensorId=4). Modifications for processing other
sensor data (i.e. MetopA, F16/F18, AMSRE, etc.) would be relatively
minor.


SCENARIO 1: Complete end-to-end processing of sensor data, starting
with the sensor raw data records (RDRs), all the way through to the
generation of bias figures and figures for data quality monitoring. In
this scenario, we also interpolate available NWP analyses (usually
from GDAS and/or ECMWF) by setting step_nwp=1. We run the forward
radiative transfer model on the collocated NWP profiles to simulate
brightness temperatures by setting step_fwd=1, and we generate figures
comparing the simulated and observed brightness temperatures by
setting step_biasGen=1. Note that both step_choppRadFiles=0 and
step_mergeEdr=0. This corresponds to processing in full orbit
mode. If, for reasons of computational efficiency or timing, it is
desirable to process smaller portions of orbital data, then it is
necessary to set BOTH step_choppRadFiles=1 and
step_mergeEdr=1. Another important element for the 1dvar retrieval
(step_fmsdr2edr=1) is the use of a first guess geophysical profile
(temperature, water vapor, clouds, hydrometeors, surface properties,
etc.) In this example a regression-based first guess is used
(simultaneously setting switch step_externalDataFromRegress=1 which
actually generates the regression data, and flags
externalDataAvailable=1, and externalDataSrc=2 which instruct the
1dvar to use this regression data as a first guess). Alternatively, a
climatology or NWP first guess can be used by setting
externalDataAvailable=0 (climatology), or by setting step_nwp=1,
externalDataAvailable=1, and externalDataSrc=1 (NWP). It should also
be noted that, strictly speaking, if a regression first guess is used
(step_externalDataFromRegress=1, etc.) then NWP data are not used,
and so one can set step_nwp=0, step_fwd=0, and step_biasGen=0 with no
impact on the MIRS retrievals. (However, see discussion in SCENARIO 5
below for impacts on bias figures generation).

# MIRS Controlling Switches
step_rdr2tdrSensor1=1	#RDR->TDR (Sensor1)
step_rdr2tdrSensor2=1	#RDR->TDR (Sensor2)
step_mergeNedt=1	#MERGE NEDTs (Sensor1 and Sensor2)
step_tdr2sdrSensor1=1	#TDR->SDR (Sensor1)
step_tdr2sdrSensor2=1	#TDR->SDR (Sensor2)
step_fm=1	#FOOTPRINT MATCHING
step_nwp=1	#CREATE NWP SCENE (GDAS)
step_fwd=1	#USE FWD OPERATOR ON NWP SCENE
step_biasGen=1	#GENERATE A NEW TB EC
step_choppRadFiles=0	#CHOPPING RADIANCE FILES FOR MULTIPLE PROCESS SUBMISSION
step_externalDataFromRegress=1	#USE OF REGRESSION ALGORIHMS TO GENERATE EXTERN DATA
step_fmsdr2edr=1	#FMSDR->EDR
step_mergeEdr=0	#MERGE THE MINI EDR FILES INTO A FULL ORBITAL EDR FILE 
step_vipp=1	#VERTICAL INTEGRATION AND POST PROCESSING
step_grid=1	#Gridded LEVEL III DATA GENERATION
step_figsGen=1	#FIGS GENERATION
step_biasFigsGen=1	#BIAS FIGS GENERATION
step_dataMonitor=1	#MONITORING OF DATA QUALITY
step_clean=0	#DISK CLEANING/PURGING

NOTE: step_sfr (snow fall rate retrieval) is omitted here, but can be
turned on (set=1). However, this also requires processing ancillary
GFS model data, which is processing environment and implementation-dependent. 
Sample gfs ungrib and correspoding setup script are provided under scripts/ and setup/
They are scripts/gfs_atms_sfr.bash and setup/gfs_atms_sfr_pcf.bash

# MIRS Controlling Flags relating to first guess selection
externalDataAvailable=1
externalDataSrc=2

Summary of switch and flag options to control first guess selection
(note that some switches are also needed for other processing steps such
as bias figure generation):

First Guess step_nwp step_externalDataFromRegress externalDataAvailable externalDataSrc 
----------- -------- ---------------------------- --------------------- ---------------
Regression  Not used		1			1			2
NWP 		1		Not used		1			1
Climatology Not used		Not used		0			Not used


SCENARIO 2: Processing of all inputs to the retrieval system, but
without actually running the 1dvar retrieval itself (and all
subsequent steps). This would be the case when one is interested in
working with and modifying some aspect of the retrieval algorithm
itself. In this case one may be experimenting with different versions
of the retrieval code, running it multiple times, but always on the
same set of observed radiances. Since the preprocessing of the
radiometric data and comparisons with NWP analyses can be
time-consuming, running these steps once and reusing the data can help
save time in the long run.

step_rdr2tdrSensor1=1	#RDR->TDR (Sensor1)
step_rdr2tdrSensor2=1	#RDR->TDR (Sensor2)
step_mergeNedt=1	#MERGE NEDTs (Sensor1 and Sensor2)
step_tdr2sdrSensor1=1	#TDR->SDR (Sensor1)
step_tdr2sdrSensor2=1	#TDR->SDR (Sensor2)
step_fm=1	#FOOTPRINT MATCHING
step_nwp=1	#CREATE NWP SCENE (GDAS)
step_fwd=1	#USE FWD OPERATOR ON NWP SCENE
step_biasGen=1	#GENERATE A NEW TB EC
step_choppRadFiles=0	#CHOPPING RADIANCE FILES FOR MULTIPLE PROCESS SUBMISSION
step_externalDataFromRegress=1	#USE OF REGRESSION ALGORIHMS TO GENERATE EXTERN DATA
step_fmsdr2edr=0	#FMSDR->EDR
step_mergeEdr=0	#MERGE THE MINI EDR FILES INTO A FULL ORBITAL EDR FILE 
step_vipp=0	#VERTICAL INTEGRATION AND POST PROCESSING
step_grid=0	#Gridded LEVEL III DATA GENERATION
step_figsGen=0	#FIGS GENERATION
step_biasFigsGen=0	#BIAS FIGS GENERATION
step_dataMonitor=0	#MONITORING OF DATA QUALITY
step_clean=0	#DISK CLEANING/PURGING


SCENARIO 3: Running all steps beginning with the 1dvar retrieval on
radiometric data and NWP analyses previously processed. This would be
the complement to SCENARIO 2 above. As noted above, once all the input
data have been prepared, one need only run the retrieval and all
subsequent steps each time.

step_rdr2tdrSensor1=0	#RDR->TDR (Sensor1)
step_rdr2tdrSensor2=0	#RDR->TDR (Sensor2)
step_mergeNedt=0	#MERGE NEDTs (Sensor1 and Sensor2)
step_tdr2sdrSensor1=0	#TDR->SDR (Sensor1)
step_tdr2sdrSensor2=0	#TDR->SDR (Sensor2)
step_fm=0	#FOOTPRINT MATCHING
step_nwp=0	#CREATE NWP SCENE (GDAS)
step_fwd=0	#USE FWD OPERATOR ON NWP SCENE
step_biasGen=0	#GENERATE A NEW TB EC
step_choppRadFiles=0	#CHOPPING RADIANCE FILES FOR MULTIPLE PROCESS SUBMISSION
step_externalDataFromRegress=0	#USE OF REGRESSION ALGORIHMS TO GENERATE EXTERN DATA
step_fmsdr2edr=1	#FMSDR->EDR
step_mergeEdr=0	#MERGE THE MINI EDR FILES INTO A FULL ORBITAL EDR FILE 
step_vipp=1	#VERTICAL INTEGRATION AND POST PROCESSING
step_grid=1	#Gridded LEVEL III DATA GENERATION
step_figsGen=1	#FIGS GENERATION
step_biasFigsGen=1	#BIAS FIGS GENERATION
step_dataMonitor=1	#MONITORING OF DATA QUALITY
step_clean=0	#DISK CLEANING/PURGING


SCENARIO 4: Running all steps beginning with the vipp (vertical
integration and post-processing) step on EDRs produced by the
1dvar. In some cases, we may be interested in testing part of the vipp
processing. For example, updating the algorithms for snow cover or
sea ice, which operate on retrieved surface emissivities, affects only
the vipp code, not the 1dvar processing. Running only the vipp step
will greatly speed up algorithm development and testing.

step_rdr2tdrSensor1=0	#RDR->TDR (Sensor1)
step_rdr2tdrSensor2=0	#RDR->TDR (Sensor2)
step_mergeNedt=0	#MERGE NEDTs (Sensor1 and Sensor2)
step_tdr2sdrSensor1=0	#TDR->SDR (Sensor1)
step_tdr2sdrSensor2=0	#TDR->SDR (Sensor2)
step_fm=0	#FOOTPRINT MATCHING
step_nwp=0	#CREATE NWP SCENE (GDAS)
step_fwd=0	#USE FWD OPERATOR ON NWP SCENE
step_biasGen=0	#GENERATE A NEW TB EC
step_choppRadFiles=0	#CHOPPING RADIANCE FILES FOR MULTIPLE PROCESS SUBMISSION
step_externalDataFromRegress=0	#USE OF REGRESSION ALGORIHMS TO GENERATE EXTERN DATA
step_fmsdr2edr=0	#FMSDR->EDR
step_mergeEdr=0	#MERGE THE MINI EDR FILES INTO A FULL ORBITAL EDR FILE 
step_vipp=1	#VERTICAL INTEGRATION AND POST PROCESSING
step_grid=1	#Gridded LEVEL III DATA GENERATION
step_figsGen=1	#FIGS GENERATION
step_biasFigsGen=1	#BIAS FIGS GENERATION
step_dataMonitor=1	#MONITORING OF DATA QUALITY
step_clean=0	#DISK CLEANING/PURGING


SCENARIO 5: As in SCENARIO 3 above, running all steps beginning with
the 1dvar retrieval on radiometric data and NWP analyses previously
processed. However, in this case we do not generate any of the
diagnostic figures at the end of the processing sequence. Figure
generation using the underlying IDL codes is rather time consuming. In
some cases, we might skip some or all of these steps to save
time. This might be the case if we have a stable version of the MIRS
processing codes, and do not need to look at the figures for each day
of processed data. Another possibility is that the user has developed
his/her own suite of diagnostic and graphics codes, and therefore has
no need for the figures generated in these steps. The individual
processing steps needed to generate figures correspond to: gridding
the retrieved fields from the original footprint locations
(step_grid=1), plotting retrieved fields themselves (step_figsGen=1),
plotting differences with NWP analyses (step_biasFigsGen=1), and data
quality monitoring (step_dataMonitor=1). Any or all of these may be
turned off, however if any figures are to be generated, then it is
required that step_grid=1. In addition, if step_biasFigsGen=1, then
the NWP analyses MUST have been previously collocated with the sensor
footprint locations, and the corresponding radiances simulated
(step_nwp=1, and step_fwd=1).

step_rdr2tdrSensor1=0	#RDR->TDR (Sensor1)
step_rdr2tdrSensor2=0	#RDR->TDR (Sensor2)
step_mergeNedt=0	#MERGE NEDTs (Sensor1 and Sensor2)
step_tdr2sdrSensor1=0	#TDR->SDR (Sensor1)
step_tdr2sdrSensor2=0	#TDR->SDR (Sensor2)
step_fm=0	#FOOTPRINT MATCHING
step_nwp=0	#CREATE NWP SCENE (GDAS)
step_fwd=0	#USE FWD OPERATOR ON NWP SCENE
step_biasGen=0	#GENERATE A NEW TB EC
step_choppRadFiles=0	#CHOPPING RADIANCE FILES FOR MULTIPLE PROCESS SUBMISSION
step_externalDataFromRegress=0	#USE OF REGRESSION ALGORIHMS TO GENERATE EXTERN DATA
step_fmsdr2edr=1	#FMSDR->EDR
step_mergeEdr=0	#MERGE THE MINI EDR FILES INTO A FULL ORBITAL EDR FILE 
step_vipp=1	#VERTICAL INTEGRATION AND POST PROCESSING
step_grid=0	#Gridded LEVEL III DATA GENERATION
step_figsGen=0	#FIGS GENERATION
step_biasFigsGen=0	#BIAS FIGS GENERATION
step_dataMonitor=0	#MONITORING OF DATA QUALITY
step_clean=0	#DISK CLEANING/PURGING



O) Special Operating Environments (NDE):
-------------------------------

MIRS has been designed to run in a variety of environments including stand-alone
and operational systems. In limited circumstances, special configuration guidelines
may be presented in this section specific to some environments.

- In the NPOESS Data Exploitation (NDE) environment, the directory src/testbed/mirs2nc 
  contains a makefile that needs to be modified. This folder contains the program and 
  library files to convert MIRS outputs from binary to netCDF 4.  
- To compile the encoder/decoder, modify the 'makefile' as follows depending 
  on operation on IBM AIX or Linux. For IBM AIX, uncomment the INCLUDES and
  LIBS under section "Uncomment for C compiler options on IBM".  For Linux, 
  uncomment the INCLUDES and LIBS under "Uncomment for C compiler options 
  on Linux". Be sure to comment out the variables for respective O/S not being used.
- In the setup/paths file, uncomment/comment out appropriate CC and CC_FLAGS if
  using IBM AIX or Linux.
- In the setup/paths file, set proper library paths for HDF5, netCDF4, and szip.
- In the mirs2nc directory type 'make' to compile the mirs2nc program. The binary
  will be placed in the MIRS EXEC_PATH defined in 'paths', typically bin/. Compilation
  will also occur if any higher level (parent) makefile is invoked.
- This program is invoked by the 'mirs2nc' script function in 
  'scripts/script_functions.bash' which can be called from any SCS driver script. For
  convenience, this call is included in the npp_scs_nde.bash for NDE.

P) Quality Control Functionality
-------------------------------

The MIRS DAP contains functionality related to Quality Control (QC) monitoring of the
MIRS output QC summary flags (convergence, percentage of QC good/bad), sensor
radiometric NEDT, as well as forward model radiometric bias.  The QC is separated into
two tiers.

Tier 1:  Tier 1 QC monitoring consists of monitoring the QC summary flags as well as
the sensor radiometric NEDT.  To enable functionality, the "step_dataMonitor" flag
must be set to "1" in the PCF to turn on the Data Quality Monitoring step. 

Tier 2: Tier 2 QC is more detailed monitoring of radiometric biases. In this QC,
differences between measured and simulated brightness temperatures are calculated
and monitored. This tier of QC requires access to gridded analyses or forecasts
from operational NWP models. Tier 2 is a subset of the QC monitoring done
in the step_biasFigsGen step shown above and also in the stand-alone
QC DAP monitoring described below.

QC DAP: The Quality Control Delivery Algorithm Package is a subset of the
MiRS Package. It is designed to run as a separate process from the actual MiRS
retrieval system which generates the output retrieval product files. However, 
it *requires and assumes* that the MiRS retrievals have been run prior to the 
running of the QC DAP since it relies on the outputs of various MiRS processing steps 
for some of its input. In order to test the QC DAP on a sample of S-NPP/ATMS data
a benchmark testing script (SCS) and associated configuration
file (PCF) are located in the scripts and setup directories, respectively:

scripts/npp_qc_benchmark.bash
setup/npp_pcf_qc_benchmark.bash

- In order run the benchmark script, both the MiRS package (compile instructions above), 
  and the QC DAP package need to be built. 
  Then, compile the QC DAP package manually by going to the src/ 
  directory and typing 'make -f makefile_qc'. This will compile all source code 
  and create all executables needed for the QC DAP. 
- In the case when only the QC DAP is needed, then only the 'make -f makefile_qc'
  command is needed.
   

To process a test granule for ATMS:
- Change the 'rootPath' in setup/npp_pcf_qc_benchmark.bash to local MIRS 
  directory
- Also, change the variable 'email' in the next to last line of setup/npp_pcf_qc_benchmark.bash
  to the user email account which will receive the QC alert messages. Note that
  the automatic email alert will not work if a Unix mail service is not running on the 
  target server.
- From the scripts/ directory, run:
	./npp_qc_benchmark.bash 2012-06-30
- Allow MIRS processing to complete. Because this is a benchmarking script,
  this process will run all the necessary steps,
  including the MiRS retrievals themselves. In normal operations, only the QC 
  steps would be run (see scripts below).

The benchmark script runs the following steps:
- qcRetrieval (checks percentages of internal QC flags and sends email alert if thresholds are exceeded)
- qcNedt (checks measurement noise levels and sends email alert if thresholds are exceeded)
- qcRadioBias (checks difference between static and dynamic radiometric biases and sends email alert if thresholds are exceeded)

After the script has completed, the user should receive an email alert informing them
"The message is triggered when ABS(Delta Bias) > 5 * ABS(nedt)" with a list of 
channels and scan positions and associated values for the static and dynamic biases.
The checks for qc flags and nedt should have passed, and no emails will have been generated
for these steps.

Outside of the NDE computing system, to run the QC DAP 
as part of an automated monitoring system, the following scripts and PCFs are to be used:

n18_qc.bash
n18_pcf_qc.bash

n19_qc.bash
n19_pcf_qc.bash

metopA_qc.bash
metopA_pcf_qc.bash

metopB_qc.bash
metopB_pcf_qc.bash

f16_qc.bash
f16_pcf_qc.bash

f18_qc.bash
f18_pcf_qc.bash

npp_qc.bash
npp_pcf_qc.bash

IMPORTANT: For implementation at NDE the following script is to be used:

npp_qc_nde.bash 

since it has specific features required to run in the NDE computing environment.

Q) More Information:
-----------------

For additional information, please refer to the documents under doc/. 
If help is still needed, please send an email to sid.boukabara@noaa.gov. 
Comments, suggestions for changes/additions, bug reports, are welcome as well.

R) Miscellaneous:
--------------
In this version the use of IMS data (set imsSfcUse=1 and step_ims4retr=1 in the PCF) is only 
implemented for GPM/GMI data processing. For all other satellites, these should be set to 0.

Also visit the web site: www.star.nesdis.noaa.gov/mirs/

-------------------------------------------------------------------------
Written by: Sid-Ahmed Boukabara, NOAA/NESDIS/STAR. June 2007.
Updated by Sid-Ahmed Boukabara, NOAA/NESDIS/STAR. June 2008.
Updated by Chris Grassotti, IMSG at NOAA/NESDIS/STAR. February 2009.
Updated by Kevin Garrett, IMSG at NOAA/NESDIS/STAR. April 2011.
Updated by Chris Grassotti, AER at NOAA/NESDIS/STAR. April 2013.
Updated by Chris Grassotti, U. Maryland/ESSIC/CICS at NOAA/NESDIS/STAR. August 2015.
Updated by Chris Grassotti, U. Maryland/ESSIC/CICS at NOAA/NESDIS/STAR. August 2016.
Updated by Shuyan Liu, Colorado State Univerisy/CIRA at NOAA/NESDIS/STAR. August 2019.