Configuration for ngen Logs

[mxkpp]

This adds:

1. File ./bin_mounted/ngen_logging.json to control ngen log levels per module.
2. New variable NGEN_LOG_TO_RTE in config.bashrc, which when set to true, causes ngen logs to be written to a subfolder of ./ngen_logs/ rather than to the built realization's folder.

---

This was tested in conjunction with the following PRs of other repositories:

For essential functionality changes:

NGWPC/nwm-cal-mgr#34

NGWPC/nwm-fcst-mgr#11

For pending EWTS integration:

NGWPC/ngen-forcing#121

And this branch of ngen which also includes updates to various extern submodules: https://github.com/NGWPC/ngen/tree/cmaynard_ewts_separate_repo

[mxkpp]

Note that with the new logging approach of the pending EWTS integration, particularly the per-rank log files, RTE's existing run_tests.py is not able to discover and read the ngen.log files since those will be per-rank. The test script has been edited here to disable aspects related to this log file, and TODOs have been added to return to this after the EWTS changes have been merged.

[kyle-larkin]

This is working as described.

The primary objectives were to enable non-ngenCERF users to specify log levels, and to avoid having to descend very deep into each run directory to retrieve log files. This achieves both of those objectives.

I think I was imagining users would be able to specify their own log location, but this would probably be needlessly complex to do in a safe/secure manner, and in retrospect, probably isn't what I really wanted.

I wonder if the TODO is even necessary, but it's possible/probable that I'm overlooking the advantage.

[kyle-larkin]

What do you see as the advantage of this ngen_log?

[mxkpp]

For forecasts the ngen_log attribute had been referring to the file .../logs/ngen.log. I believe EWTS changes have replaced this with the per-rank log files.

The RTE's forecast test had been reading this file (as well as ngen_stdout_stderr.log) to look for FATAL lines and marking the formulation test as a fail if FATAL lines exist. ngen.log was also being read to enable the test to intentionally stop ngen once it reached a certain step (rather than waiting for completion).

The RTE tests also had been reporting the first and last lines, and lines of certain log levels: SEVERE, CRITICAL, FATAL, for both types of log files.

[mxkpp]

This is working as described.

The primary objectives were to enable non-ngenCERF users to specify log levels, and to avoid having to descend very deep into each run directory to retrieve log files. This achieves both of those objectives.

I think I was imagining users would be able to specify their own log location, but this would probably be needlessly complex to do in a safe/secure manner, and in retrospect, probably isn't what I really wanted.

I wonder if the TODO is even necessary, but it's possible/probable that I'm overlooking the advantage.

utils.configure_ngen_log and/or its caller could be updated to use the existing value of OS env var NGEN_RESULTS_DIR if it is already set. Then the user could set this in config.bashrc or elsewhere. If this was added as a 3rd option to the choices of log path handling, I think it would need to be made clear to the user that using a static path in this way could lead to unexpected outcomes.