Add support for grid search

[sampottinger]

Add support for grid search

Adding support for consider multiple different configuration sets and run them in a grid-search like manner.

Background

We recently added support for external configuration (see commit a6f59ac) and for multiple replicates (see commit ) when running via the command line (see RunCommand and RunRemoteCommand). This allows one to provide jshd files containing precomputed data to be used in simulations (see PreprocessCommand) and allows us to use jshc files which are parsed via JoshConfigParserVisitor. Presently we cannot try alternatives to configuration values in executions of Josh and, instead, one must execute Josh multiple times in order to try different versions. See --data and test_data_option.sh.

Objective

We should add support for grid search where different options for jshd and jshc files. This should look like the following:

--data example.jshc=test_data/example_1.jshc,test_data/example_2.jshc:other.jshd=test_data/other_1.jshd,test_data/other_2.jshd

Under the hypothetical that this is executed with 10 replicates, we should see a total of 40 executions. In order to support this, we will continue writing multiple replicates to a single file (see CsvWriteStrategy and NetcdfWriteStrategy with exception of geotiff see GeotiffWriteStrategy where replicates are broken out). However, we should add support for writing out to 4 files with one per combination in the grid search. This should happen through the following in .josh files:

exportFiles.patch = "file:///tmp/simple_josh_{example}_{other}_.csv"

This should result in the following:

• simple_josh_example_1_other_1.csv: example_1.jshc is run with other_1.jshd with 10 replicates present
• simple_josh_example_2_other_1.csv: example_2.jshc is run with other_1.jshd with 10 replicates present
• simple_josh_example_1_other_2.csv: example_1.jshc is run with other_2.jshd with 10 replicates present
• simple_josh_example_2_other_2.csv: example_2.jshc is run with other_2.jshd with 10 replicates present

Note that the editor writes to memory:// and will not have this capability. We will also still support {replicate} but the replicate column should still be present in the output CSV files or equivalent for netcdf like in file:///tmp/simple_josh_{replicate}.csv.

Implementation

This requires multiple steps for implementation to work our way into this new functionality.

Component 1: Make pipeline package

We should collect some existing functionality into org.joshsim.pipeline. This should include all classes currently in org.joshsim.cloud.pipeline which should be placed in org.joshsim.pipeline.cloud. However, this should also include the following to be placed in org.joshsim.pieline.remote:

• RunRemoteContext
• RunRemoteContextBuilder
• RunRemoteLocalLeaderStrategy
• RunRemoteOffloadLeaderStrategy
• RunRemoteStrategy

Component 2: Make Job object plus builder

Right now we are achieving multiple replicates different ways depending on how the simulation is executed. For example, in RunCommand we have a for loop with a substantial amount of parameters passed to JoshSimFacade.runSimulation. However, in RunRemoteCommand we have RunRemoteContextBuilder serving this role. We also have a file mapping with duplicative parseDataFiles in RunRemoteCommand and RunCommand. Let's clean this up a little bit with a JoshJob and JoshJobBuilder in org.joshsim.pipeline.job. Let's have JoshJob contain a mapping from name of file to the path of file with getFilePath but no setter (immutable). Let's also have JoshJob contain a getter for the number of replicates. Let's then have JoshJobBuilder allow us to set the file path to use for a named file and number of replicates to return these immutable JoshJob files. Let's then have a DataFilesStringParser in org.joshsim.pipeline that replaces the duplicative logic in parseDataFiles. We can pass it a JoshJobBuilder to modify and have it return the same JoshJobBuilder.

Component 2: Align csv and netcdf

We have a replicate column when exporting as CSV (see CsvExportFacade). We should netCDF to the be similar (see NetcdfExportFacade) with an attribute for the replicate number as opposed to separate netCDF files per replicate. All that said, geotiff should work the same as before with different files per replicate.

Component 3: Ensure name of input can vary

We should validate that the name of the file can change and be able to record both the name of the file and its path. Specifically, let's expand JoshJob from mapping String name of file to String file path to mapping String name of file to a JoshJobFileInfo also in org.joshsim.pipeline.job with the name of the file like example_1 and the path like test_data/example_1.jshc accessible via getters but immutable. We can then change DataFilesStringParser to provide these JoshJobFileInfo objects instead.

Component 4: Formalize data input string

Let's start getting prepared for a more complex specification of data files that might exceed what we want to do with basic string manipulation in DataFilesStringParser. We will still use this as an entry point (facade) but we will start getting ready for more complex inputs.

Specifically, let's make an ANTLR grammar that can support the following:

example.jshc=test_data/example_1.jshc:other.jshd=test_data/other_1.jshd

So we are switching from comma separation to colon separation. However, let's use this opportunity to . Please be sure to update our tests including those scripts evaluated in CI / CD (see github actions).

When done, DataFilesStringParser which we should rename to JobVariationParser should still be the entry point and should perform the parse / visit operations. See JshcConfigGetter and ConfigInterpreter. Let's put the new objects we make for parsing these strings into org.joshsim.pipeline.job.config.

Let's call this josh job variation language.

Component 5: Add template to export statements

We presently deal with limited template strings within JvmExportFacade using template.replaceAll and withVariable.replaceAll. Let's centralize this logic into a TemplateStringRenderer within org.joshsim.pipeline.job.config. This should add support for replacing the names of mapped files to the name of the file actually used. For example, consider the following statement:

exportFiles.patch = "file:///tmp/simple_josh_{example}_{other}.csv"

Let's say that the user provided the following:

--data example.jshc=test_data/example_1.jshc:other.jshd=test_data/other_1.jshd

This should cause the output to be placed at /tmp/simple_josh_example_1_other_1.csv. To do this, let's please have TemplateStringRenderer take in a JoshJob. Throw a runtime exception if there is a value within curly braces that can't be replaced because the name is not found.

Component 5: Bring back replicate option

We lost the option to add support for {replicate} to the export filename. Let's take this example:

exportFiles.patch = "file:///tmp/simple_josh_{example}_{other}_{replicate}.csv"

This should result in simple_josh_example_1_other_1.csvwhere example_1.jshc is run with other_1.jshd with 10 replicates present becoming ten files like simple_josh_example_1_other_1_1.csv, simple_josh_example_1_other_1_2.csv, etc. These should still have the replicate column for csv, replicate attribute for netCDF, and geotiffs should work the same as before.

Component 6: Add live test

Let's ensure that we have CI / CD running tests for this updated functionality. See test_data_option.sh. Let's make a new shell script called test_job_config.sh with a josh script in examples like in simulations that takes advantage of exporting with template strings and data files. Let's re-use existing data files if possible. This will provide a valuable integration test in addition to our new unit tests.

Component 7: Allow list but use first

Let's update the ANTLR grammar for josh job variation language to allow for multiple variants per named file. Afterwards, we will be able to support:

--data example.jshc=test_data/example_1.jshc,test_data/example_2.jshc:other.jshd=test_data/other_1.jshd,test_data/other_2.jshd

However, for this component, let's simply ignore everything except the first file given (so ignore example_2.jshc and other_2.jshd). We will add support for them later.

Component 8: Grid search implementation

Let's update JobVariationParser to return an Iterable<JoshJobBuilder> instead of a single JoshJobBuilder. Then, let's add support for using the full list of file targets. So, returning to the following:

--data example.jshc=test_data/example_1.jshc,test_data/example_2.jshc:other.jshd=test_data/other_1.jshd,test_data/other_2.jshd

This should result in 4 JoshJobBuilder objects to be returned. Then, RunCommand and RunRemoteCommand can handle this specific to their logic. See also RunRemoteStrategy.

Component 9: Custom parameters

Let's also add support for --custom-tag tag=value in both RunCommand and RunRemoteCommand. This should work on export strings like:

exportFiles.patch = "file:///tmp/simple_josh_{example}_{other}_{tag}.csv"

Such that we have a file like simple_josh_example_1_other_1_value.csv. Let's include this into JoshJob to be given to JoshJobBuilder before passing to JobVariationParser. Let's also update our live test from component 6 to use / test this functionality.

Component 10: Error in editor

Let's ensure that specifying things in curly braces in memory:// targets in the editor raise a runtime exception indicating that the template strings are not supported in IDE execution.

Notes

We should not refer to component numbers in comments. We should also have a commit after the end of each component and, if possible, tests (./gradlew test checkstyleMain checkstyleTest) should be green in each of those commits. Please keep those commits short (no more than one line no longer than 80 characters) as we will squash.