In-memory Finch-ExaCA coupling with multiple Finch simulations

.github/workflows/CI.yml

@@ -124,8 +124,8 @@ jobs:
         uses: actions/checkout@v3
         with:
           repository: ORNL-MDF/Finch
-          # This is after the ability to extract data bounds with an input MPI comm was added
-          ref: 928d56776550e1a546a93197f713fc4d7e1a618d
+          # This is after the capability to extract multilayer temperature data was added
+          ref: 9f79a05ca6ff4fe31525161859e074ea881fc2b6
           path: finch
       - name: Build Finch
         if: ${{ matrix.heat_transfer == 'Finch' }}
@@ -173,7 +173,7 @@ jobs:
         if: ${{ matrix.heat_transfer == 'Finch' }}
         working-directory: utilities/Finch
         run: |
-          $HOME/exaca/bin/Finch-ExaCA inputs_small.json ../../examples/Inp_SmallFinch.json
+          $HOME/exaca/bin/Finch-ExaCA ../../examples/Inp_SmallFinch.json
 
   HIP:
     defaults:

README.md

@@ -219,11 +219,12 @@ Alternatively, the `Finch-ExaCA` executable can be used for coupled Finch-ExaCA
  * `Inp_SmallFinch.json`: simulates melting and solidification of a small melt pool segment at a coarse resolution
  * `Inp_Finch.json`: simulates melting and solidification of a small melt pool segment at a fine resolution, repeated for 3 layers
  * `Inp_FinchTranslate.json`: simulates melting and solidification of the small melt pool segment at the fine resolution translated in space to form 3 overlapping segments
-
-For coupled Finch-ExaCA runs, caution should be taken such that the cell size given in the CA input file matches that given in the Finch input file. Additionally, `scan_path_file` in the Finch input file (e.g. for `Inp_Finch.json` this is `examples/single_line/inputs_small_refined.json` in the Finch repository) should be modified to represent a global path name. Run by calling the created executable with a Finch input file and an ExaCA input file on the command line, with the Finch input file listed first:
+ * `Inp_FinchVariedLayers.json`: performs 3 Finch simulations of simulates melting and solidification of a small melt pool segment at the fine resolution, where each simulation uses a unique value for laser absorption, then uses ExaCA to simulate the solidification microstructure
+For simulation using the `Finch-ExaCA` executable Finch-ExaCA, both the Finch and ExaCA inputs are given in a combined input file given on the command line (see `examples/README.md` for more details on the format of this file, and the Finch README for the required Finch inputs). Alternatively, if the Finch inputs are not present in the combined Finch-ExaCA input file, and the combined simulation requires a single Finch heat transport simulation, separate input files for Finch and ExaCA can be given on the command line with the Finch input file listed first:
 ```
 mpiexec -n 1 ./build/install/bin/Finch-ExaCA $PATH_TO_FINCH/examples/single_line/inputs_small.json examples/Inp_SmallFinch.json
 ```
+For coupled Finch-ExaCA runs, caution should be taken such that the cell size given in the CA input file matches that used for heat transport as given in the Finch input file or the Finch-ExaCA combined input file. Additionally, `scan_path_file` in the Finch input file or in the Finch-ExaCA combined input file (e.g. for `Inp_Finch.json` this is `examples/single_line/inputs_small_refined.json` in the Finch repository) should be modified to represent an accurate file path.
 
 ## Output and post-processing analysis
 

bin/run.cpp

@@ -45,14 +45,116 @@ int main(int argc, char *argv[]) {
             std::string input_file = argv[1];
             // Read input file
             Inputs inputs(id, input_file);
+            std::string simulation_type = inputs.simulation_type;
+            // Full domain solidification - all cells initially liquid or active and end up solid
+            bool full_domain_solidification;
+            if ((simulation_type == "Directional") || (simulation_type == "SingleGrain"))
+                full_domain_solidification = true;
+            else
+                full_domain_solidification = false;
 
             // Setup local and global grids, decomposing domain (needed to construct temperature)
             Grid grid(inputs.simulation_type, id, np, inputs.domain.number_of_layers, inputs.domain, inputs.substrate,
                       inputs.temperature);
             // Temperature fields characterized by data in this structure
             Temperature<memory_space> temperature(grid, inputs.temperature, inputs.print);
 
-            runExaCA(id, np, inputs, timers, grid, temperature);
+            // Material response function
+            InterfacialResponseFunction irf(inputs.domain.deltat, grid.deltax, inputs.irf);
+
+            // Read temperature data if necessary. For the spot problem, store spot melt data as if it were read from a
+            // file
+            if (simulation_type == "FromFile")
+                temperature.readTemperatureData(id, grid, 0);
+            else if (simulation_type == "Spot")
+                temperature.storeSpotData(id, grid, irf.freezingRange(), inputs.domain.deltat,
+                                          inputs.domain.spot_radius);
+
+            // Initialize the temperature fields for the simulation type of interest. These are either simple
+            // unidirectional fields, or more complex data stored in a view in the temperature struct
+            if (full_domain_solidification)
+                temperature.initialize(id, simulation_type, grid, inputs.domain.deltat);
+            else
+                temperature.initialize(0, id, grid, irf.freezingRange(), inputs.domain.deltat);
+            MPI_Barrier(MPI_COMM_WORLD);
+
+            // Initialize grain orientations
+            Orientation<memory_space> orientation(id, inputs.grain_orientation_file, false, inputs.rng_seed,
+                                                  inputs.irf.num_phases, irf.solidificationTransformation());
+            MPI_Barrier(MPI_COMM_WORLD);
+
+            // Initialize cell types, grain IDs, and layer IDs
+            CellData<memory_space> celldata(grid, inputs.substrate, inputs.print.store_melt_pool_edge);
+            if (simulation_type == "Directional")
+                celldata.initSubstrate_Directional(id, grid, inputs.rng_seed);
+            else if (simulation_type == "SingleGrain")
+                celldata.initSubstrate_SingleGrain(id, grid);
+            else
+                celldata.initSubstrate_BaseplatePowder(id, grid, inputs.rng_seed);
+            MPI_Barrier(MPI_COMM_WORLD);
+
+            // Variables characterizing the active cell region within each rank's grid, including buffers for ghost node
+            // data (fixed size) and the steering vector/steering vector size on host/device
+            Interface<memory_space> interface(id, grid.domain_size, inputs.substrate.init_oct_size);
+            // Initialize octahedra for initial active cells, if necessary for this problem type
+            if (full_domain_solidification)
+                createOctahedra_NoRemelt(grid, celldata, temperature, orientation, interface);
+            MPI_Barrier(MPI_COMM_WORLD);
+
+            // Nucleation data structure, containing views of nuclei locations, time steps, and ids, and nucleation
+            // event counters - initialized with an estimate on the number of nuclei in the layer Without knowing
+            // estimated_nuclei_this_rank_this_layer yet, initialize nucleation data structures to estimated sizes,
+            // resize inside of placeNuclei when the number of nuclei per rank is known
+            int estimated_nuclei_this_rank_this_layer =
+                inputs.nucleation.n_max * pow(grid.deltax, 3) * grid.domain_size;
+            Nucleation<memory_space> nucleation(estimated_nuclei_this_rank_this_layer, inputs.nucleation,
+                                                celldata.num_prior_nuclei);
+            // Fill in nucleation data structures, and assign nucleation undercooling values to potential nucleation
+            // events Potential nucleation grains are only associated with liquid cells in layer 0 - they will be
+            // initialized for each successive layer when layer 0 is complete
+            nucleation.placeNuclei(simulation_type, temperature, irf, inputs.rng_seed, 0, grid, id,
+                                   inputs.domain.deltat);
+
+            // Initialize printing struct from inputs
+            Print print(grid, np, inputs.print);
+
+            // End of initialization
+            timers.stopInit();
+            MPI_Barrier(MPI_COMM_WORLD);
+
+            int cycle = 0;
+            timers.startRun();
+
+            // Run ExaCA to model solidification of each layer
+            for (int layernumber = 0; layernumber < grid.number_of_layers; layernumber++) {
+                timers.startLayer();
+                runExaCALayer(id, np, layernumber, cycle, inputs, timers, grid, temperature, irf, orientation, celldata,
+                              interface, nucleation, print, simulation_type, full_domain_solidification);
+
+                if (layernumber != grid.number_of_layers - 1) {
+                    // Initialize new temperature field data for layer "layernumber + 1"
+                    // TODO: reorganize these temperature functions calls into a temperature.init_next_layer as done
+                    // with the substrate If the next layer's temperature data isn't already stored, it should be read
+                    if ((simulation_type == "FromFile") && (inputs.temperature.layerwise_temp_read))
+                        temperature.readTemperatureData(id, grid, layernumber + 1);
+                    MPI_Barrier(MPI_COMM_WORLD);
+
+                    // Initialize next layer of the simulation
+                    initExaCALayer(id, layernumber, cycle, simulation_type, inputs, grid, irf, temperature, celldata,
+                                   interface, nucleation);
+                    MPI_Barrier(MPI_COMM_WORLD);
+                    timers.stopLayer(layernumber);
+                }
+                else {
+                    MPI_Barrier(MPI_COMM_WORLD);
+                    timers.stopLayer();
+                }
+            }
+            timers.stopRun();
+            MPI_Barrier(MPI_COMM_WORLD);
+
+            // Print ExaCA end-of-run data
+            finalizeExaCA(id, np, cycle, inputs, timers, grid, temperature, orientation, celldata, interface, print);
         }
     }
     // Finalize Kokkos