diff --git a/Docs/sphinx_doc/Particles.rst b/Docs/sphinx_doc/Particles.rst index a6d5c8d6e8..dbd69d8a5f 100644 --- a/Docs/sphinx_doc/Particles.rst +++ b/Docs/sphinx_doc/Particles.rst @@ -10,10 +10,6 @@ Particles ERF has the option to include Lagrangian particles in addition to the mesh-based solution. Currently there is one example of particle types available in ERF: tracer_particles. -The particle functionality is very simple and meant for demonstration. - -The particles are initialized one per mesh cell in a vertical plane at :math:`i = 3` for tracer particles. - The tracer particles are advected by the velocity field with optional sedimentation. We note that unless the domain is periodic in the vertical direction, any particles that @@ -30,6 +26,9 @@ different roles in future. Two-dimensional squall line simulation with particles. The particles and contours of cloud water mixing ratio are shown. +Enabling Particles +------------------ + To enable the use of particles, one must set :: @@ -52,25 +51,111 @@ One must also set in the inputs file or on the command line at runtime. -The time at which the particles are initialize can be controlled by a parameter in the inputs file. -For tracer particles one would set this as +Initialization +-------------- + +Particles can be initialized in a box-shaped subregion of the domain by specifying + +:: + + tracer_particles.initial_distribution_type = box + tracer_particles.particle_box_lo = 3.95 -1.0 -1.0 + tracer_particles.particle_box_hi = 4.00 2.0 3.0 + tracer_particles.place_randomly_in_cells = false + +One particle is placed per cell that falls within the specified box. +If ``place_randomly_in_cells`` is true, the particle position within each cell +is randomized; otherwise it is placed at the cell center. + +The time at which the particles are initialized can be controlled by :: tracer_particles.start_time = 0.5 +Verbosity of particle operations can be controlled per species: + +:: + + tracer_particles.verbose = 1 + +Particles with AMR +------------------- + +Tracer particles work with adaptive mesh refinement, including anisotropic +refinement ratios (e.g., ``amr.ref_ratio_vect = 2 1 2``) and terrain-following +coordinates. The particles are automatically redistributed across AMR levels +as they move through the domain. + +With terrain-following (``StaticFittedMesh``) coordinates, the particle vertical +index is mapped between the physical z-coordinate and the terrain-conforming +mesh using the ``z_phys_nd`` field. This mapping is performed during particle +evolution and redistribution so that particles follow the physical coordinate +system rather than the index space. + +The AMR level coupling type is controlled by + +:: -Caveat: the particle information is currently output when using the AMReX-native plotfile format, but not -when using netcdf. Writing particles into the netcdf files is a WIP. + erf.coupling_type = "TwoWay" -To see an example of using the particle functionality, build the executable using gmake in ``ERF/Exec`` setting. -``USE\_PARTICLES = TRUE``. +where ``TwoWay`` (default) enables refluxing and averaging down between coarse +and fine levels, and ``OneWay`` disables the fine-to-coarse feedback. +``OneWay`` is preferred for particle simulations as it avoids potential +instabilities at coarse-fine boundaries. + +.. figure:: figures/ParticleAdvect_AMR2_box.gif + :alt: Tracer particles with 2 AMR levels using static box tagging + :align: center + :width: 100% + + Tracer particle advection over flat terrain with 2 AMR levels and static box + tagging covering a partial z-extent (``inputs_over_flat_AMR2_box_partialz``). + The refined region is fixed in space; particles enter and exit it as they advect. + +.. figure:: figures/ParticleAdvect_AMR2_pcount.gif + :alt: Tracer particles with 2 AMR levels using particle-count tagging + :align: center + :width: 100% + + Tracer particle advection over flat terrain with 2 AMR levels and dynamic + particle-count-based tagging (``inputs_over_flat_AMR2_particlecount``). + The refined region tracks the particles as they move through the domain. + +Particle-count-based refinement +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Particles can drive dynamic AMR by tagging cells for refinement based on the +deposited particle count. This is configured through the standard refinement +indicator mechanism: + +:: + + erf.refinement_indicators = tracer_count_refine + erf.tracer_count_refine.field_name = tracer_particles_count + erf.tracer_count_refine.value_greater = 0.1 + erf.tracer_count_refine.max_level = 1 + erf.tracer_count_refine.start_time = 0.0 + +This refines cells where the deposited tracer particle count exceeds the +specified threshold. When using multiple AMR levels, the particle counts are +averaged down level-by-level (finest to coarsest) to ensure correct tagging +at all levels. + +Output +------ + +The particle information is output when using the AMReX-native plotfile format, but not +when using netcdf. To visualize the number of particles per cell as a mesh-based variable, add -``tracer_particle_count`` (if you have set ``erf.use_tracer_particles``) and +``tracer_particles_count`` (if you have set ``erf.use_tracer_particles``) to the line in the inputs file that begins :: erf.plot_vars_1 = +Example input files for particles with AMR can be found in +``Exec/RegTests/ParticleTests/``. + diff --git a/Docs/sphinx_doc/RegressionTests.rst b/Docs/sphinx_doc/RegressionTests.rst index 7b6c3cd0cd..fb43642c5d 100644 --- a/Docs/sphinx_doc/RegressionTests.rst +++ b/Docs/sphinx_doc/RegressionTests.rst @@ -106,6 +106,15 @@ The following problems are currently tested in the CI: | TaylorGreenAdvectingDiffusing | 16 16 16 | Periodic | Periodic | SlipWall | None | | | | | | | SlipWall | | | +-------------------------------+----------+----------+----------+------------+-------+---------------------------------+ +| ParticleAdvect_AMR1_box | 128 4 32 | Inflow | Periodic | SlipWall | None | particle advection, 1 AMR level | +| | | Outflow | | SlipWall | | static box tagging, partial z | ++-------------------------------+----------+----------+----------+------------+-------+---------------------------------+ +| ParticleAdvect_AMR1_pcount | 128 4 32 | Inflow | Periodic | SlipWall | None | particle advection, 1 AMR level | +| | | Outflow | | SlipWall | | particle-count tagging | ++-------------------------------+----------+----------+----------+------------+-------+---------------------------------+ +| ParticleAdvect_AMR2_pcount | 128 4 32 | Inflow | Periodic | SlipWall | None | particle advection, 2 AMR levels| +| | | Outflow | | SlipWall | | particle-count tagging | ++-------------------------------+----------+----------+----------+------------+-------+---------------------------------+ while the following tests are run nightly: diff --git a/Docs/sphinx_doc/figures/ParticleAdvect_AMR2_box.gif b/Docs/sphinx_doc/figures/ParticleAdvect_AMR2_box.gif new file mode 100644 index 0000000000..45eb36d640 Binary files /dev/null and b/Docs/sphinx_doc/figures/ParticleAdvect_AMR2_box.gif differ diff --git a/Docs/sphinx_doc/figures/ParticleAdvect_AMR2_pcount.gif b/Docs/sphinx_doc/figures/ParticleAdvect_AMR2_pcount.gif new file mode 100644 index 0000000000..5c88000c43 Binary files /dev/null and b/Docs/sphinx_doc/figures/ParticleAdvect_AMR2_pcount.gif differ diff --git a/Exec/RegTests/ParticleTests/README b/Exec/RegTests/ParticleTests/README index cacb58b784..a919545749 100644 --- a/Exec/RegTests/ParticleTests/README +++ b/Exec/RegTests/ParticleTests/README @@ -4,3 +4,12 @@ The terrain-fitted coordinates follow the contour of the hill. This problem is primarily designed to test the implementation of the terrain-fitted coordinates. +The AMR input files test tracer particle advection with adaptive mesh refinement. +The naming convention is inputs_over_{terrain}_{AMRlevels}_{tagging}, where: + - terrain: "flat" (no terrain) or "hill" (Witch of Agnesi profile) + - AMRlevels: "AMR1" (1 refined level) or "AMR2" (2 refined levels) + - tagging: "box_fullz" (static box covering full z-extent), + "box_partialz" (static box covering partial z-extent), + "particlecount" (dynamic tagging based on deposited particle count) + +All cases use anisotropic refinement (2,1,2) per level. diff --git a/Exec/RegTests/ParticleTests/inputs_over_flat b/Exec/RegTests/ParticleTests/inputs_over_flat index ecd89de630..098e20b435 100644 --- a/Exec/RegTests/ParticleTests/inputs_over_flat +++ b/Exec/RegTests/ParticleTests/inputs_over_flat @@ -3,7 +3,7 @@ erf.prob_name = "Particles Over Flat Ground" erf.init_type = Isentropic -max_step = 80 +max_step = 400 amrex.fpe_trap_invalid = 1 @@ -23,6 +23,7 @@ xlo.velocity = 10. 0. 0. xlo.density = 1.16 xlo.theta = 300. xlo.scalar = 0. +xlo.nonreflecting = true zlo.type = "SlipWall" zhi.type = "SlipWall" @@ -30,9 +31,10 @@ zhi.type = "SlipWall" # PARTICLES erf.use_tracer_particles = 1 tracer_particles.initial_distribution_type = box -tracer_particles.particle_box_lo = 3.95 -1.0 -1.0 -tracer_particles.particle_box_hi = 4.00 2.0 3.0 -tracer_particles.place_randomly_in_cells = true +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 # TIME STEP CONTROL erf.fixed_dt = 1E-3 @@ -43,32 +45,19 @@ erf.sum_interval = 1 # timesteps between computing mass erf.v = 1 # verbosity in ERF.cpp amr.v = 1 # verbosity in Amr.cpp -# REFINEMENT / REGRIDDING -amr.max_level = 0 # maximum level number allowed - # CHECKPOINT FILES erf.check_file = chk # root name of checkpoint file -erf.check_int = 100 # number of timesteps between checkpoints +erf.check_int = -1 # number of timesteps between checkpoints # PLOTFILES erf.plot_file_1 = plt # prefix of plotfile name -erf.plot_int_1 = 50 # number of timesteps between plotfiles -erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density # SOLVER CHOICE erf.use_gravity = true -erf.use_coriolis = false erf.les_type = "None" -# MULTILEVEL -amr.max_level = 0 -amr.ref_ratio_vect = 2 2 1 - -erf.refinement_indicators = box1 -erf.box1.max_level = 1 -erf.box1.in_box_lo = 2. 0.25 -erf.box1.in_box_hi = 8. 0.75 - erf.dycore_horiz_adv_type = Centered_2nd erf.dycore_vert_adv_type = Centered_2nd erf.dryscal_horiz_adv_type = Centered_2nd @@ -83,8 +72,9 @@ erf.alpha_T = 0.0 # [m^2/s] #erf.abl_pressure_grad = -0.2 0. 0. # PROBLEM PARAMETERS (optional) -prob.custom_terrain_type = "FlatRaised" - prob.T_0 = 300.0 prob.U_0 = 10.0 prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 1 diff --git a/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_box_fullz b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_box_fullz new file mode 100644 index 0000000000..58ed4ea912 --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_box_fullz @@ -0,0 +1,91 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# REFINEMENT / REGRIDDING +amr.max_level = 0 # maximum level number allowed + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 1 +erf.box1.in_box_lo = 4.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_box_partialz b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_box_partialz new file mode 100644 index 0000000000..c650d6c78d --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_box_partialz @@ -0,0 +1,91 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# REFINEMENT / REGRIDDING +amr.max_level = 0 # maximum level number allowed + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 1 +erf.box1.in_box_lo = 4.0 0.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 0.5 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_particlecount b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_particlecount new file mode 100644 index 0000000000..782bc55954 --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_particlecount @@ -0,0 +1,90 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +erf.coupling_type = "OneWay" +amr.n_error_buf = 2 +erf.regrid_int = 1 + +erf.refinement_indicators = tracer_count_refine +erf.tracer_count_refine.max_level = 1 +erf.tracer_count_refine.field_name = tracer_particles_count +erf.tracer_count_refine.value_greater = 1.0e-1 +erf.tracer_count_refine.start_time = 0.0 diff --git a/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_box_fullz b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_box_fullz new file mode 100644 index 0000000000..d6f4a395ca --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_box_fullz @@ -0,0 +1,88 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 2 +amr.ref_ratio_vect = 2 1 2 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 2 +erf.box1.in_box_lo = 4.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_box_partialz b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_box_partialz new file mode 100644 index 0000000000..b30d1a4704 --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_box_partialz @@ -0,0 +1,88 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 2 +amr.ref_ratio_vect = 2 1 2 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 2 +erf.box1.in_box_lo = 4.0 0.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 0.5 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_particlecount b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_particlecount new file mode 100644 index 0000000000..708d892278 --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_flat_AMR2_particlecount @@ -0,0 +1,90 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 2 +amr.ref_ratio_vect = 2 1 2 2 1 2 +erf.coupling_type = "OneWay" +amr.n_error_buf = 4 +erf.regrid_int = 1 + +erf.refinement_indicators = tracer_count_refine +erf.tracer_count_refine.max_level = 2 +erf.tracer_count_refine.field_name = tracer_particles_count +erf.tracer_count_refine.value_greater = 1.0e-1 +erf.tracer_count_refine.start_time = 0.0 diff --git a/Exec/RegTests/ParticleTests/inputs_over_hill b/Exec/RegTests/ParticleTests/inputs_over_hill index 02b3c91e1a..8f968c635f 100644 --- a/Exec/RegTests/ParticleTests/inputs_over_hill +++ b/Exec/RegTests/ParticleTests/inputs_over_hill @@ -3,7 +3,7 @@ erf.prob_name = "Particles Over Witch of Agnesi Hill" erf.init_type = Isentropic -max_step = 500 +max_step = 400 amrex.fpe_trap_invalid = 1 @@ -31,9 +31,10 @@ zhi.type = "SlipWall" # PARTICLES erf.use_tracer_particles = 1 tracer_particles.initial_distribution_type = box -tracer_particles.particle_box_lo = 1.95 -1.0 -1.0 -tracer_particles.particle_box_hi = 2.00 2.0 1.0 +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 # TIME STEP CONTROL erf.fixed_dt = 1E-3 @@ -44,12 +45,9 @@ erf.sum_interval = 1 # timesteps between computing mass erf.v = 1 # verbosity in ERF.cpp amr.v = 1 # verbosity in Amr.cpp -# REFINEMENT / REGRIDDING -amr.max_level = 0 # maximum level number allowed - # CHECKPOINT FILES erf.check_file = chk # root name of checkpoint file -erf.check_int = 100 # number of timesteps between checkpoints +erf.check_int = -1 # number of timesteps between checkpoints # PLOTFILES erf.plot_file_1 = plt # prefix of plotfile name @@ -58,17 +56,8 @@ erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pr # SOLVER CHOICE erf.use_gravity = true -erf.use_coriolis = false erf.les_type = "None" -# MULTILEVEL -amr.max_level = 0 -amr.ref_ratio_vect = 2 2 1 - -erf.refinement_indicators = box1 -erf.box1.max_level = 1 -erf.box1.in_box_lo = 2. 0.25 -erf.box1.in_box_hi = 8. 0.75 # TERRRAIN GRID TYPE erf.terrain_type = StaticFittedMesh @@ -93,3 +82,6 @@ prob.U_0 = 10.0 prob.rho_0 = 1.16 prob.custom_terrain_type = "WoA" + +# MULTILEVEL +amr.max_level = 0 diff --git a/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_box_fullz b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_box_fullz new file mode 100644 index 0000000000..5f8052788d --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_box_fullz @@ -0,0 +1,97 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Witch of Agnesi Hill" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# REFINEMENT / REGRIDDING +amr.max_level = 0 # maximum level number allowed + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# TERRRAIN GRID TYPE +erf.terrain_type = StaticFittedMesh +erf.terrain_smoothing = 0 + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +prob.custom_terrain_type = "WoA" + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 1 +erf.box1.in_box_lo = 4.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_box_partialz b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_box_partialz new file mode 100644 index 0000000000..0cc3379353 --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_box_partialz @@ -0,0 +1,97 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Witch of Agnesi Hill" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# REFINEMENT / REGRIDDING +amr.max_level = 0 # maximum level number allowed + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# TERRRAIN GRID TYPE +erf.terrain_type = StaticFittedMesh +erf.terrain_smoothing = 0 + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +prob.custom_terrain_type = "WoA" + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 1 +erf.box1.in_box_lo = 4.0 0.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 0.5 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_particlecount b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_particlecount new file mode 100644 index 0000000000..50f453937e --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR1_particlecount @@ -0,0 +1,96 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Witch of Agnesi Hill" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# TERRRAIN GRID TYPE +erf.terrain_type = StaticFittedMesh +erf.terrain_smoothing = 0 + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +prob.custom_terrain_type = "WoA" + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +erf.coupling_type = "OneWay" +amr.n_error_buf = 2 +erf.regrid_int = 1 + +erf.refinement_indicators = tracer_count_refine +erf.tracer_count_refine.max_level = 1 +erf.tracer_count_refine.field_name = tracer_particles_count +erf.tracer_count_refine.value_greater = 1.0e-1 +erf.tracer_count_refine.start_time = 0.0 diff --git a/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_box_fullz b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_box_fullz new file mode 100644 index 0000000000..aeea6eb18d --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_box_fullz @@ -0,0 +1,94 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Witch of Agnesi Hill" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# TERRRAIN GRID TYPE +erf.terrain_type = StaticFittedMesh +erf.terrain_smoothing = 0 + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +prob.custom_terrain_type = "WoA" + +# MULTILEVEL +amr.max_level = 2 +amr.ref_ratio_vect = 2 1 2 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 2 +erf.box1.in_box_lo = 4.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_box_partialz b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_box_partialz new file mode 100644 index 0000000000..dc4a6a1f54 --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_box_partialz @@ -0,0 +1,94 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Witch of Agnesi Hill" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# TERRRAIN GRID TYPE +erf.terrain_type = StaticFittedMesh +erf.terrain_smoothing = 0 + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +prob.custom_terrain_type = "WoA" + +# MULTILEVEL +amr.max_level = 2 +amr.ref_ratio_vect = 2 1 2 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 2 +erf.box1.in_box_lo = 4.0 0.0 0.0 +erf.box1.in_box_hi = 6.0 1.0 0.5 + diff --git a/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_particlecount b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_particlecount new file mode 100644 index 0000000000..953eea2c05 --- /dev/null +++ b/Exec/RegTests/ParticleTests/inputs_over_hill_AMR2_particlecount @@ -0,0 +1,96 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Witch of Agnesi Hill" + +erf.init_type = Isentropic + +max_step = 400 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 256 8 64 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 2.95 -1.0 -1.0 +tracer_particles.particle_box_hi = 3.00 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 20 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# TERRRAIN GRID TYPE +erf.terrain_type = StaticFittedMesh +erf.terrain_smoothing = 0 + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +prob.custom_terrain_type = "WoA" + +# MULTILEVEL +amr.max_level = 2 +amr.ref_ratio_vect = 2 1 2 2 1 2 +erf.coupling_type = "OneWay" +amr.n_error_buf = 4 +erf.regrid_int = 1 + +erf.refinement_indicators = tracer_count_refine +erf.tracer_count_refine.max_level = 2 +erf.tracer_count_refine.field_name = tracer_particles_count +erf.tracer_count_refine.value_greater = 1.0e-1 +erf.tracer_count_refine.start_time = 0.0 diff --git a/Source/ERF.cpp b/Source/ERF.cpp index 8a2a0a024f..96a7eeda78 100644 --- a/Source/ERF.cpp +++ b/Source/ERF.cpp @@ -769,7 +769,7 @@ ERF::post_timestep (int nstep, Real time, Real dt_lev0) BL_PROFILE("ERF::post_timestep()"); #ifdef ERF_USE_PARTICLES - particleData.Redistribute(); + particleData.Redistribute(z_phys_nd); #endif if (solverChoice.coupling_type == CouplingType::TwoWay) @@ -1670,6 +1670,14 @@ ERF::InitData_post () } } +#ifdef ERF_USE_PARTICLES + // Redistribute particles so the container has valid data at all AMR levels + // before the initial plotfile write + if (finest_level > 0) { + particleData.Redistribute(z_phys_nd); + } +#endif + // check for additional plotting variables that are available after particle containers // are setup. const std::string& pv3d_1 = "plot_vars_1" ; appendPlotVariables(pv3d_1,plot3d_var_names_1); diff --git a/Source/ERF_MakeNewLevel.cpp b/Source/ERF_MakeNewLevel.cpp index 45604693a8..dd7fc9b16f 100644 --- a/Source/ERF_MakeNewLevel.cpp +++ b/Source/ERF_MakeNewLevel.cpp @@ -266,9 +266,9 @@ void ERF::MakeNewLevelFromScratch (int lev, Real time, const BoxArray& ba_in, if (restart_chkfile.empty()) { if (lev == 0) { initializeTracers((ParGDBBase*)GetParGDB(),z_phys_nd,time); - } else { - particleData.Redistribute(); } + // For lev > 0: particle redistribute is handled in timeStep() AFTER + // regrid() completes, not here inside MakeNewLevelFromCoarse. } #endif } @@ -529,9 +529,6 @@ ERF::MakeNewLevelFromCoarse (int lev, Real time, const BoxArray& ba, initRayleigh_at_level(lev); } -#ifdef ERF_USE_PARTICLES - // particleData.Redistribute(); -#endif } // Remake an existing level using provided BoxArray and DistributionMapping and @@ -807,9 +804,8 @@ ERF::RemakeLevel (int lev, Real time, const BoxArray& ba, const DistributionMapp initRayleigh_at_level(lev); } -#ifdef ERF_USE_PARTICLES - particleData.Redistribute(); -#endif + // Particle redistribute handled in timeStep() after regrid() completes. + // Calling it here causes stale-grid crashes. } // diff --git a/Source/ERF_Tagging.cpp b/Source/ERF_Tagging.cpp index ab10e0eb45..187b94bf4a 100644 --- a/Source/ERF_Tagging.cpp +++ b/Source/ERF_Tagging.cpp @@ -288,25 +288,34 @@ ERF::ErrorEst (int levc, TagBoxArray& tags, Real time, int /*ngrow*/) for (ParticlesNamesVector::size_type i = 0; i < particles_namelist.size(); i++) { std::string tmp_string(particles_namelist[i]+"_count"); - IntVect rr = IntVect::TheUnitVector(); if (ref_tags[j].Field() == tmp_string) { - for (int lev = levc; lev <= finest_level; lev++) - { - MultiFab temp_dat(grids[lev], dmap[lev], 1, 0); temp_dat.setVal(0); - particleData[particles_namelist[i]]->IncrementWithTotal(temp_dat, lev); - - MultiFab temp_dat_crse(grids[levc], dmap[levc], 1, 0); temp_dat_crse.setVal(0); - - if (lev == levc) { - MultiFab::Copy(*mf, temp_dat, 0, 0, 1, 0); - } else { - for (int d = 0; d < AMREX_SPACEDIM; d++) { - rr[d] *= ref_ratio[levc][d]; - } - average_down(temp_dat, temp_dat_crse, 0, 1, rr); - MultiFab::Add(*mf, temp_dat_crse, 0, 0, 1, 0); + auto* pc = particleData[particles_namelist[i]]; + pc->resizeData(); + int pc_nlevs = static_cast(pc->GetParticles().size()); + + // Deposit particle counts at each level into per-level MultiFabs + Vector count_per_lev(finest_level+1); + for (int lev = levc; lev <= finest_level; lev++) { + count_per_lev[lev].define(grids[lev], dmap[lev], 1, 0); + count_per_lev[lev].setVal(0); + if (lev < pc_nlevs) { + pc->IncrementWithTotal(count_per_lev[lev], lev); } } + + // Average down level-by-level from finest to levc. + // This avoids multi-level coarsening (e.g. L2->L0 with + // ratio (4,1,4)) which can fail when fine-level boxes + // are not aligned to the composite refinement ratio. + for (int lev = finest_level; lev > levc; lev--) { + MultiFab temp_crse(grids[lev-1], dmap[lev-1], 1, 0); + temp_crse.setVal(0); + average_down(count_per_lev[lev], temp_crse, + 0, 1, ref_ratio[lev-1]); + MultiFab::Add(count_per_lev[lev-1], temp_crse, 0, 0, 1, 0); + } + + MultiFab::Copy(*mf, count_per_lev[levc], 0, 0, 1, 0); } } #endif @@ -485,39 +494,37 @@ ERF::refinement_criteria_setup () khi = static_cast((rbox_hi[2] - plo[2])/dx[2]-1); } - Box bx(IntVect(ilo,jlo,klo),IntVect(ihi,jhi,khi)); - // Error check for each index - if(ilo%ref_ratio[lev_for_box-1][0] != 0){ - amrex::Print()<< "Requested in_box_lo in x direction = " << rbox_lo[0] << " corresponds to ilo = " << ilo << std::endl; - amrex::Print() << "ilo = " << ilo << " is not divisible by ref_ratio in x direction = " << ref_ratio[lev_for_box-1][0] << std::endl; - amrex::Error("Adjust in_box_lo in x-direction to be divisible by ref_ratio and try again"); - } - if((ihi+1)%ref_ratio[lev_for_box-1][0] != 0){ - amrex::Print()<< "Requested in_box_hi in x direction = " << rbox_hi[0] << " corresponds to ihi+1 = " << ihi+1 << std::endl; - amrex::Print() << "ihi+1 = " << ihi+1 << " is not divisible by ref_ratio in x direction = " << ref_ratio[lev_for_box-1][0] << std::endl; - amrex::Error("Adjust in_box_hi in x-direction to be divisible by ref_ratio and try again"); - } - if(jlo%ref_ratio[lev_for_box-1][1] != 0){ - amrex::Print()<< "Requested in_box_lo in y direction = " << rbox_lo[1] << " corresponds to jlo = " << jlo << std::endl; - amrex::Print() << "jlo = " << jlo << " is not divisible by ref_ratio in y direction = " << ref_ratio[lev_for_box-1][1] << std::endl; - amrex::Error("Adjust in_box_lo in y-direction to be divisible by ref_ratio and try again"); - } - if((jhi+1)%ref_ratio[lev_for_box-1][1] != 0){ - amrex::Print()<< "Requested in_box_hi in y direction = " << rbox_hi[1] << " corresponds to jhi+1 = " << jhi+1 << std::endl; - amrex::Print() << "jhi+1 = " << jhi+1 << " is not divisible by ref_ratio in y direction = " << ref_ratio[lev_for_box-1][1] << std::endl; - amrex::Error("Adjust in_box_hi in y-direction to be divisible by ref_ratio and try again"); - } - if(klo%ref_ratio[lev_for_box-1][2] != 0){ - amrex::Print()<< "Requested in_box_lo in z direction = " << rbox_lo[2] << " corresponds to klo = " << klo << std::endl; - amrex::Print() << "klo = " << klo << " is not divisible by ref_ratio in z direction = " << ref_ratio[lev_for_box-1][2] << std::endl; - amrex::Error("Adjust in_box_lo in z-direction to be divisible by ref_ratio and try again"); - } - if((khi+1)%ref_ratio[lev_for_box-1][2] != 0){ - amrex::Print()<< "Requested in_box_hi in z direction = " << rbox_hi[2] << " corresponds to khi+1 = " << khi+1 << std::endl; - amrex::Print() << "khi+1 = " << khi+1 << " is not divisible by ref_ratio in z direction = " << ref_ratio[lev_for_box-1][2] << std::endl; - amrex::Error("Adjust in_box_hi in z-direction to be divisible by ref_ratio and try again"); + // Snap box indices to ref_ratio alignment (round lo down, hi up) + { + const auto& rr = ref_ratio[lev_for_box-1]; + auto snap_lo = [](int idx, int r) { return idx - (idx % r + r) % r; }; + auto snap_hi = [](int idx_p1, int r) { // idx_p1 = ihi+1 + int rem = idx_p1 % r; + return (rem == 0) ? idx_p1 - 1 : idx_p1 + (r - rem) - 1; + }; + int ilo_old = ilo, jlo_old = jlo, klo_old = klo; + int ihi_old = ihi, jhi_old = jhi, khi_old = khi; + ilo = snap_lo(ilo, rr[0]); + jlo = snap_lo(jlo, rr[1]); + klo = snap_lo(klo, rr[2]); + ihi = snap_hi(ihi+1, rr[0]); + jhi = snap_hi(jhi+1, rr[1]); + khi = snap_hi(khi+1, rr[2]); + if (ilo != ilo_old || ihi != ihi_old || + jlo != jlo_old || jhi != jhi_old || + klo != klo_old || khi != khi_old) { + amrex::Print() << "Refinement box indices snapped to ref_ratio alignment:\n" + << " ilo: " << ilo_old << " -> " << ilo + << " ihi: " << ihi_old << " -> " << ihi + << " jlo: " << jlo_old << " -> " << jlo + << " jhi: " << jhi_old << " -> " << jhi + << " klo: " << klo_old << " -> " << klo + << " khi: " << khi_old << " -> " << khi << "\n"; + } } + Box bx(IntVect(ilo,jlo,klo),IntVect(ihi,jhi,khi)); + bool using_pbl = (solverChoice.turbChoice[lev_for_box].pbl_type == PBLType::MYJ || solverChoice.turbChoice[lev_for_box].pbl_type == PBLType::MYNN25 || solverChoice.turbChoice[lev_for_box].pbl_type == PBLType::MYNNEDMF || @@ -577,41 +584,35 @@ ERF::refinement_criteria_setup () Print() << "Reading " << bx << " at level " << lev_for_box << std::endl; num_boxes_at_level[lev_for_box] += 1; - if(box_lo[0]%ref_ratio[lev_for_box-1][0] != 0){ - amrex::Print()<< "Requested ilo in x-direction : " << box_lo[0] << std::endl; - amrex::Print() << "ilo = " << box_lo[0] << " is not divisible by ref_ratio in x direction = " << - ref_ratio[lev_for_box-1][0] << std::endl; - amrex::Error("Adjust in_box_lo_indices in x-direction to be divisible by ref_ratio and try again"); - } - if((box_hi[0]+1)%ref_ratio[lev_for_box-1][0] != 0){ - amrex::Print()<< "Requested ihi in x-direction : " << box_hi[0] << std::endl; - amrex::Print() << "ihi+1 = " << box_hi[0]+1 << " is not divisible by ref_ratio in x direction = " << - ref_ratio[lev_for_box-1][0] << std::endl; - amrex::Error("Adjust in_box_hi_indices in x-direction to be divisible by ref_ratio and try again"); - } - if(box_lo[1]%ref_ratio[lev_for_box-1][1] != 0){ - amrex::Print()<< "Requested jlo in y-direction : " << box_lo[1] << std::endl; - amrex::Print() << "jlo = " << box_lo[1] << " is not divisible by ref_ratio in y direction = " << - ref_ratio[lev_for_box-1][1] << std::endl; - amrex::Error("Adjust in_box_lo_indices in y-direction to be divisible by ref_ratio and try again"); - } - if((box_hi[1]+1)%ref_ratio[lev_for_box-1][1] != 0){ - amrex::Print()<< "Requested jhi in y-direction : " << box_hi[1] << std::endl; - amrex::Print() << "jhi+1 = " << box_hi[1]+1 << " is not divisible by ref_ratio in y direction = " << - ref_ratio[lev_for_box-1][1] << std::endl; - amrex::Error("Adjust in_box_hi_indices in y-direction to be divisible by ref_ratio and try again"); - } - if(box_lo[2]%ref_ratio[lev_for_box-1][2] != 0){ - amrex::Print()<< "Requested klo in z-direction : " << box_lo[2] << std::endl; - amrex::Print() << "klo = " << box_lo[2] << " is not divisible by ref_ratio in z direction = " << - ref_ratio[lev_for_box-1][2] << std::endl; - amrex::Error("Adjust in_box_lo_indices in z-direction to be divisible by ref_ratio and try again"); - } - if((box_hi[2]+1)%ref_ratio[lev_for_box-1][2] != 0){ - amrex::Print()<< "Requested khi in z-direction : " << box_hi[2] << std::endl; - amrex::Print() << "khi+1 = " << box_hi[2]+1 << " is not divisible by ref_ratio in z direction = " << - ref_ratio[lev_for_box-1][2] << std::endl; - amrex::Error("Adjust in_box_hi_indices in z-direction to be divisible by ref_ratio and try again"); + // Snap box indices to ref_ratio alignment (round lo down, hi up) + { + const auto& rr = ref_ratio[lev_for_box-1]; + auto snap_lo_fn = [](int idx, int r) { return idx - (idx % r + r) % r; }; + auto snap_hi_fn = [](int idx_p1, int r) { + int rem = idx_p1 % r; + return (rem == 0) ? idx_p1 - 1 : idx_p1 + (r - rem) - 1; + }; + int lo_old[3] = {box_lo[0], box_lo[1], box_lo[2]}; + int hi_old[3] = {box_hi[0], box_hi[1], box_hi[2]}; + box_lo[0] = snap_lo_fn(box_lo[0], rr[0]); + box_lo[1] = snap_lo_fn(box_lo[1], rr[1]); + box_lo[2] = snap_lo_fn(box_lo[2], rr[2]); + box_hi[0] = snap_hi_fn(box_hi[0]+1, rr[0]); + box_hi[1] = snap_hi_fn(box_hi[1]+1, rr[1]); + box_hi[2] = snap_hi_fn(box_hi[2]+1, rr[2]); + if (box_lo[0] != lo_old[0] || box_hi[0] != hi_old[0] || + box_lo[1] != lo_old[1] || box_hi[1] != hi_old[1] || + box_lo[2] != lo_old[2] || box_hi[2] != hi_old[2]) { + amrex::Print() << "Refinement box indices snapped to ref_ratio alignment:\n" + << " ilo: " << lo_old[0] << " -> " << box_lo[0] + << " ihi: " << hi_old[0] << " -> " << box_hi[0] + << " jlo: " << lo_old[1] << " -> " << box_lo[1] + << " jhi: " << hi_old[1] << " -> " << box_hi[1] + << " klo: " << lo_old[2] << " -> " << box_lo[2] + << " khi: " << hi_old[2] << " -> " << box_hi[2] << "\n"; + } + bx = Box(IntVect(box_lo[0],box_lo[1],box_lo[2]), + IntVect(box_hi[0],box_hi[1],box_hi[2])); } bool using_pbl = (solverChoice.turbChoice[lev_for_box].pbl_type == PBLType::MYJ || diff --git a/Source/Particles/ERFPC.H b/Source/Particles/ERFPC.H index 2a5603e3ee..318d530f12 100644 --- a/Source/Particles/ERFPC.H +++ b/Source/Particles/ERFPC.H @@ -67,10 +67,32 @@ struct ERFParticlesAssignor p.idata(ERFParticlesIntIdxAoS::k) ) ); iv[0] += domain.smallEnd()[0]; iv[1] += domain.smallEnd()[1]; + iv[2] = amrex::max(domain.smallEnd()[2], + amrex::min(iv[2], domain.bigEnd()[2])); return iv; } }; +/** Particle binner using ERFParticlesAssignor (k-index for z) for DenseBins. */ +struct GetParticleBinERF +{ + amrex::GpuArray plo; + amrex::GpuArray dxi; + amrex::Box domain; + amrex::IntVect bin_size; + amrex::Box box; + + template + AMREX_GPU_HOST_DEVICE + unsigned int operator() (const P& p) const noexcept + { + amrex::Box tbx; + auto iv = ERFParticlesAssignor()(p, plo, dxi, domain); + auto tid = amrex::getTileIndex(iv, box, true, bin_size, tbx); + return static_cast(tid); + } +}; + template AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE void update_location_idata ( P& a_p, @@ -79,20 +101,48 @@ void update_location_idata ( P& a_p, const amrex::Array4& a_height_arr ) { if (a_height_arr) { + // Compute horizontal indices from particle position + int ix = int(amrex::Math::floor((a_p.pos(0)-a_plo[0])*a_dxi[0])); + int iy = int(amrex::Math::floor((a_p.pos(1)-a_plo[1])*a_dxi[1])); + + // Check if horizontal indices are within the valid array bounds. + // We need ix, ix+1, iy, iy+1 to be valid for bilinear interpolation. + // If particle has moved outside the local tile region (after advection + // but before redistribution), skip the k-index update. The particle + // will be redistributed to the correct tile and can update then. + if (ix < a_height_arr.begin[0] || ix+1 >= a_height_arr.end[0] || + iy < a_height_arr.begin[1] || iy+1 >= a_height_arr.end[1]) { + return; + } + + amrex::Real lx = (a_p.pos(0)-a_plo[0])*a_dxi[0] - static_cast(ix); + amrex::Real ly = (a_p.pos(1)-a_plo[1])*a_dxi[1] - static_cast(iy); + + // Clamp k to the valid range of the height array so the iteration + // can proceed even with partial-z AMR tiles. + int klo = a_height_arr.begin[2]; + int khi = a_height_arr.end[2] - 2; // need iz+1 < end[2] + if (khi < klo) { return; } + int kcur = a_p.idata(ERFParticlesIntIdxAoS::k); + if (kcur < klo || kcur > khi) { + a_p.idata(ERFParticlesIntIdxAoS::k) = amrex::max(klo, amrex::min(kcur, khi)); + } + while (1) { - amrex::IntVect iv( int(amrex::Math::floor((a_p.pos(0)-a_plo[0])*a_dxi[0])), - int(amrex::Math::floor((a_p.pos(1)-a_plo[1])*a_dxi[1])), - a_p.idata(ERFParticlesIntIdxAoS::k) ); - amrex::Real lx = (a_p.pos(0)-a_plo[0])*a_dxi[0] - static_cast(iv[0]); - amrex::Real ly = (a_p.pos(1)-a_plo[1])*a_dxi[1] - static_cast(iv[1]); - auto zlo = a_height_arr(iv[0] ,iv[1] ,iv[2] ) * (one-lx) * (one-ly) + - a_height_arr(iv[0]+1,iv[1] ,iv[2] ) * lx * (one-ly) + - a_height_arr(iv[0] ,iv[1]+1,iv[2] ) * (one-lx) * ly + - a_height_arr(iv[0]+1,iv[1]+1,iv[2] ) * lx * ly; - auto zhi = a_height_arr(iv[0] ,iv[1] ,iv[2]+1) * (one-lx) * (one-ly) + - a_height_arr(iv[0]+1,iv[1] ,iv[2]+1) * lx * (one-ly) + - a_height_arr(iv[0] ,iv[1]+1,iv[2]+1) * (one-lx) * ly + - a_height_arr(iv[0]+1,iv[1]+1,iv[2]+1) * lx * ly; + int iz = a_p.idata(ERFParticlesIntIdxAoS::k); + + if (iz < klo || iz > khi) { + break; + } + + auto zlo = a_height_arr(ix ,iy ,iz ) * (one-lx) * (one-ly) + + a_height_arr(ix+1,iy ,iz ) * lx * (one-ly) + + a_height_arr(ix ,iy+1,iz ) * (one-lx) * ly + + a_height_arr(ix+1,iy+1,iz ) * lx * ly; + auto zhi = a_height_arr(ix ,iy ,iz+1) * (one-lx) * (one-ly) + + a_height_arr(ix+1,iy ,iz+1) * lx * (one-ly) + + a_height_arr(ix ,iy+1,iz+1) * (one-lx) * ly + + a_height_arr(ix+1,iy+1,iz+1) * lx * ly; if (a_p.pos(2) > zhi) { a_p.idata(ERFParticlesIntIdxAoS::k) += 1; @@ -105,6 +155,72 @@ void update_location_idata ( P& a_p, } } +/*! \brief Compute vertical cell index k from particle z-position. + * + * Handles both uniform and non-uniform (stretched) vertical grids. + * For non-uniform grids, uses the level-0 z-levels array with the + * cumulative refinement ratio to compute fine-level cell boundaries. + * Starts from an initial guess and iterates (like update_location_idata). + * + * \param[in] z Particle z-position (physical coordinates) + * \param[in] plo_z Domain low z from Geometry + * \param[in] dxi_z Inverse cell size in z from Geometry + * \param[in] k_max Maximum valid k index (domain.bigEnd(2)) + * \param[in] zlevels Level-0 z-level node positions (nullptr if uniform) + * \param[in] nz_lev0 Number of level-0 cells (nz_levels - 1) + * \param[in] ref_ratio Cumulative refinement ratio from level 0 to target level + * \return Cell index k in [0, k_max] + */ +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +int compute_k_from_z (amrex::Real z, + amrex::Real plo_z, + amrex::Real dxi_z, + int k_max, + const amrex::Real* zlevels, + int nz_lev0, + int ref_ratio) +{ + if (zlevels == nullptr || nz_lev0 <= 0) { + // Uniform grid: direct computation + int k = static_cast(amrex::Math::floor((z - plo_z) * dxi_z)); + return amrex::max(0, amrex::min(k, k_max)); + } + + // Non-uniform z-levels: initial guess from uniform formula, then iterate. + // Fine-level cell boundary positions are linearly interpolated from + // the level-0 z-levels: for fine cell k_f, the coarse cell is k_f/R + // and the sub-cell position is (k_f%R)/R within that coarse cell. + int k = static_cast(amrex::Math::floor((z - plo_z) * dxi_z)); + k = amrex::max(0, amrex::min(k, k_max)); + + // Helper: compute z-position of fine cell boundary k_f + // fine_z(k_f) = zlevels[k_f/R] + (k_f%R) * (zlevels[k_f/R+1] - zlevels[k_f/R]) / R + auto fine_z = [=] AMREX_GPU_HOST_DEVICE (int k_f) -> amrex::Real { + int kc = k_f / ref_ratio; + int sub = k_f % ref_ratio; + // kc is clamped to valid range for safety + kc = amrex::max(0, amrex::min(kc, nz_lev0)); + if (kc >= nz_lev0) { return zlevels[nz_lev0]; } + return zlevels[kc] + + static_cast(sub) + * (zlevels[kc+1] - zlevels[kc]) + / static_cast(ref_ratio); + }; + + for (int iter = 0; iter <= k_max; iter++) { + amrex::Real z_lo = fine_z(k); + amrex::Real z_hi = fine_z(k + 1); + if (z >= z_hi && k < k_max) { + k++; + } else if (z < z_lo && k > 0) { + k--; + } else { + break; + } + } + return k; +} + class ERFPC : public amrex::ParticleContainer< ERFParticlesRealIdxAoS::ncomps, // AoS real attributes ERFParticlesIntIdxAoS::ncomps, // AoS integer attributes ERFParticlesRealIdxSoA::ncomps, // SoA real attributes @@ -221,6 +337,16 @@ class ERFPC : public amrex::ParticleContainer< ERFParticlesRealIdxAoS::ncomps, void initializeParticlesUniformDistributionInBox (const std::unique_ptr& a_ptr, const amrex::RealBox& particle_box); + /*! Fix k-indices for all particles after AMR regrid/removal */ + void FixKIndexAMR (const amrex::Vector>& a_z_phys_nd); + + /*! Extract and route out-of-range particles to proper levels */ + void ExtractAndRouteOORParticles ( int a_lev, + const amrex::Vector>& a_z_phys_nd ); + + /*! Diagnostic: count particles per level and halo (for AMR) */ + void CountParticlesPerLevelAndHalo ( int a_finest_level ); + protected: bool m_advect_w_flow; /*!< advect with flow velocity */ @@ -239,6 +365,8 @@ class ERFPC : public amrex::ParticleContainer< ERFParticlesRealIdxAoS::ncomps, bool m_stable_redistribute; /*!< use stable redistribute for deterministic simulations */ + amrex::Gpu::DeviceVector m_zlevels_d; /*!< Level-0 z-coordinate node positions for non-uniform vertical grids */ + /*! read inputs from file */ virtual void readInputs (); diff --git a/Source/Particles/ERFPCEvolve.cpp b/Source/Particles/ERFPCEvolve.cpp index df640d7882..8d1e81a886 100644 --- a/Source/Particles/ERFPCEvolve.cpp +++ b/Source/Particles/ERFPCEvolve.cpp @@ -17,6 +17,21 @@ void ERFPC::EvolveParticles ( int a_lev, { BL_PROFILE("ERFPCPC::EvolveParticles()"); + if (m_verbose > 0) { + Long np_total = 0; + int finest = m_gdb->finestLevel(); + amrex::Print() << "[" << m_name << "] Evolving particles on level " << a_lev + << ": "; + for (int lev = 0; lev <= finest; lev++) { + Long np_lev = NumberOfParticlesAtLevel(lev, true, true); + ParallelDescriptor::ReduceLongSum(np_lev); + amrex::Print() << "L" << lev << "=" << np_lev; + if (lev < finest) { amrex::Print() << " "; } + np_total += np_lev; + } + amrex::Print() << " (total=" << np_total << ")\n"; + } + if (m_advect_w_flow) { MultiFab* flow_vel( &a_flow_vars[a_lev][Vars::xvel] ); AdvectWithFlow( flow_vel, a_lev, a_dt_lev, a_z_phys_nd[a_lev] ); @@ -26,9 +41,51 @@ void ERFPC::EvolveParticles ( int a_lev, AdvectWithGravity( a_lev, a_dt_lev, a_z_phys_nd[a_lev] ); } + // Redistribute particles. For fine levels, use + // ExtractAndRouteOORParticles which handles partial-z refinement + // (particles escaping the fine level's z-extent) by recomputing + // k-indices for the target level before per-level Redistribute. + if (a_lev == 0) { + Redistribute(0, 0); + } else { + ExtractAndRouteOORParticles(a_lev, a_z_phys_nd); + } + + // After redistribution, recompute k-indices from z-position for all + // particles using each level's geometry. ERFParticlesAssignor uses + // idata(k) for grid assignment, so it must stay in sync with pos(z). + // z_phys_nd is always allocated (even for flat terrain). + for (int lev = 0; lev <= a_lev; lev++) { + const auto& plev = GetParticles(); + if (lev >= static_cast(plev.size())) { continue; } + if (plev[lev].empty()) { continue; } + + const Geometry& glev = m_gdb->Geom(lev); + const auto plo_lev = glev.ProbLoArray(); + const auto dxi_lev = glev.InvCellSizeArray(); + const int k_max_lev = glev.Domain().bigEnd(AMREX_SPACEDIM-1); + + for (ParIterType pti(*this, lev); pti.isValid(); ++pti) { + auto& aos = ParticlesAt(lev, pti).GetArrayOfStructs(); + const int np = aos.numParticles(); + auto* p_pbox = aos().data(); + auto zheight = (*a_z_phys_nd[lev])[pti.index()].array(); + + ParallelFor(np, [=] AMREX_GPU_DEVICE (int i) { + ParticleType& p = p_pbox[i]; + if (p.id() > 0) { + int k_guess = int(amrex::Math::floor( + (Real(p.pos(AMREX_SPACEDIM-1)) - plo_lev[AMREX_SPACEDIM-1]) + * dxi_lev[AMREX_SPACEDIM-1])); + p.idata(ERFParticlesIntIdxAoS::k) = amrex::max(0, amrex::min(k_guess, k_max_lev)); + update_location_idata(p, plo_lev, dxi_lev, zheight); + } + }); + } + } + ComputeTemperature( a_flow_vars[a_lev][Vars::cons], a_lev, a_dt_lev, a_z_phys_nd[a_lev] ); - Redistribute(); return; } @@ -118,8 +175,27 @@ void ERFPC::AdvectWithFlow ( MultiFab* a_umac, if (p.id() <= 0) { return; } ParticleReal v[AMREX_SPACEDIM]; + + // With partial-z refinement the particle's k may be outside + // the local grid's z-extent. The terrain interpolation + // stencil accesses k+2 for cell-centered components, so we + // need that to be within bounds. Zero velocity for + // out-of-bounds particles; Redistribute will move them to + // the correct level after advection. if (use_terrain) { - mac_interpolate_mapped_z(p, plo, dxi, umacarr, zheight, v); + int pk = p.idata(ERFParticlesIntIdxAoS::k); + // With partial-z refinement the particle's k may be + // outside the local grid's z-extent. The stencil + // accesses k-1 through k+2 in height_arr. + if (pk - 1 < zheight.begin[2] || pk + 2 >= zheight.end[2]) { + v[0] = 0; v[1] = 0; v[2] = 0; + } else { + mac_interpolate_mapped_z(p, plo, dxi, umacarr, zheight, v); + // Guard against NaN from edge-case stencil issues + if (amrex::isnan(v[0]) || amrex::isnan(v[1]) || amrex::isnan(v[2])) { + v[0] = 0; v[1] = 0; v[2] = 0; + } + } } else { mac_interpolate(p, plo, dxi, umacarr, v); } @@ -141,7 +217,7 @@ void ERFPC::AdvectWithFlow ( MultiFab* a_umac, // Update z-coordinate carried by the particle update_location_idata(p,plo,dxi,zheight); - // If the particle crossed below the bottom surface, move it up to Real(0.2)*dz above the surface + // If the particle crossed below the bottom surface, move it up to 0.2*dz above the surface if (!periodic_in_z) { if (p.idata(ERFParticlesIntIdxAoS::k) < 0) { int ii = domlo.x + int(amrex::Math::floor((p.pos(0)-plo[0])*dxi[0])); @@ -233,13 +309,13 @@ void ERFPC::AdvectWithGravity ( int a_lev, ParticleReal myhalf_dt = myhalf * a_dt; - // Update the particle velocity over first myhalf of step (a_dt/2) + // Update the particle velocity over first half of step (a_dt/2) vz_ptr[i] -= (grav - drag) * myhalf_dt; // Update the particle position over (a_dt) p.pos(2) += static_cast(ParticleReal(0.5)*a_dt*vz_ptr[i]); - // Update the particle velocity over second myhalf of step (a_dt/2) + // Update the particle velocity over second half of step (a_dt/2) vz_ptr[i] -= (grav - drag) * myhalf_dt; // also update z-coordinate here @@ -315,9 +391,12 @@ void ERFPC::ComputeTemperature (const MultiFab& a_ucons, ParticleType& p = p_pbox[i]; if (p.id() <= 0) { return; } - ParticleReal temperature; + ParticleReal temperature = 0; if (use_terrain) { - cic_interpolate_mapped_z( p, plo, dxi, temperature_arr, zheight, &temperature, 1 ); + int pk = p.idata(ERFParticlesIntIdxAoS::k); + if (pk >= zheight.begin[2] && pk + 2 < zheight.end[2]) { + cic_interpolate_mapped_z( p, plo, dxi, temperature_arr, zheight, &temperature, 1 ); + } } else { cic_interpolate( p, plo, dxi, temperature_arr, &temperature, 1 ); } diff --git a/Source/Particles/ERFPCInitializations.cpp b/Source/Particles/ERFPCInitializations.cpp index 4a6f414be3..7decfcac1a 100644 --- a/Source/Particles/ERFPCInitializations.cpp +++ b/Source/Particles/ERFPCInitializations.cpp @@ -60,6 +60,12 @@ void ERFPC::readInputs () } setStableRedistribute(m_stable_redistribute); + { + int verbose = 0; + pp.query("verbose", verbose); + SetVerbose(verbose); + } + return; } diff --git a/Source/Particles/ERFPCUtils.cpp b/Source/Particles/ERFPCUtils.cpp index 8f9841fb02..76ef351d9e 100644 --- a/Source/Particles/ERFPCUtils.cpp +++ b/Source/Particles/ERFPCUtils.cpp @@ -1,11 +1,21 @@ -#include -#include #include +#include +#include +#include #ifdef ERF_USE_PARTICLES using namespace amrex; +namespace { + struct ERFPCLevelGeom { + GpuArray plo; + GpuArray dxi; + int k_max; + int ref_ratio; // cumulative vertical refinement ratio + }; +} + void ERFPC::massDensity ( MultiFab& a_mf, const int& a_lev, const int& a_comp ) const @@ -39,4 +49,462 @@ void ERFPC::massDensity ( MultiFab& a_mf, return; } +/*! \brief Fix k-indices for all particles after AMR regrid */ +void ERFPC::FixKIndexAMR (const Vector>& a_z_phys_nd) +{ + BL_PROFILE("ERFPC::FixKIndexAMR()"); + + const int finest = finestLevel(); + + // Z-levels for non-uniform vertical grids (level-0 cell interfaces) + const Real* zlevels = m_zlevels_d.empty() ? nullptr : m_zlevels_d.data(); + const int nz_lev0 = m_zlevels_d.empty() ? 0 + : static_cast(m_zlevels_d.size()) - 1; + + // Helper lambda: recompute k-indices for all particles on a given level. + // Uses compute_k_from_z for the initial guess, then refines with the + // terrain height array (update_location_idata) per tile. + // z_phys_nd is always allocated (even for flat terrain). + auto recompute_k_for_level = [&](int lev, int ref_ratio) + { + const auto& particles = GetParticles(); + if (lev >= static_cast(particles.size())) { return; } + if (particles[lev].empty()) { return; } + + const Geometry& geom_lev = m_gdb->Geom(lev); + const auto plo = geom_lev.ProbLoArray(); + const auto dxi = geom_lev.InvCellSizeArray(); + const int k_max = geom_lev.Domain().bigEnd(AMREX_SPACEDIM-1); + + for (ParIterType pti(*this, lev); pti.isValid(); ++pti) { + int grid = pti.index(); + auto& ptile = ParticlesAt(lev, pti); + auto& aos = ptile.GetArrayOfStructs(); + auto* p_pbox = aos().data(); + const int np = aos.numParticles(); + + auto zheight = (*a_z_phys_nd[lev])[grid].array(); + + ParallelFor(np, [=] AMREX_GPU_DEVICE (int i) + { + auto& p = p_pbox[i]; + if (p.id() <= 0) { return; } + p.idata(ERFParticlesIntIdxAoS::k) = + compute_k_from_z(Real(p.pos(AMREX_SPACEDIM-1)), + plo[AMREX_SPACEDIM-1], + dxi[AMREX_SPACEDIM-1], + k_max, + zlevels, nz_lev0, ref_ratio); + update_location_idata(p, plo, dxi, zheight); + }); + } + Gpu::synchronize(); + }; + + // No fine levels: recompute k-indices for all particles using L0 geometry + // and move them to level 0 to avoid orphaned particles with invalid k values. + if (finest < 1) { + + // Recompute k using level-0 geometry for all particles + const auto& particles = GetParticles(); + for (int lev = 0; lev < static_cast(particles.size()); lev++) { + recompute_k_for_level(lev, 1); + } + + // Full Redistribute demotes all particles to level 0 + Redistribute(); + + return; + } + + // Cumulative refinement ratio to finest level + int finest_ref = 1; + for (int lev = 0; lev < finest; lev++) { + finest_ref *= m_gdb->refRatio(lev)[AMREX_SPACEDIM-1]; + } + + // Step 1: Set idata(0) = finest-level k for all particles + // Enables Redistribute to place particles correctly on fine grids. + // Use compute_k_from_z only (no terrain correction) because we need + // finest-level k regardless of which level's height array is available. + // Step 3 will correct using terrain after particles are on the right level. + { + const Geometry& geom_fine = m_gdb->Geom(finest); + const auto plo_fine = geom_fine.ProbLoArray(); + const auto dxi_fine = geom_fine.InvCellSizeArray(); + const int k_max_fine = geom_fine.Domain().bigEnd(AMREX_SPACEDIM-1); + + for (int lev = 0; lev <= finest; lev++) { + const auto& particles = GetParticles(); + if (lev >= static_cast(particles.size())) { continue; } + if (particles[lev].empty()) { continue; } + + for (ParIterType pti(*this, lev); pti.isValid(); ++pti) { + auto& ptile = ParticlesAt(lev, pti); + auto& aos = ptile.GetArrayOfStructs(); + auto* p_pbox = aos().data(); + const int np = aos.numParticles(); + + ParallelFor(np, [=] AMREX_GPU_DEVICE (int i) + { + auto& p = p_pbox[i]; + if (p.id() <= 0) { return; } + p.idata(ERFParticlesIntIdxAoS::k) = + compute_k_from_z(Real(p.pos(AMREX_SPACEDIM-1)), + plo_fine[AMREX_SPACEDIM-1], + dxi_fine[AMREX_SPACEDIM-1], + k_max_fine, + zlevels, nz_lev0, finest_ref); + }); + } + } + Gpu::synchronize(); + } + + // Step 2: Full multi-level Redistribute (k-clamping prevents crashes) + Redistribute(); + + // Step 3: Recompute idata(k) using each particle's level geometry. + // With terrain, the terrain-corrected k may differ from Step 1's flat k, + // potentially placing particles outside their current level's partial-z box. + // A full Redistribute is needed to move such particles to the correct level, + // followed by a second k recomputation for particles that changed level. + for (int lev = 0; lev <= finest; lev++) { + int lev_ref = 1; + for (int l = 0; l < lev; l++) { + lev_ref *= m_gdb->refRatio(l)[AMREX_SPACEDIM-1]; + } + recompute_k_for_level(lev, lev_ref); + } + + // Step 4: Full Redistribute to fix level assignment after terrain correction. + Redistribute(); + + // Step 5: Recompute k for particles that changed level in Step 4. + for (int lev = 0; lev <= finest; lev++) { + int lev_ref = 1; + for (int l = 0; l < lev; l++) { + lev_ref *= m_gdb->refRatio(l)[AMREX_SPACEDIM-1]; + } + recompute_k_for_level(lev, lev_ref); + } + + // Step 6: Per-level Redistribute to fix tile assignment after final k. + // L0 covers the full domain, so per-level is safe. For fine levels with + // partial-z boxes, per-level redistribute can fail, so use full. + Redistribute(0, 0); + if (finest > 0) { + Redistribute(); + } + + // Step 7: Final k recomputation. Steps 4-6 may have moved particles + // between levels (e.g. from L0 to L1 and back), leaving idata(k) set + // to the wrong level's geometry. Recompute once more on each + // particle's final level to guarantee a consistent k. + for (int lev = 0; lev <= finest; lev++) { + int lev_ref = 1; + for (int l = 0; l < lev; l++) { + lev_ref *= m_gdb->refRatio(l)[AMREX_SPACEDIM-1]; + } + recompute_k_for_level(lev, lev_ref); + } +} + +/*! \brief Diagnostic: count particles per level and, for level 1, in the halo. + * + * Prints particle counts for all levels. For AMR with >= 3 levels, also + * counts level-1 particles in halo cells (not covered by level 2). + * + * \param[in] a_finest_level Current finest AMR level (>= 0) + */ +void ERFPC::CountParticlesPerLevelAndHalo (int a_finest_level) +{ + BL_PROFILE("ERFPC::CountParticlesPerLevelAndHalo()"); + + // Per-level total counts + amrex::Vector n_per_lev(a_finest_level + 1, 0); + for (int lev = 0; lev <= a_finest_level; lev++) { + n_per_lev[lev] = NumberOfParticlesAtLevel(lev, true, true); + } + for (int lev = 0; lev <= a_finest_level; lev++) { + ParallelDescriptor::ReduceLongSum(n_per_lev[lev]); + } + + amrex::Print() << "[" << m_name << "] Particle counts per level (after Redistribute): "; + for (int lev = 0; lev <= a_finest_level; lev++) { + amrex::Print() << "L" << lev << "=" << n_per_lev[lev]; + if (lev < a_finest_level) { amrex::Print() << " "; } + } + amrex::Print() << "\n"; + + if (a_finest_level < 2) { return; } + + // Level-1 halo cells not covered by level 2 (1=halo, 0=covered) + const int lev1 = 1; + iMultiFab halo_mask = amrex::makeFineMask(ParticleBoxArray(lev1), ParticleDistributionMap(lev1), + IntVect(0), ParticleBoxArray(lev1 + 1), m_gdb->refRatio(lev1), + m_gdb->Geom(lev1).periodicity(), 1, 0); + + const amrex::Geometry& geom1 = m_gdb->Geom(lev1); + const auto plo = geom1.ProbLoArray(); + const auto dxi = geom1.InvCellSizeArray(); + const amrex::Box& domain1 = geom1.Domain(); + + Long n_halo = 0; + for (ParIterType pti(*this, lev1); pti.isValid(); ++pti) { + const auto& aos = ParticlesAt(lev1, pti).GetArrayOfStructs(); + const int np = aos.numParticles(); + if (np == 0) { continue; } + + const amrex::Box& box = pti.validbox(); + amrex::Array4 const& mask_arr = halo_mask.const_array(pti); + auto* p_pbox = aos().data(); + + ReduceOps reduce_op; + ReduceData reduce_data(reduce_op); + using ReduceTuple = typename decltype(reduce_data)::Type; + + reduce_op.eval(np, reduce_data, + [=] AMREX_GPU_DEVICE (int i) -> ReduceTuple + { + auto const& p = p_pbox[i]; + if (p.id() <= 0) { return {0L}; } + int ix = static_cast(amrex::Math::floor((p.pos(0) - plo[0]) * dxi[0])) + + domain1.smallEnd(0); + int iy = static_cast(amrex::Math::floor((p.pos(1) - plo[1]) * dxi[1])) + + domain1.smallEnd(1); + int iz = p.idata(ERFParticlesIntIdxAoS::k); + iz = amrex::max(domain1.smallEnd(2), amrex::min(iz, domain1.bigEnd(2))); + amrex::IntVect iv(AMREX_D_DECL(ix, iy, iz)); + if (box.contains(iv) && mask_arr(iv) == 1) { + return {1L}; + } + return {0L}; + }); + n_halo += amrex::get<0>(reduce_data.value(reduce_op)); + } + ParallelDescriptor::ReduceLongSum(n_halo); + + amrex::Print() << "[" << m_name << "] Level-1 halo particle count (cells not covered by L2): " << n_halo + << " (L1 total=" << n_per_lev[1] << ")\n"; +} + +/*! \brief Extract OOR particles from level a_lev, route to finest covering level, + * then redistribute. Avoids multi-level redistribute and excessive MPI traffic. + * + * \param[in] a_lev Fine level with potentially OOR particles + * \param[in] a_z_phys_nd Terrain height data (all levels) + */ +void ERFPC::ExtractAndRouteOORParticles ( int a_lev, + const Vector>& a_z_phys_nd ) +{ + BL_PROFILE("ERFPC::ExtractAndRouteOORParticles()"); + amrex::ignore_unused(a_z_phys_nd); + + AMREX_ALWAYS_ASSERT(a_lev > 0); + + ERFParticlesAssignor cell_assignor; + const int n_levels = a_lev + 1; + + // Z-levels data for k computation (level-0 cell interfaces) + const Real* zlevels = m_zlevels_d.empty() ? nullptr + : m_zlevels_d.data(); + const int nz_lev0 = m_zlevels_d.empty() ? 0 + : static_cast(m_zlevels_d.size()) - 1; + + // Build per-level ParticleLocators (GPU spatial indices) + Vector>> locators(n_levels); + for (int lev = 0; lev < n_levels; lev++) { + locators[lev].build(ParticleBoxArray(lev), m_gdb->Geom(lev)); + } + + // Collect the per-level AssignGrid objects into a device array + using AGType = AssignGrid>; + Gpu::PinnedVector h_assignors(n_levels); + for (int lev = 0; lev < n_levels; lev++) { + h_assignors[lev] = locators[lev].getGridAssignor(); + } + Gpu::DeviceVector d_assignors(n_levels); + Gpu::copyAsync( Gpu::hostToDevice, + h_assignors.begin(), + h_assignors.end(), + d_assignors.begin()); + + // Per-level geometry data for compute_k_from_z on the GPU + Gpu::PinnedVector h_lg(n_levels); + for (int lev = 0; lev < n_levels; lev++) { + const auto& geom = m_gdb->Geom(lev); + h_lg[lev].plo = geom.ProbLoArray(); + h_lg[lev].dxi = geom.InvCellSizeArray(); + h_lg[lev].k_max = geom.Domain().bigEnd(AMREX_SPACEDIM-1); + int ref = 1; + for (int l = 0; l < lev; l++) { + ref *= m_gdb->refRatio(l)[AMREX_SPACEDIM-1]; + } + h_lg[lev].ref_ratio = ref; + } + Gpu::DeviceVector d_lg(n_levels); + Gpu::copyAsync(Gpu::hostToDevice, h_lg.begin(), h_lg.end(), d_lg.begin()); + Gpu::synchronize(); + + auto* ag_ptr = d_assignors.data(); + auto* lg_ptr = d_lg.data(); + + // Find one local grid per level for receiving extracted particles + Gpu::PinnedVector dest_grids(n_levels, 0); + { + const int my_proc = ParallelDescriptor::MyProc(); + for (int lev = 0; lev < n_levels; lev++) { + const auto& dm = ParticleDistributionMap(lev); + for (int i = 0; i < static_cast(dm.size()); i++) { + if (dm[i] == my_proc) { dest_grids[lev] = i; break; } + } + } + } + + int finest = a_lev; + Vector levels_modified(n_levels, 0); + Vector n_routed_to(n_levels, 0); + + { + const int lev = a_lev; + auto src_lev_grid = locators[lev].getGridAssignor(); + + for (ParIterType pti(*this, lev); pti.isValid(); ++pti) { + auto& src_tile = ParticlesAt(lev, pti); + auto& aos = src_tile.GetArrayOfStructs(); + auto* p_pbox = aos().data(); + const int np = aos.numParticles(); + if (np == 0) { continue; } + + // Create OOR mask on GPU + Gpu::DeviceVector mask(np); + auto* mask_ptr = mask.data(); + + ParallelFor(np, [=] AMREX_GPU_DEVICE (int i) + { + auto& p = p_pbox[i]; + if (p.id() <= 0) { mask_ptr[i] = 0; return; } + int grd = src_lev_grid(p, 0, cell_assignor); + mask_ptr[i] = (grd < 0) ? 1 : 0; + }); + + // Count OOR particles + ReduceOps reduce_op; + ReduceData reduce_data(reduce_op); + using ReduceTuple = typename decltype(reduce_data)::Type; + reduce_op.eval(np, reduce_data, + [=] AMREX_GPU_DEVICE (int i) -> ReduceTuple + { return {mask_ptr[i]}; }); + int n_oor = amrex::get<0>(reduce_data.value(reduce_op)); + if (n_oor == 0) { continue; } + + // Extract OOR particles to a temporary tile + ParticleTileType tmp_tile; + tmp_tile.define(NumRuntimeRealComps(), NumRuntimeIntComps()); + tmp_tile.resize(n_oor); + + [[maybe_unused]] int n_copied = amrex::filterParticles(tmp_tile, src_tile, mask_ptr, + int(0), int(0), np); + AMREX_ASSERT(n_copied == n_oor); + + // Find the correct target level for each particle + Gpu::DeviceVector target_lev_vec(n_oor); + auto* tgt_ptr = target_lev_vec.data(); + auto* tmp_pbox = tmp_tile.GetArrayOfStructs()().data(); + + ParallelFor(n_oor, [=] AMREX_GPU_DEVICE (int i) + { + auto& p = tmp_pbox[i]; + int found_lev = -1; + + for (int tl = finest; tl >= 0; tl--) { + const auto& lg = lg_ptr[tl]; + int k = compute_k_from_z( Real(p.pos(AMREX_SPACEDIM-1)), + lg.plo[AMREX_SPACEDIM-1], + lg.dxi[AMREX_SPACEDIM-1], + lg.k_max, zlevels, nz_lev0, + lg.ref_ratio ); + p.idata(ERFParticlesIntIdxAoS::k) = k; + int grd = ag_ptr[tl](p, 0, cell_assignor); + if (grd >= 0) { found_lev = tl; break; } + } + tgt_ptr[i] = found_lev; + }); + Gpu::synchronize(); + + // Route particles to their target levels + for (int tl = 0; tl < n_levels; tl++) { + if (tl == lev) { continue; } + + Gpu::DeviceVector tl_mask(n_oor); + auto* tl_mask_ptr = tl_mask.data(); + int target = tl; + ParallelFor(n_oor, [=] AMREX_GPU_DEVICE (int i) + { + tl_mask_ptr[i] = (tgt_ptr[i] == target) ? 1 : 0; + }); + + ReduceOps tl_reduce_op; + ReduceData tl_reduce_data(tl_reduce_op); + using TLReduceTuple = typename decltype(tl_reduce_data)::Type; + tl_reduce_op.eval(n_oor, tl_reduce_data, + [=] AMREX_GPU_DEVICE (int i) -> TLReduceTuple + { return {tl_mask_ptr[i]}; }); + int n_to_lev = amrex::get<0>(tl_reduce_data.value(tl_reduce_op)); + if (n_to_lev == 0) { continue; } + + ParticleTileType tl_tile; + tl_tile.define(NumRuntimeRealComps(), NumRuntimeIntComps()); + tl_tile.resize(n_to_lev); + [[maybe_unused]] int nc = amrex::filterParticles(tl_tile, tmp_tile, tl_mask_ptr, + int(0), int(0), n_oor); + AMREX_ASSERT(nc == n_to_lev); + + // Recompute idata(k) for the actual target level + auto* tl_pbox = tl_tile.GetArrayOfStructs()().data(); + const auto tl_plo = h_lg[tl].plo; + const auto tl_dxi = h_lg[tl].dxi; + int tl_kmax = h_lg[tl].k_max; + int tl_ref = h_lg[tl].ref_ratio; + ParallelFor(n_to_lev, [=] AMREX_GPU_DEVICE (int i) + { + auto& p = tl_pbox[i]; + p.idata(ERFParticlesIntIdxAoS::k) = compute_k_from_z( Real(p.pos(AMREX_SPACEDIM-1)), + tl_plo[AMREX_SPACEDIM-1], + tl_dxi[AMREX_SPACEDIM-1], + tl_kmax, zlevels, nz_lev0, tl_ref); + }); + Gpu::synchronize(); + + auto& dst_tile = DefineAndReturnParticleTile(tl, dest_grids[tl], 0); + auto dst_start = static_cast(dst_tile.numParticles()); + dst_tile.resize(dst_start + n_to_lev); + amrex::copyParticles(dst_tile, tl_tile, int(0), dst_start, n_to_lev); + levels_modified[tl] = 1; + n_routed_to[tl] += n_to_lev; + } + + // Mark OOR particles as dead in the original tile + ParallelFor(np, [=] AMREX_GPU_DEVICE (int i) + { + if (mask_ptr[i]) { p_pbox[i].id() = -1; } + }); + Gpu::synchronize(); + } + } + + // Ensure all ranks agree on which levels were modified + ParallelDescriptor::ReduceIntMax(levels_modified.data(), n_levels); + + // Per-level redistribute: only for levels that need it. + // Level a_lev always needs it (particles were advected, some marked dead). + // Lower levels need it only if particles were routed there. + for (int lev = 0; lev <= a_lev; lev++) { + if (lev == a_lev || levels_modified[lev]) { + Redistribute(lev, lev); + } + } +} + #endif diff --git a/Source/Particles/ERF_ParticleData.H b/Source/Particles/ERF_ParticleData.H index df94dd5742..ed639db1d2 100644 --- a/Source/Particles/ERF_ParticleData.H +++ b/Source/Particles/ERF_ParticleData.H @@ -127,6 +127,17 @@ class ParticleData } } + /*! Redistribute with terrain-aware k-index fix and level-boundary split/merge */ + inline void Redistribute ( + const amrex::Vector>& a_z_phys_nd) + { + BL_PROFILE("ParticleData::Redistribute(z_phys_nd)"); + for (ParticlesNamesVector::size_type i = 0; i < m_namelist.size(); i++) { + auto* pc = m_particle_species[m_namelist[i]]; + pc->FixKIndexAMR(a_z_phys_nd); + } + } + /*! Get species of a given name */ inline bool HasSpecies ( const std::string& a_name ) { diff --git a/Source/Prob/ERF_InitCustomPertVels_ParticleTests.H b/Source/Prob/ERF_InitCustomPertVels_ParticleTests.H index ee9ce1eb30..511eeb16f6 100644 --- a/Source/Prob/ERF_InitCustomPertVels_ParticleTests.H +++ b/Source/Prob/ERF_InitCustomPertVels_ParticleTests.H @@ -1,15 +1,16 @@ ParmParse pp_for_pert_vels("prob"); Real U_0 = zero; pp_for_pert_vels.query("U_0",U_0); - const int klo = geomdata.Domain().smallEnd()[2]; - const int khi = geomdata.Domain().bigEnd()[2]; + const Real ztop = geomdata.ProbHi()[2]; // Set the x-velocity + // NOTE: z_nd(i,j,0) is the terrain surface height. This works when the + // refined patch abuts the terrain (klo=0) but will fail if L1 is elevated + // above the surface (k=0 would be out of bounds for the local z_nd data). ParallelFor(xbx, [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept { - Real ztop = z_nd(i,j,khi+1); - Real zht = z_nd(i,j,klo); - x_vel_pert(i, j, k) = U_0 * ztop / (ztop - zht); + Real zbot = z_nd(i,j,0); + x_vel_pert(i, j, k) = U_0 * ztop / (ztop - zbot); }); const auto dx = geomdata.CellSize(); diff --git a/Source/Prob/ERF_InitCustomPert_ParticleTests.H b/Source/Prob/ERF_InitCustomPert_ParticleTests.H index 045b537ffc..046c9d0c7b 100644 --- a/Source/Prob/ERF_InitCustomPert_ParticleTests.H +++ b/Source/Prob/ERF_InitCustomPert_ParticleTests.H @@ -8,10 +8,6 @@ // Overridden physical constants Real C_p = amrex::Real(1004.0); - const int khi = geomdata.Domain().bigEnd()[2]; - - AMREX_ALWAYS_ASSERT(bx.length()[2] == khi+1); - ParallelFor(bx, [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept { // Geometry (note we must include these here to get the data on device) diff --git a/Source/TimeIntegration/ERF_TimeStep.cpp b/Source/TimeIntegration/ERF_TimeStep.cpp index 0765630660..891f03dc4b 100644 --- a/Source/TimeIntegration/ERF_TimeStep.cpp +++ b/Source/TimeIntegration/ERF_TimeStep.cpp @@ -145,9 +145,7 @@ ERF::timeStep (int lev, Real time, int /*iteration*/) regrid(lev, time); #ifdef ERF_USE_PARTICLES - if (finest_level != old_finest) { - particleData.Redistribute(); - } + particleData.Redistribute(z_phys_nd); #endif // mark that we have regridded this level already diff --git a/Tests/CTestList.cmake b/Tests/CTestList.cmake index 69579ac33e..b9ff65b244 100644 --- a/Tests/CTestList.cmake +++ b/Tests/CTestList.cmake @@ -193,6 +193,12 @@ add_test_r(SuperCell_3D "" "erf_exec" "plt00010") if(ERF_ENABLE_PARTICLES) add_test_r(ParticleAdvect "" "erf_exec" "plt00010") add_test_r(ParticleWoA "" "erf_exec" "plt00010") + add_test_r(ParticleAdvect_AMR1_box "" "erf_exec" "plt00050") + add_test_sdm(ParticleAdvect_AMR1_pcount "" "erf_exec" "plt00050" 2e-8 3e-9) + # Skip AMR2_pcount for Debug/RelWithDebInfo builds with AMD GPUs (it freezes!) + if((CMAKE_BUILD_TYPE STREQUAL "Release") OR (NOT ERF_ENABLE_HIP)) + add_test_sdm(ParticleAdvect_AMR2_pcount "" "erf_exec" "plt00050" 1e-7 5e-9) + endif() endif( ) if(ERF_ENABLE_RRGMTP) add_test_r(Radiation "" "erf_exec" "plt00010") diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Header b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Header new file mode 100644 index 0000000000..36bbff13e9 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Header @@ -0,0 +1,58 @@ +HyperCLaw-V1.1 +8 +density +x_velocity +y_velocity +z_velocity +theta +pressure +tracer_particles_count +tracer_particles_mass_density +3 +0.10000000000000007 +1 +0 0 0 +10 1 2 +2 +((0,0,0) (127,3,31) (0,0,0)) ((0,0,0) (255,3,63) (0,0,0)) +50 100 +0.078125 0.25 0.0625 +0.0390625 0.25 0.03125 +0 +0 +0 4 0.10000000000000007 +50 +0 2.5 +0 1 +0 2 +2.5 5 +0 1 +0 2 +5 7.5 +0 1 +0 2 +7.5 10 +0 1 +0 2 +Level_0/Cell +1 6 0.10000000000000007 +100 +4.765625 4.921875 +0 0.5 +0 0.5 +4.765625 4.921875 +0.5 1 +0 0.5 +4.921875 5.078125 +0 0.5 +0 0.5 +4.921875 5.078125 +0.5 1 +0 0.5 +5.078125 5.234375 +0 0.5 +0 0.5 +5.078125 5.234375 +0.5 1 +0 0.5 +Level_1/Cell diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00000 new file mode 100644 index 0000000000..de552f0d3c Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00001 new file mode 100644 index 0000000000..cf30f628f5 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00002 new file mode 100644 index 0000000000..b8093b897e Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00003 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00003 new file mode 100644 index 0000000000..e4d8f2b471 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_D_00003 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_H new file mode 100644 index 0000000000..2893977ba8 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_0/Cell_H @@ -0,0 +1,28 @@ +1 +1 +8 +0 +(4 0 +((0,0,0) (31,3,31) (0,0,0)) +((32,0,0) (63,3,31) (0,0,0)) +((64,0,0) (95,3,31) (0,0,0)) +((96,0,0) (127,3,31) (0,0,0)) +) +4 +FabOnDisk: Cell_D_00000 0 +FabOnDisk: Cell_D_00001 0 +FabOnDisk: Cell_D_00002 0 +FabOnDisk: Cell_D_00003 0 + +4,8 +1.15973479768518506e+00,9.99906441406716340e+00,0.00000000000000000e+00,-2.73097528753337478e-04,3.00000000000018986e+02,9.99773898477564944e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125328669585004e+00,9.99974868860475041e+00,0.00000000000000000e+00,-6.37169072166706270e-05,2.99999999999997897e+02,9.99774719034096488e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125425512493607e+00,1.00001176663143276e+01,0.00000000000000000e+00,-2.22393390016540612e-05,2.99999999999997897e+02,9.99775886303562002e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125394677342109e+00,9.99996122659689490e+00,0.00000000000000000e+00,-4.02498539460611484e-05,2.99999999999998010e+02,9.99775514640469046e+04,0.00000000000000000e+00,0.00000000000000000e+00, + +4,8 +1.16143848008011674e+00,1.00010584661061799e+01,0.00000000000000000e+00,6.99379832755060303e-04,3.00440440728662338e+02,9.99998108138229290e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143837395848570e+00,1.00003009952707167e+01,0.00000000000000000e+00,1.89623307401239470e-05,3.00000000000005286e+02,9.99997815970241791e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143796966876023e+00,1.00002259061539061e+01,0.00000000000000000e+00,1.63778969640805680e-05,3.00000000000001421e+02,9.99997328639743500e+04,1.00000000000000000e+00,4.91070587881888595e-04, +1.16143783485438923e+00,1.00001857456511605e+01,0.00000000000000000e+00,-9.74756608753035271e-07,3.00000000000001137e+02,9.99997166134627041e+04,0.00000000000000000e+00,0.00000000000000000e+00, + diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00000 new file mode 100644 index 0000000000..53b25ec9aa Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00001 new file mode 100644 index 0000000000..fd1a68a5ce Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00002 new file mode 100644 index 0000000000..dbe4520c7e Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00003 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00003 new file mode 100644 index 0000000000..e780300c49 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_D_00003 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_H new file mode 100644 index 0000000000..5c563aea3c --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/Level_1/Cell_H @@ -0,0 +1,36 @@ +1 +1 +8 +0 +(6 0 +((122,0,0) (125,1,15) (0,0,0)) +((122,2,0) (125,3,15) (0,0,0)) +((126,0,0) (129,1,15) (0,0,0)) +((126,2,0) (129,3,15) (0,0,0)) +((130,0,0) (133,1,15) (0,0,0)) +((130,2,0) (133,3,15) (0,0,0)) +) +6 +FabOnDisk: Cell_D_00002 0 +FabOnDisk: Cell_D_00000 0 +FabOnDisk: Cell_D_00000 8283 +FabOnDisk: Cell_D_00003 0 +FabOnDisk: Cell_D_00001 0 +FabOnDisk: Cell_D_00001 8283 + +6,8 +1.16139467248878936e+00,9.99954719462549946e+00,0.00000000000000000e+00,-3.47014145399554736e-04,2.99999999999996817e+02,9.99945138647976128e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16139467248878936e+00,9.99954719462549946e+00,0.00000000000000000e+00,-3.47014145399554736e-04,2.99999999999996817e+02,9.99945138647976128e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16139468010035118e+00,9.99960366143046642e+00,0.00000000000000000e+00,-1.97632919655272123e-04,2.99999999999994941e+02,9.99945147822804865e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16139468010035118e+00,9.99960366143046642e+00,0.00000000000000000e+00,-1.97632919655272123e-04,2.99999999999994941e+02,9.99945147822804865e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16139469222368552e+00,9.99954410496330759e+00,0.00000000000000000e+00,-2.41292967116755189e-04,2.99999999999996135e+02,9.99945162436046667e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16139469222368552e+00,9.99954410496330759e+00,0.00000000000000000e+00,-2.41292967116755189e-04,2.99999999999996135e+02,9.99945162436046667e+04,0.00000000000000000e+00,0.00000000000000000e+00, + +6,8 +1.16143901432787966e+00,1.00004946770015728e+01,0.00000000000000000e+00,2.23750490021074760e-04,3.00000000000005514e+02,9.99998587871104683e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143901432787966e+00,1.00004946770015728e+01,0.00000000000000000e+00,2.23750490021074760e-04,3.00000000000005514e+02,9.99998587871104683e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143901910515734e+00,1.00005007682635156e+01,0.00000000000000000e+00,1.80642844535397923e-04,3.00000000000004093e+02,9.99998593629636016e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143901910515734e+00,1.00005007682635156e+01,0.00000000000000000e+00,1.80642844535397923e-04,3.00000000000004093e+02,9.99998593629636016e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143904663701214e+00,1.00005461073899937e+01,0.00000000000000000e+00,2.42611012525100909e-04,3.00000000000002558e+02,9.99998626816526230e+04,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143904663701214e+00,1.00005461073899937e+01,0.00000000000000000e+00,2.42611012525100909e-04,3.00000000000002558e+02,9.99998626816526230e+04,0.00000000000000000e+00,0.00000000000000000e+00, + diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Header b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Header new file mode 100644 index 0000000000..968474452f --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Header @@ -0,0 +1,26 @@ +Version_Two_Dot_Zero_double +3 +5 +xvel +yvel +zvel +mass +temperature +1 +int_comp0 +1 +64 +65 +1 +4 +6 +0 0 0 +1 0 0 +2 64 0 +3 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Level_0/DATA_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Level_0/DATA_00002 new file mode 100644 index 0000000000..4e1ddc4625 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Level_0/DATA_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Level_0/Particle_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Level_0/Particle_H new file mode 100644 index 0000000000..cd94c723cc --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_box/tracer_particles/Level_0/Particle_H @@ -0,0 +1,6 @@ +(4 0 +((0,0,0) (31,3,31) (0,0,0)) +((32,0,0) (63,3,31) (0,0,0)) +((64,0,0) (95,3,31) (0,0,0)) +((96,0,0) (127,3,31) (0,0,0)) +) diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Header b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Header new file mode 100644 index 0000000000..d504d983c1 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Header @@ -0,0 +1,66 @@ +HyperCLaw-V1.1 +16 +density +x_velocity +y_velocity +z_velocity +theta +pres_hse +dens_hse +pressure +pert_pres +pert_dens +dpdx +dpdy +pres_hse_x +pres_hse_y +tracer_particles_count +tracer_particles_mass_density +3 +0.050000000000000037 +1 +0 0 0 +10 1 2 +2 +((0,0,0) (127,3,31) (0,0,0)) ((0,0,0) (255,3,63) (0,0,0)) +50 100 +0.078125 0.25 0.0625 +0.0390625 0.25 0.03125 +0 +0 +0 4 0.050000000000000037 +50 +0 2.5 +0 1 +0 2 +2.5 5 +0 1 +0 2 +5 7.5 +0 1 +0 2 +7.5 10 +0 1 +0 2 +Level_0/Cell +1 6 0.050000000000000037 +100 +4.84375 5 +0 0.5 +0 1.125 +4.84375 5 +0.5 1 +0 1.125 +5 5.078125 +0 0.5 +0 1.125 +5 5.078125 +0.5 1 +0 1.125 +5.078125 5.234375 +0 0.5 +0 1.125 +5.078125 5.234375 +0.5 1 +0 1.125 +Level_1/Cell diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00000 new file mode 100644 index 0000000000..63582e718c Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00001 new file mode 100644 index 0000000000..d7e4c1e0f5 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00002 new file mode 100644 index 0000000000..ee4cf42143 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00003 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00003 new file mode 100644 index 0000000000..61e43446de Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_D_00003 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_H new file mode 100644 index 0000000000..0e526aac11 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_0/Cell_H @@ -0,0 +1,28 @@ +1 +1 +16 +0 +(4 0 +((0,0,0) (31,3,31) (0,0,0)) +((32,0,0) (63,3,31) (0,0,0)) +((64,0,0) (95,3,31) (0,0,0)) +((96,0,0) (127,3,31) (0,0,0)) +) +4 +FabOnDisk: Cell_D_00000 0 +FabOnDisk: Cell_D_00001 0 +FabOnDisk: Cell_D_00002 0 +FabOnDisk: Cell_D_00003 0 + +4,16 +1.15977864826455446e+00,9.99950946255856898e+00,0.00000000000000000e+00,-1.56743781732000950e-04,2.99999999999997954e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99774769498316891e+04,-1.21248070674482733e-01,-1.65858374737060110e-03,-3.84282087348401580e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125600907509341e+00,9.99859893952543999e+00,0.00000000000000000e+00,-5.41744465960278455e-05,2.99999999999997954e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99778000380330923e+04,2.17584697224083357e-01,1.80508421121494678e-06,-4.05254083592444692e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125692170780503e+00,9.99951812144883867e+00,0.00000000000000000e+00,-1.60011089737593424e-05,2.99999999999997613e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99779100398191949e+04,3.18834214034723118e-01,2.64505055103114728e-06,-1.54669058229774226e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125429926336721e+00,1.00001330173949725e+01,0.00000000000000000e+00,5.77882844076058824e-07,2.99999999999998181e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99775939504621347e+04,2.35554782120743766e-02,1.95428796700980456e-07,-2.07408081181347403e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + +4,16 +1.16143859171073238e+00,1.00001499766907891e+01,0.00000000000000000e+00,3.19038780611120777e-04,3.00428988923108307e+02,9.99996439468985627e+04,1.16143723201192506e+00,9.99998078448643937e+04,1.75007935176836327e-01,1.45195875589365642e-06,7.32466983702033780e-01,0.00000000000000000e+00,6.39997721260150895e+05,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144292665772597e+00,9.99960672225435232e+00,0.00000000000000000e+00,-1.13323960893324768e-06,3.00000000000001421e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000330379477862e+05,7.00458842271473259e-01,5.81137411992393993e-06,1.19729126198217273e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144144034428831e+00,1.00001837803255533e+01,0.00000000000000000e+00,2.96885754341421637e-05,3.00000000000001194e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000151219132953e+05,5.31501950448728167e-01,4.40962014969237259e-06,5.02265961840748804e-03,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143983290868902e+00,1.00002937807562322e+01,0.00000000000000000e+00,4.75902119897772440e-05,3.00000000000001023e+02,9.99996439468985627e+04,1.16143723201192506e+00,9.99999574588159740e+04,3.32688134862110019e-01,2.76015679379071344e-06,-1.31999995559453978e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00000 new file mode 100644 index 0000000000..0cbfe67a95 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00001 new file mode 100644 index 0000000000..01a9d89781 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00002 new file mode 100644 index 0000000000..e13b0e0e52 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00003 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00003 new file mode 100644 index 0000000000..5444626c06 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_D_00003 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_H new file mode 100644 index 0000000000..5129b33c56 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/Level_1/Cell_H @@ -0,0 +1,36 @@ +1 +1 +16 +0 +(6 0 +((124,0,0) (127,1,35) (0,0,0)) +((124,2,0) (127,3,35) (0,0,0)) +((128,0,0) (129,1,35) (0,0,0)) +((128,2,0) (129,3,35) (0,0,0)) +((130,0,0) (133,1,35) (0,0,0)) +((130,2,0) (133,3,35) (0,0,0)) +) +6 +FabOnDisk: Cell_D_00003 0 +FabOnDisk: Cell_D_00001 0 +FabOnDisk: Cell_D_00001 36956 +FabOnDisk: Cell_D_00000 0 +FabOnDisk: Cell_D_00002 0 +FabOnDisk: Cell_D_00000 18524 + +6,16 +1.16133961367402549e+00,9.99940811460301759e+00,0.00000000000000000e+00,-2.22261884351733084e-05,2.99999999999996419e+02,9.99873606536800507e+04,1.16133532790355476e+00,9.99878772422219772e+04,5.02091507398290560e-01,4.16535305980403336e-06,-1.10396386869251725e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133961367402549e+00,9.99940811460301759e+00,0.00000000000000000e+00,-2.22261884351733084e-05,2.99999999999996419e+02,9.99873606536800507e+04,1.16133532790355476e+00,9.99878772422219772e+04,5.02091507398290560e-01,4.16535305980403336e-06,-1.10396386869251725e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133958296914686e+00,9.99945921409307736e+00,0.00000000000000000e+00,-1.70447018550600674e-05,2.99999999999997783e+02,9.99873606536800507e+04,1.16133532790355476e+00,9.99878735411852540e+04,4.97873759741196409e-01,4.13036263990917973e-06,-5.63695829361677142e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133958296914686e+00,9.99945921409307736e+00,0.00000000000000000e+00,-1.70447018550600674e-05,2.99999999999997783e+02,9.99873606536800507e+04,1.16133532790355476e+00,9.99878735411852540e+04,4.97873759741196409e-01,4.13036263990917973e-06,-5.63695829361677142e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133952581486977e+00,9.99948389266326743e+00,0.00000000000000000e+00,-1.50690089840744089e-05,2.99999999999997726e+02,9.99873606536800507e+04,1.16133532790355476e+00,9.99878666520463448e+04,4.88917881724773906e-01,4.05606468834029954e-06,-1.36109380051493645e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133952581486977e+00,9.99948389266326743e+00,0.00000000000000000e+00,-1.50690089840744089e-05,2.99999999999997726e+02,9.99873606536800507e+04,1.16133532790355476e+00,9.99878666520463448e+04,4.88917881724773906e-01,4.05606468834029954e-06,-1.36109380051493645e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + +6,16 +1.16144145165625789e+00,9.99948803594938695e+00,0.00000000000000000e+00,1.85501691978815197e-06,3.00000000000001307e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000152582679206e+05,5.22192242962773889e-01,4.33226038820677672e-06,-4.01335202157497434e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144145165625789e+00,9.99948803594938695e+00,0.00000000000000000e+00,1.85501691978815197e-06,3.00000000000001307e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000152582679206e+05,5.22192242962773889e-01,4.33226038820677672e-06,-4.01335202157497434e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144138200526648e+00,9.99951441700581967e+00,0.00000000000000000e+00,-4.33327234139841541e-07,3.00000000000001080e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000144186944701e+05,5.15007694353698753e-01,4.27265529934217625e-06,-4.73732700571417864e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,1.00000000000000000e+00,2.94454921935148215e-03, +1.16144138200526648e+00,9.99951441700581967e+00,0.00000000000000000e+00,-4.33327234139841541e-07,3.00000000000001080e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000144186944701e+05,5.15007694353698753e-01,4.27265529934217625e-06,-4.73732700571417864e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,1.00000000000000000e+00,2.94454921935148172e-03, +1.16144134547074418e+00,9.99956129308787212e+00,0.00000000000000000e+00,3.21540393981469127e-06,3.00000000000001023e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000139783071034e+05,5.11173020757269114e-01,4.24084173711314349e-06,-4.02240477502346094e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144134547074418e+00,9.99956129308787212e+00,0.00000000000000000e+00,3.21540393981469127e-06,3.00000000000001023e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000139783071034e+05,5.11173020757269114e-01,4.24084173711314349e-06,-4.02240477502346094e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Header b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Header new file mode 100644 index 0000000000..2c08a6fd64 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Header @@ -0,0 +1,26 @@ +Version_Two_Dot_Zero_double +3 +5 +xvel +yvel +zvel +mass +temperature +1 +int_comp0 +1 +64 +65 +1 +4 +6 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +3 0 0 +1 0 0 +1 32 0 +0 32 0 +2 0 0 +0 0 2432 diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/DATA_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/DATA_00000 new file mode 100644 index 0000000000..d2f58e55ff Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/DATA_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/DATA_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/DATA_00001 new file mode 100644 index 0000000000..98621022b9 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/DATA_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/Particle_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/Particle_H new file mode 100644 index 0000000000..67a340dace --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR1_pcount/tracer_particles/Level_1/Particle_H @@ -0,0 +1,8 @@ +(6 0 +((124,0,0) (127,1,35) (0,0,0)) +((124,2,0) (127,3,35) (0,0,0)) +((128,0,0) (129,1,35) (0,0,0)) +((128,2,0) (129,3,35) (0,0,0)) +((130,0,0) (133,1,35) (0,0,0)) +((130,2,0) (133,3,35) (0,0,0)) +) diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Header b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Header new file mode 100644 index 0000000000..27a6faea62 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Header @@ -0,0 +1,79 @@ +HyperCLaw-V1.1 +16 +density +x_velocity +y_velocity +z_velocity +theta +pres_hse +dens_hse +pressure +pert_pres +pert_dens +dpdx +dpdy +pres_hse_x +pres_hse_y +tracer_particles_count +tracer_particles_mass_density +3 +0.050000000000000037 +2 +0 0 0 +10 1 2 +2 2 +((0,0,0) (127,3,31) (0,0,0)) ((0,0,0) (255,3,63) (0,0,0)) ((0,0,0) (511,3,127) (0,0,0)) +50 100 198 +0.078125 0.25 0.0625 +0.0390625 0.25 0.03125 +0.01953125 0.25 0.015625 +0 +0 +0 4 0.050000000000000037 +50 +0 2.5 +0 1 +0 2 +2.5 5 +0 1 +0 2 +5 7.5 +0 1 +0 2 +7.5 10 +0 1 +0 2 +Level_0/Cell +1 4 0.050000000000000037 +100 +4.765625 4.921875 +0 1 +0 1.1875 +4.921875 5.078125 +0 1 +0 1.1875 +5.078125 5.234375 +0 1 +0 1.1875 +5.234375 5.3125 +0 1 +0 1.1875 +Level_1/Cell +2 5 0.050000000000000037 +198 +4.8828125 4.9609375 +0 1 +0 1.09375 +4.9609375 5 +0 1 +0 1.09375 +5 5.078125 +0 1 +0 1.09375 +5.078125 5.15625 +0 1 +0 1.09375 +5.15625 5.234375 +0 1 +0 1.09375 +Level_2/Cell diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00000 new file mode 100644 index 0000000000..63582e718c Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00001 new file mode 100644 index 0000000000..d7e4c1e0f5 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00002 new file mode 100644 index 0000000000..ee4cf42143 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00003 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00003 new file mode 100644 index 0000000000..61e43446de Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_D_00003 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_H new file mode 100644 index 0000000000..0e526aac11 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_0/Cell_H @@ -0,0 +1,28 @@ +1 +1 +16 +0 +(4 0 +((0,0,0) (31,3,31) (0,0,0)) +((32,0,0) (63,3,31) (0,0,0)) +((64,0,0) (95,3,31) (0,0,0)) +((96,0,0) (127,3,31) (0,0,0)) +) +4 +FabOnDisk: Cell_D_00000 0 +FabOnDisk: Cell_D_00001 0 +FabOnDisk: Cell_D_00002 0 +FabOnDisk: Cell_D_00003 0 + +4,16 +1.15977864826455446e+00,9.99950946255856898e+00,0.00000000000000000e+00,-1.56743781732000950e-04,2.99999999999997954e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99774769498316891e+04,-1.21248070674482733e-01,-1.65858374737060110e-03,-3.84282087348401580e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125600907509341e+00,9.99859893952543999e+00,0.00000000000000000e+00,-5.41744465960278455e-05,2.99999999999997954e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99778000380330923e+04,2.17584697224083357e-01,1.80508421121494678e-06,-4.05254083592444692e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125692170780503e+00,9.99951812144883867e+00,0.00000000000000000e+00,-1.60011089737593424e-05,2.99999999999997613e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99779100398191949e+04,3.18834214034723118e-01,2.64505055103114728e-06,-1.54669058229774226e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16125429926336721e+00,1.00001330173949725e+01,0.00000000000000000e+00,5.77882844076058824e-07,2.99999999999998181e+02,9.99775703949839226e+04,1.16125410383457051e+00,9.99775939504621347e+04,2.35554782120743766e-02,1.95428796700980456e-07,-2.07408081181347403e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + +4,16 +1.16143859171073238e+00,1.00001499766907891e+01,0.00000000000000000e+00,3.19038780611120777e-04,3.00428988923108307e+02,9.99996439468985627e+04,1.16143723201192506e+00,9.99998078448643937e+04,1.75007935176836327e-01,1.45195875589365642e-06,7.32466983702033780e-01,0.00000000000000000e+00,6.39997721260150895e+05,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144292665772597e+00,9.99960672225435232e+00,0.00000000000000000e+00,-1.13323960893324768e-06,3.00000000000001421e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000330379477862e+05,7.00458842271473259e-01,5.81137411992393993e-06,1.19729126198217273e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144144034428831e+00,1.00001837803255533e+01,0.00000000000000000e+00,2.96885754341421637e-05,3.00000000000001194e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000151219132953e+05,5.31501950448728167e-01,4.40962014969237259e-06,5.02265961840748804e-03,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16143983290868902e+00,1.00002937807562322e+01,0.00000000000000000e+00,4.75902119897772440e-05,3.00000000000001023e+02,9.99996439468985627e+04,1.16143723201192506e+00,9.99999574588159740e+04,3.32688134862110019e-01,2.76015679379071344e-06,-1.31999995559453978e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00000 new file mode 100644 index 0000000000..ecbfcb9d5b Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00001 new file mode 100644 index 0000000000..4dddb64be8 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00002 new file mode 100644 index 0000000000..e6129698a9 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00003 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00003 new file mode 100644 index 0000000000..7f3f468f46 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_D_00003 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_H new file mode 100644 index 0000000000..7e04aa21e7 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_1/Cell_H @@ -0,0 +1,28 @@ +1 +1 +16 +0 +(4 0 +((122,0,0) (125,3,37) (0,0,0)) +((126,0,0) (129,3,37) (0,0,0)) +((130,0,0) (133,3,37) (0,0,0)) +((134,0,0) (135,3,37) (0,0,0)) +) +4 +FabOnDisk: Cell_D_00002 0 +FabOnDisk: Cell_D_00000 0 +FabOnDisk: Cell_D_00001 0 +FabOnDisk: Cell_D_00003 0 + +4,16 +1.16133374485770213e+00,9.99936775247437026e+00,0.00000000000000000e+00,-7.16577145167466136e-05,2.99999999999997669e+02,9.99866486115418084e+04,1.16132942058130739e+00,9.99871698403212940e+04,5.06644494307693094e-01,4.20313921756232389e-06,-2.37952863518148661e+00,0.00000000000000000e+00,-2.27873695157468337e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133372477096675e+00,9.99944204372820522e+00,0.00000000000000000e+00,-1.90714394291285837e-05,2.99999999999997783e+02,9.99866486117680906e+04,1.16132942058356115e+00,9.99871674191548809e+04,5.02710355067392811e-01,4.17050149570385997e-06,-4.17476849630475100e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133369092047967e+00,9.99953062282918381e+00,0.00000000000000000e+00,-1.46203391460407160e-05,2.99999999999998295e+02,9.99866486117680906e+04,1.16132942058356115e+00,9.99871633389671188e+04,4.96068196123815142e-01,4.11539795885929038e-06,-8.40809565037489021e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16133364711103759e+00,9.99956983061545301e+00,0.00000000000000000e+00,-1.16420147848248855e-05,2.99999999999998522e+02,9.99866486117680906e+04,1.16132942058356115e+00,9.99871580583703471e+04,4.89852688959217630e-01,4.06383394468967651e-06,-2.54826683178544067e-01,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + +4,16 +1.16144298950954394e+00,9.99948692011774831e+00,0.00000000000000000e+00,1.69837462205257839e-05,3.00000000000002160e+02,9.99998219732229045e+04,1.16143870891967493e+00,1.00000337955643554e+05,5.30189139244612306e-01,4.39861427392607141e-06,2.20227959416806707e+00,0.00000000000000000e+00,2.27874274682253608e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144142519047877e+00,9.99954402593827396e+00,0.00000000000000000e+00,1.05625231717073957e-05,3.00000000000001592e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000149392492021e+05,5.19918481979402713e-01,4.31340538797897466e-06,1.04272605851292617e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144140327183720e+00,9.99958738649674217e+00,0.00000000000000000e+00,2.65198899544584064e-06,3.00000000000002274e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000146750417800e+05,5.19579643572797067e-01,4.31059428551705537e-06,-3.03779914975166408e-03,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144132410383505e+00,9.99959420600609050e+00,0.00000000000000000e+00,1.25166293853450642e-06,3.00000000000001194e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000137207503227e+05,5.11921761455596425e-01,4.24706215063075376e-06,-6.75916386768221911e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00000 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00000 new file mode 100644 index 0000000000..e50632bddb Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00000 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00001 new file mode 100644 index 0000000000..fe24f16c6e Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00002 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00002 new file mode 100644 index 0000000000..792cf8b5cf Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00002 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00003 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00003 new file mode 100644 index 0000000000..f45d0da51c Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_D_00003 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_H new file mode 100644 index 0000000000..1a475c2720 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/Level_2/Cell_H @@ -0,0 +1,32 @@ +1 +1 +16 +0 +(5 0 +((250,0,0) (253,3,69) (0,0,0)) +((254,0,0) (255,3,69) (0,0,0)) +((256,0,0) (259,3,69) (0,0,0)) +((260,0,0) (263,3,69) (0,0,0)) +((264,0,0) (267,3,69) (0,0,0)) +) +5 +FabOnDisk: Cell_D_00000 0 +FabOnDisk: Cell_D_00002 0 +FabOnDisk: Cell_D_00001 0 +FabOnDisk: Cell_D_00003 0 +FabOnDisk: Cell_D_00003 143452 + +5,16 +1.16134185144096658e+00,9.99940547326832885e+00,0.00000000000000000e+00,-2.17919828878322964e-05,2.99999999999996248e+02,9.99876276703025069e+04,1.16133754315305948e+00,9.99881469734104030e+04,5.05974754487397149e-01,4.19757761260797224e-06,-7.66142006963491523e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16134184652653039e+00,9.99947312233219776e+00,0.00000000000000000e+00,-1.86293201066442155e-05,2.99999999999996874e+02,9.99876276703025069e+04,1.16133754315305948e+00,9.99881463810434070e+04,5.04778850183356553e-01,4.18765850462854416e-06,-3.67577500641345992e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16134183786243561e+00,9.99949149432009854e+00,0.00000000000000000e+00,-1.99644871712057174e-05,2.99999999999996589e+02,9.99876276703025069e+04,1.16133754315305948e+00,9.99881453367090726e+04,5.02762082745903172e-01,4.17092523519180247e-06,-3.91723360866308226e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16134181480119758e+00,9.99951608060166031e+00,0.00000000000000000e+00,-1.75095671227034854e-05,2.99999999999996646e+02,9.99876276703025069e+04,1.16133754315305948e+00,9.99881425570017309e+04,4.99296652444172651e-01,4.14217806987693393e-06,-5.70906613022089046e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16134179016101413e+00,9.99953596658102306e+00,0.00000000000000000e+00,-1.37566484818815476e-05,2.99999999999996305e+02,9.99876276703025069e+04,1.16133754315305948e+00,9.99881395869746484e+04,4.95314432933810167e-01,4.10913940140922307e-06,-7.79635775834321976e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + +5,16 +1.16144144539158711e+00,9.99949436168979489e+00,0.00000000000000000e+00,2.60507258922308484e-05,3.00000000000002217e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000151827535039e+05,5.19934976560762152e-01,4.31353007512846887e-06,8.48734751343727220e-04,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144142909190728e+00,9.99951203667782629e+00,0.00000000000000000e+00,1.14598681614761936e-05,3.00000000000001307e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000149862770078e+05,5.19336261597345583e-01,4.30856296285142548e-06,-1.61980032920837416e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144141752702290e+00,9.99954293363477831e+00,0.00000000000000000e+00,1.67873833347616083e-05,3.00000000000001592e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000148468738596e+05,5.18554237307398580e-01,4.30207507129409805e-06,-3.74096184968948425e-03,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,1.00000000000000000e+00,4.60597597802688691e-03, +1.16144139817788172e+00,9.99956312540805214e+00,0.00000000000000000e+00,4.37421780726980317e-06,3.00000000000001137e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000146136391195e+05,5.16990248215734027e-01,4.28909977845393087e-06,-3.25188592076301602e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, +1.16144136770354578e+00,9.99958693868727444e+00,0.00000000000000000e+00,4.80082121902810439e-06,3.00000000000002672e+02,9.99996439468985627e+04,1.16143723201192506e+00,1.00000142463014636e+05,5.14248954219510779e-01,4.26635722772239490e-06,-3.15306458622217206e-02,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00,0.00000000000000000e+00, + diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Header b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Header new file mode 100644 index 0000000000..0f4858d669 --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Header @@ -0,0 +1,30 @@ +Version_Two_Dot_Zero_double +3 +5 +xvel +yvel +zvel +mass +temperature +1 +int_comp0 +1 +64 +65 +2 +4 +4 +5 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +0 0 0 +2 0 0 +1 64 0 +3 0 0 +3 0 0 diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Level_2/DATA_00001 b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Level_2/DATA_00001 new file mode 100644 index 0000000000..11cacb88e0 Binary files /dev/null and b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Level_2/DATA_00001 differ diff --git a/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Level_2/Particle_H b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Level_2/Particle_H new file mode 100644 index 0000000000..071b2e72fa --- /dev/null +++ b/Tests/ERFGoldFiles/ParticleAdvect_AMR2_pcount/tracer_particles/Level_2/Particle_H @@ -0,0 +1,7 @@ +(5 0 +((250,0,0) (253,3,69) (0,0,0)) +((254,0,0) (255,3,69) (0,0,0)) +((256,0,0) (259,3,69) (0,0,0)) +((260,0,0) (263,3,69) (0,0,0)) +((264,0,0) (267,3,69) (0,0,0)) +) diff --git a/Tests/test_files/ParticleAdvect_AMR1_box/ParticleAdvect_AMR1_box.i b/Tests/test_files/ParticleAdvect_AMR1_box/ParticleAdvect_AMR1_box.i new file mode 100644 index 0000000000..c1758093ba --- /dev/null +++ b/Tests/test_files/ParticleAdvect_AMR1_box/ParticleAdvect_AMR1_box.i @@ -0,0 +1,88 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 50 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 128 4 32 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 4.55 -1.0 -1.0 +tracer_particles.particle_box_hi = 4.60 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 2E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 50 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = box1 +erf.box1.max_level = 1 +erf.box1.in_box_lo = 4.75 0.0 0.0 +erf.box1.in_box_hi = 5.25 1.0 0.5 + diff --git a/Tests/test_files/ParticleAdvect_AMR1_pcount/ParticleAdvect_AMR1_pcount.i b/Tests/test_files/ParticleAdvect_AMR1_pcount/ParticleAdvect_AMR1_pcount.i new file mode 100644 index 0000000000..d3e56ba9d3 --- /dev/null +++ b/Tests/test_files/ParticleAdvect_AMR1_pcount/ParticleAdvect_AMR1_pcount.i @@ -0,0 +1,90 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 50 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 128 4 32 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 4.55 -1.0 -1.0 +tracer_particles.particle_box_hi = 4.60 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 50 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 1 +amr.ref_ratio_vect = 2 1 2 +amr.n_error_buf = 2 +erf.regrid_int = 1 +erf.coupling_type = "OneWay" + +erf.refinement_indicators = tracer_count_refine +erf.tracer_count_refine.max_level = 1 +erf.tracer_count_refine.field_name = tracer_particles_count +erf.tracer_count_refine.value_greater = 1.0e-1 +erf.tracer_count_refine.start_time = 0.0 diff --git a/Tests/test_files/ParticleAdvect_AMR2_pcount/ParticleAdvect_AMR2_pcount.i b/Tests/test_files/ParticleAdvect_AMR2_pcount/ParticleAdvect_AMR2_pcount.i new file mode 100644 index 0000000000..294b23e37f --- /dev/null +++ b/Tests/test_files/ParticleAdvect_AMR2_pcount/ParticleAdvect_AMR2_pcount.i @@ -0,0 +1,90 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +erf.prob_name = "Particles Over Flat Ground" + +erf.init_type = Isentropic + +max_step = 50 + +amrex.fpe_trap_invalid = 1 + +fabarray.mfiter_tile_size = 1024 1024 1024 + +# PROBLEM SIZE & GEOMETRY +geometry.prob_lo = 0. 0. 0. +geometry.prob_hi = 10. 1. 2. + +amr.n_cell = 128 4 32 + +geometry.is_periodic = 0 1 0 + +xlo.type = "Inflow" +xhi.type = "Outflow" +xlo.velocity = 10. 0. 0. +xlo.density = 1.16 +xlo.theta = 300. +xlo.scalar = 0. +xlo.nonreflecting = true + +zlo.type = "SlipWall" +zhi.type = "SlipWall" + +# PARTICLES +erf.use_tracer_particles = 1 +tracer_particles.initial_distribution_type = box +tracer_particles.particle_box_lo = 4.55 -1.0 -1.0 +tracer_particles.particle_box_hi = 4.60 2.0 1.0 +tracer_particles.place_randomly_in_cells = false +tracer_particles.verbose = 1 + +# TIME STEP CONTROL +erf.fixed_dt = 1E-3 +erf.fixed_mri_dt_ratio = 12 + +# DIAGNOSTICS & VERBOSITY +erf.sum_interval = 1 # timesteps between computing mass +erf.v = 1 # verbosity in ERF.cpp +amr.v = 1 # verbosity in Amr.cpp + +# CHECKPOINT FILES +erf.check_file = chk # root name of checkpoint file +erf.check_int = -1 # number of timesteps between checkpoints + +# PLOTFILES +erf.plot_file_1 = plt # prefix of plotfile name +erf.plot_int_1 = 50 # number of timesteps between plotfiles +erf.plot_vars_1 = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density + +# SOLVER CHOICE +erf.use_gravity = true +erf.les_type = "None" + +erf.dycore_horiz_adv_type = Centered_2nd +erf.dycore_vert_adv_type = Centered_2nd +erf.dryscal_horiz_adv_type = Centered_2nd +erf.dryscal_vert_adv_type = Centered_2nd + +# Constant diffusion coefficient +erf.molec_diff_type = "ConstantAlpha" +erf.dynamic_viscosity = 0.0 # [kg/(m-s)] +erf.alpha_T = 0.0 # [m^2/s] + +#erf.abl_driver_type = "PressureGradient" +#erf.abl_pressure_grad = -0.2 0. 0. + +# PROBLEM PARAMETERS (optional) +prob.T_0 = 300.0 +prob.U_0 = 10.0 +prob.rho_0 = 1.16 + +# MULTILEVEL +amr.max_level = 2 +amr.ref_ratio_vect = 2 1 2 2 1 2 +amr.n_error_buf = 4 +erf.coupling_type = "OneWay" +erf.regrid_int = 1 + +erf.refinement_indicators = tracer_count_refine +erf.tracer_count_refine.max_level = 2 +erf.tracer_count_refine.field_name = tracer_particles_count +erf.tracer_count_refine.value_greater = 1.0e-1 +erf.tracer_count_refine.start_time = 0.0