AMR support for particles with terrain-following coordinates

Docs/sphinx_doc/Particles.rst

@@ -10,10 +10,6 @@ Particles
 ERF has the option to include Lagrangian particles in addition to the mesh-based solution.
 Currently there is one example of particle types available in ERF: tracer_particles.
 
-The particle functionality is very simple and meant for demonstration.
-
-The particles are initialized one per mesh cell in a vertical plane at :math:`i = 3` for tracer particles.
-
 The tracer particles are advected by the velocity field with optional sedimentation.
 
 We note that unless the domain is periodic in the vertical direction, any particles that
@@ -30,6 +26,9 @@ different roles in future.
 
    Two-dimensional squall line simulation with particles. The particles and contours of cloud water mixing ratio are shown.
 
+Enabling Particles
+------------------
+
 To enable the use of particles, one must set
 
 ::
@@ -52,25 +51,111 @@ One must also set
 
 in the inputs file or on the command line at runtime.
 
-The time at which the particles are initialize can be controlled by a parameter in the inputs file.
-For tracer particles one would set this as
+Initialization
+--------------
+
+Particles can be initialized in a box-shaped subregion of the domain by specifying
+
+::
+
+   tracer_particles.initial_distribution_type = box
+   tracer_particles.particle_box_lo = 3.95 -1.0 -1.0
+   tracer_particles.particle_box_hi = 4.00  2.0  3.0
+   tracer_particles.place_randomly_in_cells = false
+
+One particle is placed per cell that falls within the specified box.
+If ``place_randomly_in_cells`` is true, the particle position within each cell
+is randomized; otherwise it is placed at the cell center.
+
+The time at which the particles are initialized can be controlled by
 
 ::
 
    tracer_particles.start_time = 0.5
 
+Verbosity of particle operations can be controlled per species:
+
+::
+
+   tracer_particles.verbose = 1
+
+Particles with AMR
+-------------------
+
+Tracer particles work with adaptive mesh refinement, including anisotropic
+refinement ratios (e.g., ``amr.ref_ratio_vect = 2 1 2``) and terrain-following
+coordinates. The particles are automatically redistributed across AMR levels
+as they move through the domain.
+
+With terrain-following (``StaticFittedMesh``) coordinates, the particle vertical
+index is mapped between the physical z-coordinate and the terrain-conforming
+mesh using the ``z_phys_nd`` field. This mapping is performed during particle
+evolution and redistribution so that particles follow the physical coordinate
+system rather than the index space.
+
+The AMR level coupling type is controlled by
+
+::
 
-Caveat: the particle information is currently output when using the AMReX-native plotfile format, but not
-when using netcdf.  Writing particles into the netcdf files is a WIP.
+   erf.coupling_type = "TwoWay"
 
-To see an example of using the particle functionality, build the executable using gmake in ``ERF/Exec`` setting.
-``USE\_PARTICLES = TRUE``.
+where ``TwoWay`` (default) enables refluxing and averaging down between coarse
+and fine levels, and ``OneWay`` disables the fine-to-coarse feedback.
+``OneWay`` is preferred for particle simulations as it avoids potential
+instabilities at coarse-fine boundaries.
+
+.. figure:: figures/ParticleAdvect_AMR2_box.gif
+   :alt: Tracer particles with 2 AMR levels using static box tagging
+   :align: center
+   :width: 100%
+
+   Tracer particle advection over flat terrain with 2 AMR levels and static box
+   tagging covering a partial z-extent (``inputs_over_flat_AMR2_box_partialz``).
+   The refined region is fixed in space; particles enter and exit it as they advect.
+
+.. figure:: figures/ParticleAdvect_AMR2_pcount.gif
+   :alt: Tracer particles with 2 AMR levels using particle-count tagging
+   :align: center
+   :width: 100%
+
+   Tracer particle advection over flat terrain with 2 AMR levels and dynamic
+   particle-count-based tagging (``inputs_over_flat_AMR2_particlecount``).
+   The refined region tracks the particles as they move through the domain.
+
+Particle-count-based refinement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Particles can drive dynamic AMR by tagging cells for refinement based on the
+deposited particle count. This is configured through the standard refinement
+indicator mechanism:
+
+::
+
+   erf.refinement_indicators = tracer_count_refine
+   erf.tracer_count_refine.field_name = tracer_particles_count
+   erf.tracer_count_refine.value_greater = 0.1
+   erf.tracer_count_refine.max_level = 1
+   erf.tracer_count_refine.start_time = 0.0
+
+This refines cells where the deposited tracer particle count exceeds the
+specified threshold. When using multiple AMR levels, the particle counts are
+averaged down level-by-level (finest to coarsest) to ensure correct tagging
+at all levels.
+
+Output
+------
+
+The particle information is output when using the AMReX-native plotfile format, but not
+when using netcdf.
 
 To visualize the number of particles per cell as a mesh-based variable, add
-``tracer_particle_count`` (if you have set ``erf.use_tracer_particles``) and
+``tracer_particles_count`` (if you have set ``erf.use_tracer_particles``)
 to the line in the inputs file that begins
 
 ::
 
    erf.plot_vars_1 =
 
+Example input files for particles with AMR can be found in
+``Exec/RegTests/ParticleTests/``.
+

Docs/sphinx_doc/RegressionTests.rst

@@ -106,6 +106,15 @@ The following problems are currently tested in the CI:
 | TaylorGreenAdvectingDiffusing | 16 16 16 | Periodic | Periodic | SlipWall   | None  |                                 |
 |                               |          |          |          | SlipWall   |       |                                 |
 +-------------------------------+----------+----------+----------+------------+-------+---------------------------------+
+| ParticleAdvect_AMR1_box       | 128 4 32 | Inflow   | Periodic | SlipWall   | None  | particle advection, 1 AMR level |
+|                               |          | Outflow  |          | SlipWall   |       | static box tagging, partial z   |
++-------------------------------+----------+----------+----------+------------+-------+---------------------------------+
+| ParticleAdvect_AMR1_pcount    | 128 4 32 | Inflow   | Periodic | SlipWall   | None  | particle advection, 1 AMR level |
+|                               |          | Outflow  |          | SlipWall   |       | particle-count tagging          |
++-------------------------------+----------+----------+----------+------------+-------+---------------------------------+
+| ParticleAdvect_AMR2_pcount    | 128 4 32 | Inflow   | Periodic | SlipWall   | None  | particle advection, 2 AMR levels|
+|                               |          | Outflow  |          | SlipWall   |       | particle-count tagging          |
++-------------------------------+----------+----------+----------+------------+-------+---------------------------------+
 
 while the following tests are run nightly:
 

Exec/RegTests/ParticleTests/README

@@ -4,3 +4,12 @@ The terrain-fitted coordinates follow the contour of the hill.
 
 This problem is primarily designed to test the implementation of the terrain-fitted coordinates.
 
+The AMR input files test tracer particle advection with adaptive mesh refinement.
+The naming convention is inputs_over_{terrain}_{AMRlevels}_{tagging}, where:
+  - terrain: "flat" (no terrain) or "hill" (Witch of Agnesi profile)
+  - AMRlevels: "AMR1" (1 refined level) or "AMR2" (2 refined levels)
+  - tagging: "box_fullz" (static box covering full z-extent),
+             "box_partialz" (static box covering partial z-extent),
+             "particlecount" (dynamic tagging based on deposited particle count)
+
+All cases use anisotropic refinement (2,1,2) per level.

Exec/RegTests/ParticleTests/inputs_over_flat

@@ -3,7 +3,7 @@ erf.prob_name = "Particles Over Flat Ground"
 
 erf.init_type = Isentropic
 
-max_step =  80
+max_step =  400
 
 amrex.fpe_trap_invalid = 1
 
@@ -23,16 +23,18 @@ xlo.velocity = 10. 0. 0.
 xlo.density  = 1.16
 xlo.theta    = 300.
 xlo.scalar   = 0.
+xlo.nonreflecting = true
 
 zlo.type = "SlipWall"
 zhi.type = "SlipWall"
 
 # PARTICLES
 erf.use_tracer_particles = 1
 tracer_particles.initial_distribution_type = box
-tracer_particles.particle_box_lo = 3.95 -1.0 -1.0
-tracer_particles.particle_box_hi = 4.00  2.0  3.0
-tracer_particles.place_randomly_in_cells = true
+tracer_particles.particle_box_lo = 2.95 -1.0 -1.0
+tracer_particles.particle_box_hi = 3.00  2.0  1.0
+tracer_particles.place_randomly_in_cells = false
+tracer_particles.verbose = 1
 
 # TIME STEP CONTROL
 erf.fixed_dt           = 1E-3
@@ -43,32 +45,19 @@ erf.sum_interval   = 1        # timesteps between computing mass
 erf.v              = 1        # verbosity in ERF.cpp
 amr.v              = 1        # verbosity in Amr.cpp
 
-# REFINEMENT / REGRIDDING
-amr.max_level       = 0       # maximum level number allowed
-
 # CHECKPOINT FILES
 erf.check_file      = chk     # root name of checkpoint file
-erf.check_int       = 100 # number of timesteps between checkpoints
+erf.check_int       = -1  # number of timesteps between checkpoints
 
 # PLOTFILES
 erf.plot_file_1     = plt     # prefix of plotfile name
-erf.plot_int_1      = 50      # number of timesteps between plotfiles
-erf.plot_vars_1     = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density
+erf.plot_int_1      = 20      # number of timesteps between plotfiles
+erf.plot_vars_1     = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density
 
 # SOLVER CHOICE
 erf.use_gravity = true
-erf.use_coriolis = false
 erf.les_type = "None"
 
-# MULTILEVEL
-amr.max_level = 0
-amr.ref_ratio_vect = 2 2 1
-
-erf.refinement_indicators = box1
-erf.box1.max_level = 1
-erf.box1.in_box_lo =  2. 0.25
-erf.box1.in_box_hi =  8. 0.75
-
 erf.dycore_horiz_adv_type  = Centered_2nd
 erf.dycore_vert_adv_type   = Centered_2nd
 erf.dryscal_horiz_adv_type = Centered_2nd
@@ -83,8 +72,9 @@ erf.alpha_T           = 0.0 # [m^2/s]
 #erf.abl_pressure_grad = -0.2 0. 0.
 
 # PROBLEM PARAMETERS (optional)
-prob.custom_terrain_type = "FlatRaised"
-
 prob.T_0   = 300.0
 prob.U_0   = 10.0
 prob.rho_0 = 1.16
+
+# MULTILEVEL
+amr.max_level = 1

Exec/RegTests/ParticleTests/inputs_over_flat_AMR1_box_fullz

@@ -0,0 +1,91 @@
+# ------------------  INPUTS TO MAIN PROGRAM  -------------------
+erf.prob_name = "Particles Over Flat Ground"
+
+erf.init_type = Isentropic
+
+max_step =  400
+
+amrex.fpe_trap_invalid = 1
+
+fabarray.mfiter_tile_size = 1024 1024 1024
+
+# PROBLEM SIZE & GEOMETRY
+geometry.prob_lo     = 0.   0.  0. 
+geometry.prob_hi     = 10.  1.  2.
+
+amr.n_cell           = 256  8   64
+
+geometry.is_periodic = 0 1 0
+
+xlo.type = "Inflow"
+xhi.type = "Outflow"
+xlo.velocity = 10. 0. 0.
+xlo.density  = 1.16
+xlo.theta    = 300.
+xlo.scalar   = 0.
+xlo.nonreflecting = true
+
+zlo.type = "SlipWall"
+zhi.type = "SlipWall"
+
+# PARTICLES
+erf.use_tracer_particles = 1
+tracer_particles.initial_distribution_type = box
+tracer_particles.particle_box_lo = 2.95 -1.0 -1.0
+tracer_particles.particle_box_hi = 3.00  2.0  1.0
+tracer_particles.place_randomly_in_cells = false
+tracer_particles.verbose = 1
+
+# TIME STEP CONTROL
+erf.fixed_dt           = 1E-3
+erf.fixed_mri_dt_ratio = 12
+
+# DIAGNOSTICS & VERBOSITY
+erf.sum_interval   = 1        # timesteps between computing mass
+erf.v              = 1        # verbosity in ERF.cpp
+amr.v              = 1        # verbosity in Amr.cpp
+
+# REFINEMENT / REGRIDDING
+amr.max_level       = 0       # maximum level number allowed
+
+# CHECKPOINT FILES
+erf.check_file      = chk     # root name of checkpoint file
+erf.check_int       = -1  # number of timesteps between checkpoints
+
+# PLOTFILES
+erf.plot_file_1     = plt     # prefix of plotfile name
+erf.plot_int_1      = 20      # number of timesteps between plotfiles
+erf.plot_vars_1     = density x_velocity y_velocity z_velocity pressure theta pres_hse dens_hse pert_pres pert_dens z_phys detJ dpdx dpdy pres_hse_x pres_hse_y tracer_particles_count tracer_particles_mass_density
+
+# SOLVER CHOICE
+erf.use_gravity = true
+erf.les_type = "None"
+
+erf.dycore_horiz_adv_type  = Centered_2nd
+erf.dycore_vert_adv_type   = Centered_2nd
+erf.dryscal_horiz_adv_type = Centered_2nd
+erf.dryscal_vert_adv_type  = Centered_2nd
+
+# Constant diffusion coefficient
+erf.molec_diff_type   = "ConstantAlpha"
+erf.dynamic_viscosity = 0.0 # [kg/(m-s)]
+erf.alpha_T           = 0.0 # [m^2/s]
+
+#erf.abl_driver_type = "PressureGradient"
+#erf.abl_pressure_grad = -0.2 0. 0.
+
+# PROBLEM PARAMETERS (optional)
+prob.T_0   = 300.0
+prob.U_0   = 10.0
+prob.rho_0 = 1.16
+
+# MULTILEVEL
+amr.max_level = 1
+amr.ref_ratio_vect = 2 1 2
+erf.coupling_type = "OneWay"
+
+erf.refinement_indicators = box1
+erf.box1.max_level = 1
+erf.box1.in_box_lo =  4.0 0.0
+erf.box1.in_box_hi =  6.0 1.0
+