Currently, the adapted version of the Agora restart problem type from Enzo has issues with particle initialization when using many particles or many initial refinement levels.
-
Currently (and as is done in Enzo), each processor stores the entire IC particle lists in memory. This can be a problem when running on many-core nodes with limited memory / core, or when running with very large numbers of particles. However, each processor only needs to store particles that live on blocks on that processor. This could be fixed (somewhat) if the Isolated Galaxy initialization routine can know the min / max coordinates of blocks on that processor, throwing out particles that do not reside there. This should lead to a reduction in memory.... The other fix is to just run with a small number of proc / node during initialization, and re-adjust after the first cycle(s)....
-
Possibly more serious issue. In Enzo, particle's are deposited to the grid on the root level first, then the gas ICs are applied iteratively as the hierarchy is refined to higher levels. This means the particle list is only looped over by blocks on the root level. In Enzo-E it looks like the natural behavior is to apply refinement first, then have each block handle ICs. With many blocks, this makes placing particle ICs take a long time, as each block has to check the full particle list. This currently places practical limitations on the number of blocks at initialization...
Currently, the adapted version of the Agora restart problem type from Enzo has issues with particle initialization when using many particles or many initial refinement levels.
Currently (and as is done in Enzo), each processor stores the entire IC particle lists in memory. This can be a problem when running on many-core nodes with limited memory / core, or when running with very large numbers of particles. However, each processor only needs to store particles that live on blocks on that processor. This could be fixed (somewhat) if the Isolated Galaxy initialization routine can know the min / max coordinates of blocks on that processor, throwing out particles that do not reside there. This should lead to a reduction in memory.... The other fix is to just run with a small number of proc / node during initialization, and re-adjust after the first cycle(s)....
Possibly more serious issue. In Enzo, particle's are deposited to the grid on the root level first, then the gas ICs are applied iteratively as the hierarchy is refined to higher levels. This means the particle list is only looped over by blocks on the root level. In Enzo-E it looks like the natural behavior is to apply refinement first, then have each block handle ICs. With many blocks, this makes placing particle ICs take a long time, as each block has to check the full particle list. This currently places practical limitations on the number of blocks at initialization...