Memory Requirements

[hwhitehead]

Are there any straightforward ways to predict the amount of memory a run will require? If a simulation is running without magnetic fields and with a known number of MeshBlocks/known MeshBlockSize, is there a way to calculate the required memory?

[c-white]

If your block size is Nx * Ny * Nz, then you will have Nc = (Nx + 2 * Ng) * (Ny + 2 * Ng) * (Nz + 2 * Ng) cells, where Ng is the number of ghost cells. Each cell will contain Nv = 5 hydro variables (4 for isothermal) plus Nu (default 0) user-output variables. There will generally be Nr = 4 registers (2 copies of the primitives, and 2 copies of the conserved variables, though this can change with choice of time integrator). Finally, if everything is double precision, there are 8 bytes per value. All other memory allocations (refinement buffers, 1D scratch arrays, etc.) are usually small compared to this, so you can estimate 8 * Nc * (Nr * Nv + Nu) bytes per block with just the basic physics.

[hwhitehead]

Thanks Chris, really apprecaite it.

[hwhitehead]

Apologies to bump this thread, but I am trying to pin down exactly what is taking up space in my system

It seems to be that more memory is being allocated than Nv = 5, Nr = 4, specifically due to keeping track of both cons and prim variables and flux in multiple dimensions. From looking at hydro.cpp it seems to me that there are 9 major arrays allocated: u, w, u1, w1, flux (in 3 directions), coarse_cons_ and coarse_prim_. Each of these arrays have dimensions NHYDRO * ncells3 * ncells2 * ncells1 (flux is technically slightly larger).

In my case there in an additional array of similar dimenion assosiated with using sts, specifically u2. This means that for me if we keep Nv=NHYDRO=5 it seems that Nr = 10. Does this sound correct? If there are any other large arrays being allocated that I am unaware of please let me know. Do the flux arrays get any registers allocated?

I am unsure of the functionality of coarse_cons_ and coarse_prim_ so please let me know if their size is different for some reason.

[hwhitehead]

To add to this, when look-up tables are implemented under the general equation of state system, is a copy of this table maintained in memory on each rank? For instance, if a pure MPI-parallelized simulation occupies 56 cores on a node, does Athena++ load 56 LUTs into memory? For suitably high-resolution LUTs, this may represent a significant increase in memory usage.

[tomidakn]

Sorry I did not notice this thread, and unfortunately Chris left the field.

coarse_cons_ and coarse_prim_ are used for mesh refinement, and they have only half cells in each direction, so they consume only ~1/8 memory. So, Nr ~ 8.25.

And yes, each MPI rank allocates a copy of look up tables. So if your table is big, you should consider MPI+OpenMP hybrid parallelization.

The memory issue is getting more serious these days. In particular, Intel Xeon Max has very fast HBM but it's only ~1GB/core.

[hwhitehead]

Many thanks for the response, I think this gives a clearer picture of the memory consumption.

In my case the extra usage was definitely due to the LUT usage, I will definitely consider hybridization going forward.