Make using The_Comms_Arena() for particle communication opt-in

[atmyers]

For mesh data it is better to use a small, separate Arena for communication. But, we found that for particle communication this approach uses too much memory (see the ImpactX Issue here: BLAST-ImpactX/impactx#769)

This restores the old default behavior (prior to PR #4175), but allows users to opt-in to using The_Comms_Arena if desired.

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

[ax3l]

Thank you, @atmyers !

Quick question, is it required to use the comms arena for GPU-aware MPI/GPU-direct comms or is it also opt-in independent of that?

[atmyers]

It's independent of whether amrex.use_gpu_aware_mpi is on or not.

[ax3l]

Thanks! I thought so, the prior control vars changed in this diff just confused me :)