doc: add tuning parameters for MPI_THREAD_MULTIPLE

These are the currently supported tuning parameters for
MPI_THREAD_MULTIPLE.

doc/mpich/TUNING_PARAMETERS

@@ -2,3 +2,32 @@ This document discusses some of the new features added to the MPICH-based MPI im
 
 Notice that performance of the application depends on how ranks are mapped to the cores of the processor. For information on how to specify process-core binding with mpirun, information is provided here:
 https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Process-core_Binding.
+
+This following section of this document provides information about threading optimizations (Section 1).
+
+**1. Multi-threading**
+
+
+MPI provides four different threading modes: MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, MPI_THREAD_SERIALIZED, and MPI_THREAD_MULTIPLE. MPI_THREAD_MULTIPLE is the most advanced threading mode and the focus of this Section. With this mode, an application runs with several threads, and any thread can make calls to MPI at any time. Thus, the MPI library needs to protect access to its internal data structures appropriately. It also tries to exploit concurrency by taking advantage of the availability of multiple hardware contexts in the network card, which we call VCIs (Virtual Communication Interfaces).
+
+MPICH provides two threading synchronization strategies. One strategy is to use a `global` coarse-grained lock to protect every MPI call that accesses shared state inside the MPI runtime. With this model, if two threads call concurrently the MPI runtime, the runtime will serialize the calls. Another strategy uses a fine-grained lock to protect the access to each network context or VCI. With this approach, threads trying to access the same VCI get serialized, but threads accessing different VCIs can execute in parallel. With the fine-grained locking model, MPICH supports a `direct` mode where the MPI runtime itself grabs a per VCI lock and a `lockless` mode where MPI does not grab a lock and relies on the network provider to provide fine grain synchronization (MPICH uses the FI_THREAD_SAFE mode from OFI to support the `lockless` mode).
+
+**1.1 CVARs (Environment Variables)**
+
+MPICH users can use the CVAR MPIR_CVAR_CH4_MT_MODEL to select at runtime among the threading synchronization models, `global`, `direct`, or `lockless`. If the CVAR is not set, the default model is `direct`.
+
+Users can specify the number of VCIs to use by setting the environment variable `MPIR_CVAR_CH4_NUM_VCIS=<number_vcis>`, where the value `<num_vcis>` should be less or equal than the number of network contexts provided by the OFI provider. This number can be retrieved from the OFI provider. Running the  `fi_info -v` command should return the number of transmit contexts (max_ep_tx_ctx) and receive contexts (max_ep_rx_ctx) it supports. MPICH uses the minimum of max_ep_tx_ctx and max_ep_rx_ctx as the maximum number of VCIs it can support. If the system has multiple NICS, this value is the `minimum across all the NICS`.
+
+When multiple VCIs are used, MPICH tries to make per-vci progress to avoid lock contention inside the progress engine. However, it also progresses all VCIs, i.e., make global progress, once in a while to avoid potential deadlocks. Such global progress is turned on by default but can be turned off by setting `MPIR_CVAR_CH4_GLOBAL_PROGRESS=0`, which can improve performance under the use of multiple VCIs. However, caution should be taken as it can result in a deadlock.
+
+**3.2 Communicator Info Hints**
+
+When selecting the fine-grain modes (`direct` or `lockless`), messages sent using different communicators can be sent/received using different VCIs. However, messages using the same communicator may have to be sent/received through the same VCI, to guarantee the ordering semantics imposed by the MPI standard. However, messages sent through the same communicator can be sent/received through different VCIs if the MPI runtime knows that wildcards will not be used on that communicator. Thus, MPICH implements the MPI 4.0 per-communicator info hints and recommend users to use them to  indicate the MPI runtime when wildcards are not used, specially when using `direct` or `lockless` threading modes with multiple VCIs if the threads use the same communicator.
+Here are the standard hints from the MPI 4.0 standard that are supported (we recommend using both info hints if the application is not using wildcards):
+
+* `mpi_assert_no_any_source`: Set this info hint if the application does not use the wildcard MPI_ANY_SOURCE on the given communicator.
+* `mpi_assert_no_any_tag`: Set this hint if the application does not use MPI_ANY_TAG on the given communciator.
+
+MPICH also supports a non-standard per-communicator info hint that provides the user with more control of a communicator-VCI mapping:
+* `vci`: Specific sender and receiver VCI can be selected for each communicator using this hint. Applications can utilize this feature to control how messages are spread across the VCIs.
+For instance, programmers can write the applications so that pairs of threads communicate by using different communicator, where each communicator can be mapped to a different VCI through the `vci` info hint. Valid values for this info hints are 0 to (total vcis - 1). This hint is also utilized by MPICH to select a progress thread to process reduction collective operation.