Merge pull request open-mpi#15 from miked-mellanox/topic/sync_with_up…

…stream

sync with ompi-release/v1.8

NEWS

@@ -4,7 +4,7 @@ Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
 Copyright (c) 2004-2006 The University of Tennessee and The University
                         of Tennessee Research Foundation.  All rights
                         reserved.
-Copyright (c) 2004-2008 High Performance Computing Center Stuttgart, 
+Copyright (c) 2004-2008 High Performance Computing Center Stuttgart,
                         University of Stuttgart.  All rights reserved.
 Copyright (c) 2004-2006 The Regents of the University of California.
                         All rights reserved.
@@ -55,19 +55,35 @@ included in the vX.Y.Z section and be denoted as:
 
 1.8.4
 -----
+
 - Removed inadvertent change that set --enable-mpi-thread-multiple "on"
-  by default, thus impacting performance for non-threaded apps
+  by default, thus impacting performance for non-threaded apps.
 - Significantly reduced startup time by optimizing internal hash table
-  implementation
-- Fixed segfault in neighborhood collectives under certain use-cases
-- Fixed Solaris support
-- Fixed PMI configure tests for certain Slurm installation patterns
-- Fixed param registration issue in Java bindings
-- Several man page fixes
-- Silence several warnings and close some memory leaks
-- Re-enabled the use of CMA and knem in the shared memory BTL
-- Updated mpirun manpage to correctly explain new map/rank/binding options
-- Numerous updates and performance improvements to OSHMEM
+  implementation.
+- Fixed OS X linking with the Fortran mpi module when used with
+  gfortran >= 4.9.  Thanks to Github user yafshar for raising the
+  issue.
+- Fixed memory leak on Cygwin platforms.  Thanks for Marco Atzeri for
+  reporting the issue.
+- Fixed seg fault in neighborhood collectives when the degree of the
+  topology is higher than the communicator size.  Thanks to Lisandro
+  Dalcin for reporting the issue.
+- Fixed segfault in neighborhood collectives under certain use-cases.
+- Fixed Solaris support.
+- Fixed PMI configure tests for certain Slurm installation patterns.
+- Fixed param registration issue in Java bindings.  Thanks to Takahiro
+  Kawashima and Siegmar Gross for identifying the issue.
+- Several man page fixes.
+- Silence several warnings and close some memory leaks.
+- Re-enabled the use of CMA and knem in the shared memory BTL.
+- Updated mpirun manpage to correctly explain new map/rank/binding options.
+- Fixed MPI_IALLGATHER problem with intercommunicators.  Thanks for
+  Takahiro Kawashima for the patch.
+- Numerous updates and performance improvements to OSHMEM.
+- Turned off message coalescing in the openib BTL until a proper fix
+  for that capability can be provided (tentatively expected for 1.8.5)
+- Fix a bug in iof output that dates back to the dinosaurs which would
+  output extra bytes if the system was very heavily loaded
 
 
 1.8.3
@@ -229,7 +245,7 @@ included in the vX.Y.Z section and be denoted as:
 - Significantly improved the performance of MPI_DIMS_CREATE for large
   values.  Thanks to Andreas Sch�fer for the contribution.
 - Removed ASYNCHRONOUS keyword from the "ignore TKR" mpi_f08 module.
-- Deprecated the following mpirun options: 
+- Deprecated the following mpirun options:
   --bynode, --bycore, --byslot: replaced with --map-by node|core|slot.
   --npernode, --npersocket: replaced with --map-by ppr:N:node and
        --map-by ppr:N:socket, respectively
@@ -273,7 +289,7 @@ included in the vX.Y.Z section and be denoted as:
 - Restore version number output in "ompi_info --all".
 - Various bug fixes for the mpi_f08 Fortran bindings.
 - Fix ROMIO compile error with Lustre 2.4.  Thanks to Adam Moody for
-  reporting the issue. 
+  reporting the issue.
 - Various fixes for 32 bit platforms.
 - Add ability to selectively disable building the mpi or mpi_f08
   module.  See the README file for details.
@@ -383,7 +399,7 @@ included in the vX.Y.Z section and be denoted as:
   executables on systems that support it.  This behavior can be
   disabled via --disable-wrapper-rpath.  See note in README about ABI
   issues when using rpath in MPI applications.
-- Added a new parallel I/O component and multiple new frameworks to 
+- Added a new parallel I/O component and multiple new frameworks to
   support parallel I/O operations.
 - Fixed MPI_STATUS_SIZE Fortran issue when used with 8-byte Fortran
   INTEGERs and 4-byte C ints.  Since this issue affects ABI, it is
@@ -458,9 +474,9 @@ included in the vX.Y.Z section and be denoted as:
 - Added new "mindist" process mapper, allowing placement of processes
   via PCI locality information reported by the BIOS.
 - MPI-2.2: Add support for MPI_Dist_graph functionality.
-- Enable generic, client-side support for PMI2 implementations. Can 
-  be leveraged by any resource manager that implements PMI2; e.g. SLURM, 
-  versions 2.6 and higher.   
+- Enable generic, client-side support for PMI2 implementations. Can
+  be leveraged by any resource manager that implements PMI2; e.g. SLURM,
+  versions 2.6 and higher.
 
 1.7.2
 -----
@@ -528,7 +544,7 @@ included in the vX.Y.Z section and be denoted as:
 1.7
 ---
 
-- Added MPI-3 functionality: 
+- Added MPI-3 functionality:
     - MPI_GET_LIBRARY_VERSION
     - Matched probe
     - MPI_TYPE_CREATE_HINDEXED_BLOCK
@@ -590,7 +606,7 @@ included in the vX.Y.Z section and be denoted as:
 - Added new MCA parameter to provide protection against IO forwarding
   backlog.
 - Dropped support for native Windows due to loss of maintainers. :-(
-- Added a new parallel I/O component and multiple new frameworks to 
+- Added a new parallel I/O component and multiple new frameworks to
   support parallel I/O operations.
 - Fix typo in orte_setup_hadoop.m4. Thanks to Aleksej Saushev for
   reporting it
@@ -945,7 +961,7 @@ included in the vX.Y.Z section and be denoted as:
   Thanks to Dominik Goeddeke for finding this.
   (** also to appear in 1.4.4)
 - Improved LoadLeveler ORTE support.
-- Add new WinVerbs BTL plugin, supporting native OpenFabrics verbs on 
+- Add new WinVerbs BTL plugin, supporting native OpenFabrics verbs on
   Windows (the "wv" BTL).
 - Add new btl_openib_gid_index MCA parameter to allow selecting which
   GID to use on an OpenFabrics device's GID table.
@@ -1094,7 +1110,7 @@ included in the vX.Y.Z section and be denoted as:
 - Fixes for Solaris oversubscription detection.
 - If the PML determines it can't reach a peer process, print a
   slightly more helpful message.  Thanks to Nick Edmonds for the
-  suggestion. 
+  suggestion.
 - Make btl_openib_if_include/exclude function the same way
   btl_tcp_if_include/exclude works (i.e., supplying an _include list
   overrides supplying an _exclude list).
@@ -1161,7 +1177,7 @@ included in the vX.Y.Z section and be denoted as:
   (** also to appear in 1.5.5)
 - Removed an unnecessary memmove() and plugged a couple of small memory leaks
   in the openib OOB connection setup code.
-- Fixed some QLogic bugs. Thanks to Mark Debbage from QLogic for the patches. 
+- Fixed some QLogic bugs. Thanks to Mark Debbage from QLogic for the patches.
 - Fixed problem with MPI_IN_PLACE and other sentinel Fortran constants
   on OS X.
   (** also to appear in 1.5.5)
@@ -1196,7 +1212,7 @@ included in the vX.Y.Z section and be denoted as:
 - Corrected the reported default value of btl_openib_ib_timeout
   in the "IB retries exceeded" error message.  Thanks to Kevin Buckley
   for this correction.
-- Increased rdmacm address resolution timeout from 1s to 30s & 
+- Increased rdmacm address resolution timeout from 1s to 30s &
   updated Chelsio T4 openib BTL defaults.  Thanks to Steve Wise
   for these updates.
   (** also to appear in 1.5.5)
@@ -1371,7 +1387,7 @@ included in the vX.Y.Z section and be denoted as:
   details).
 - Fixed some problems in processor affinity support, including when
   there are "holes" in the processor namespace (e.g., offline
-  processors). 
+  processors).
 - Ensure that Open MPI's "session directory" (usually located in /tmp)
   is cleaned up after process termination.
 - Fixed some problems with the collective "hierarch" implementation
@@ -1399,7 +1415,7 @@ included in the vX.Y.Z section and be denoted as:
 - Various fixes for Catamount support.
 - Minor update to wrapper compilers in how user-specific argv is
   ordered on the final command line.  Thanks to Jed Brown for the
-  suggestions. 
+  suggestions.
 - Removed flex.exe binary from Open MPI tarballs; now generate flex
   code from a newer (Windows-friendly) flex when we make official
   tarballs.
@@ -1461,7 +1477,7 @@ and v1.4.
   change the environment variable that we use for obtaining the job's
   allocation.
 - Revert to an older, safer (but slower) communicator ID allocation
-  algorithm. 
+  algorithm.
 - Fixed minimum distance finding for OpenFabrics devices in the openib
   BTL.
 - Relax the parameter checking MPI_CART_CREATE a bit.
@@ -2097,7 +2113,7 @@ and v1.4.
 - Addition of "udapl" BTL for transport across uDAPL interconnects.
 - Really check that the $CXX given to configure is a C++ compiler
   (not a C compiler that "sorta works" as a C++ compiler).
-- Properly check for local host only addresses properly, looking 
+- Properly check for local host only addresses properly, looking
   for 127.0.0.0/8, rather than just 127.0.0.1.
 
 
@@ -2244,7 +2260,7 @@ and v1.4.
   MPI::Datatype::Create_resized, MPI::Datatype::Get_true_extent.
 - Fix OSX linker issue with Fortran bindings.
 - Fixed MPI_COMM_SPAWN to start spawning new processes in slots that
-  (according to Open MPI) are not already in use.  
+  (according to Open MPI) are not already in use.
 - Added capability to "mpirun a.out" (without specifying -np) that
   will run on all currently-allocated resources (e.g., within a batch
   job such as SLURM, Torque, etc.).
@@ -2399,7 +2415,7 @@ and v1.4.
 - Fixed a problem with handling long mpirun app files.  Thanks to Ravi
   Manumachu for identifying the problem.
 - Fix handling of strings in several of the Fortran 77 bindings.
-- Fix LinuxPPC assembly issues.  Thanks to Julian Seward for reporting 
+- Fix LinuxPPC assembly issues.  Thanks to Julian Seward for reporting
   the problem.
 - Enable pty support for standard I/O forwarding on platforms that
   have ptys but do not have openpty().  Thanks to Pierre Valiron for

ompi/mca/btl/vader/btl_vader_frag.h

@@ -61,6 +61,7 @@ struct mca_btl_vader_segment_t {
     mca_btl_base_segment_t base;
 #if OMPI_BTL_VADER_HAVE_KNEM
     uint64_t cookie;
+    intptr_t registered_base;
 #endif
 };
 typedef struct mca_btl_vader_segment_t mca_btl_vader_segment_t;

ompi/mca/btl/vader/btl_vader_get.c

@@ -101,6 +101,7 @@ int mca_btl_vader_get_knem (struct mca_btl_base_module_t *btl,
     mca_btl_vader_segment_t *src = (mca_btl_vader_segment_t *) des->des_src;
     mca_btl_vader_segment_t *dst = (mca_btl_vader_segment_t *) des->des_dst;
     const size_t size = min(dst->base.seg_len, src->base.seg_len);
+    intptr_t offset = src->base.seg_addr.lval - src->registered_base;
     struct knem_cmd_param_iovec recv_iovec;
     struct knem_cmd_inline_copy icopy;
 
@@ -111,7 +112,7 @@ int mca_btl_vader_get_knem (struct mca_btl_base_module_t *btl,
     icopy.local_iovec_array = (uintptr_t) &recv_iovec;
     icopy.local_iovec_nr    = 1;
     icopy.remote_cookie     = src->cookie;
-    icopy.remote_offset     = 0;
+    icopy.remote_offset     = offset;
     icopy.write             = 0;
     icopy.flags             = 0;
 

ompi/mca/btl/vader/btl_vader_module.c

@@ -482,6 +482,7 @@ struct mca_btl_base_descriptor_t *vader_prepare_dst(struct mca_btl_base_module_t
         }
 
         frag->segments[0].cookie = knem_cr.cookie;
+        frag->segments[0].registered_base = (intptr_t) data_ptr;
         frag->cookie = knem_cr.cookie;
     }
 #endif /* OMPI_BTL_SM_HAVE_KNEM */
@@ -611,7 +612,10 @@ static struct mca_btl_base_descriptor_t *vader_prepare_src (struct mca_btl_base_
 
             knem_cr.iovec_array = (uintptr_t) &knem_iov;
             knem_cr.iovec_nr = 1;
-            knem_cr.protection = PROT_READ;
+            /* NTH: there is no way to register a region for both read and write and get one cookie.
+             * As a workaround prepare_src should register for both. This is not in master as the BTL
+             * interface is changing to fix these kinds of issues. */
+            knem_cr.protection = PROT_WRITE | PROT_READ;
             /* Vader will explicitly destroy this cookie */
             knem_cr.flags = 0;
             if (OPAL_UNLIKELY(ioctl(mca_btl_vader.knem_fd, KNEM_CMD_CREATE_REGION, &knem_cr) < 0)) {
@@ -620,6 +624,7 @@ static struct mca_btl_base_descriptor_t *vader_prepare_src (struct mca_btl_base_
             }
 
             frag->segments[0].cookie = knem_cr.cookie;
+            frag->segments[0].registered_base = (intptr_t) data_ptr;
             frag->cookie = knem_cr.cookie;
         }
 #endif /* OMPI_BTL_SM_HAVE_KNEM */

ompi/mca/btl/vader/btl_vader_put.c

@@ -2,6 +2,8 @@
 /*
  * Copyright (c) 2010-2014 Los Alamos National Security, LLC. All rights
  *                         reserved.
+ * Copyright (c) 2014      Research Organization for Information Science
+ *                         and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
  *
  * Additional copyrights may follow
@@ -101,17 +103,18 @@ int mca_btl_vader_put_knem (struct mca_btl_base_module_t *btl,
     mca_btl_vader_segment_t *src = (mca_btl_vader_segment_t *) des->des_src;
     mca_btl_vader_segment_t *dst = (mca_btl_vader_segment_t *) des->des_dst;
     const size_t size = min(dst->base.seg_len, src->base.seg_len);
+    intptr_t offset = dst->base.seg_addr.lval - dst->registered_base;
     struct knem_cmd_param_iovec send_iovec;
     struct knem_cmd_inline_copy icopy;
 
     /* Fill in the ioctl data fields.  There's no async completion, so
        we don't need to worry about getting a slot, etc. */
-    send_iovec.base = (uintptr_t) dst->base.seg_addr.lval;
+    send_iovec.base = (uintptr_t) src->base.seg_addr.lval;
     send_iovec.len = size;
     icopy.local_iovec_array = (uintptr_t) &send_iovec;
     icopy.local_iovec_nr    = 1;
     icopy.remote_cookie     = dst->cookie;
-    icopy.remote_offset     = 0;
+    icopy.remote_offset     = offset;
     icopy.write             = 1;
     icopy.flags             = 0;
 

ompi/mca/coll/libnbc/nbc_ireduce_scatter.c

@@ -69,9 +69,12 @@ int ompi_coll_libnbc_ireduce_scatter(void* sendbuf, void* recvbuf, int *recvcoun
   redbuf = ((char*)handle->tmpbuf)+(ext*count);
 
   /* copy data to redbuf if we only have a single node */
-  if((p==1) && !inplace) {
-    res = NBC_Copy(sendbuf, count, datatype, redbuf, count, datatype, comm);
-    if (NBC_OK != res) { printf("Error in NBC_Copy() (%i)\n", res); return res; }
+  if(p==1) {
+    if(!inplace) {
+      res = NBC_Copy(sendbuf, count, datatype, redbuf, count, datatype, comm);
+      if (NBC_OK != res) { printf("Error in NBC_Copy() (%i)\n", res); return res; }
+    }
+    goto submit_and_return;
   }
 
   firstred = 1;
@@ -136,6 +139,7 @@ int ompi_coll_libnbc_ireduce_scatter(void* sendbuf, void* recvbuf, int *recvcoun
     if (NBC_OK != res) { free(handle->tmpbuf); printf("Error in NBC_Sched_copy() (%i)\n", res); return res; }
   }
 
+submit_and_return:
   /*NBC_PRINT_SCHED(*schedule);*/
 
   res = NBC_Sched_commit(schedule);

ompi/mca/io/ompio/io_ompio.c

@@ -986,7 +986,7 @@ int ompi_io_ompio_set_aggregator_props (mca_io_ompio_file_t *fh,
                     fh->f_procs_per_group = fh->f_procs_per_group%root_offset;
                 }
             }
-            else if ((size_t)mca_io_ompio_bytes_per_agg > 
+            else if ((size_t)mca_io_ompio_bytes_per_agg >= 
                      bytes_per_proc * fh->f_procs_per_group) {
                 i = ceil ((float)mca_io_ompio_bytes_per_agg/
                           (bytes_per_proc * fh->f_procs_per_group));
@@ -1146,7 +1146,7 @@ int ompi_io_ompio_set_aggregator_props (mca_io_ompio_file_t *fh,
                 fh->f_procs_per_group = fh->f_procs_per_group%root_offset;
             }
         }
-        else if ((size_t)mca_io_ompio_bytes_per_agg > 
+        else if ((size_t)mca_io_ompio_bytes_per_agg >= 
                  max_bytes_per_proc * fh->f_procs_per_group) {
             i = ceil ((float)mca_io_ompio_bytes_per_agg/
                       (max_bytes_per_proc * fh->f_procs_per_group));