-mca prrte_tools_entry <libs> for PMPI layering (and dynamic wrapper interception)

I'll start with a brief outline of the code changes, then a description of
what this feature is all about.

* scripts mkcode.py and all of the parse_mpi_standard/ directory:
    uses *.tex and *.py from the pythonization effort in the MPI standard
    to figure out prototypes for all the MPI functions and constructs wrapper
    functions for them all
* various mpicc-wrapper-data.txt.in etc files:
    these all get a couple new lines
        libs_profilesupport=-lmpiprofilesupport
        profilesupport_lib_file=libmpiprofilesupport.so
    and opal_wrapper.c puts this early in the link line, so an "ldd" of
    an MPI app would now have a "libmpiprofilesupport.so" in front of
    libmpi.so etc.
* Makefile changes:
    build a new libmpiprofilesupport.so two places:
    1. $MPI_ROOT/lib/libmpiprofilesupport.so [stub that defines nothing]
    2. $MPI_ROOT/lib/profilesupport/libmpiprofilesupport.so [real]
* schizo component entry:
    lookup the -mca prrte_tools_entry setting and use the ORTE_JOB_PREPEND_ENVAR
    attribute to get LD_LIBRARY_PATH and possibly LD_PRELOAD modified
    to support the specified profiling libraries

Next I'll give a pretty long description of what profile interface
layering is about. Although to be honest, I think the main part of
this feature that people actually use is only the dynamic part. Eg at
runtime this lets you just add "-mca prrte_tools_entry ./libmpe.so" for example
without relinking your app and it would run as if you had linked
with -lmpe.

------------------------------------------------------------------------
MPI Profiling Interface Layering

The MPI standard defines a PMPI profiling interface, but in its normal usage only one profiling library can be used at a time.

A profiling wrapper library defines some subset of MPI_* entrypoints, and inside those redefinitions, it calls some combination of MPI_* and PMPI_* symbols.  For example

int
MPI_Allgather(void *sbuf, int scount, MPI_Datatype sdt,
    void *rbuf, int rcount, MPI_Datatype rdt, MPI_Comm comm)
{
    int rval;
    double t1, t2, t3;
    t1 = MPI_Wtime();
    MPI_Barrier(comm);
    t2 = MPI_Wtime();
    rval = PMPI_Allgather(sbuf, scount, sdt, rbuf, rcount, rdt, comm);
    t3 = MPI_Wtime();
    // record time waiting vs time spent in allgather..
    return(rval);
}
double MPI_Wtime() {
  // insert hypothetical high-resolution replacement here, for example
}

In trying to use two unrelated wrapper libraries at the same time, it's in general not possible to link them in such a way that proper layering occurs.  For example, consider two wrapper libraries:
1. "libJobLog.so" wrapping MPI_Init and MPI_Finalize to keep a log of
   every MPI job, listing hosts, runtimes, and cpu times.
2. "libCollPerf.so" wrapping MPI_Init and MPI_Finalize, and all the
   MPI collectives to gather statistics about how evenly the ranks
   enter the collectives.

With ordinary linking, each MPI_* call would resolve into one of the wrapper libraries, and from there the wrapper library's call to PMPI_* would resolve into the bottom level libmpi.so.  Only one of the libraries would have its MPI_Init and MPI_Finalize routines called.

----------------------------------------------------------------
Defining consistent layering behavior:

With dynamically loaded symbols, it is possible to design a consistent approach to layering for an arbitrary number of wrapper libraries.

When a wrapper library "libwrap.so" redefines an MPI_* symbol, there are two possibilities for what MPI calls it can make.  It can call another MPI_* entry, or it can call a PMPI_* entry.  In the case of ordinary single-level wrapping, the calls into MPI_* would resolve into "libwrap.so" first and then "libmpi.so" if not found.  And the calls to PMPI_* would resolve to "libmpi.so".

In the case of multi-level wrapping, the equivalent behavior is for MPI_* to resolve to the current level, and PMPI_* to resolve to the next level down in the list of libraries.

We would set up an array of handles
  void *lib[MAX_LEVELS];
containing the dlopened handles for all the libraries, eg
  lib[0] = dlopen("libwrap3.so", );
  lib[1] = dlopen("libwrap2.so", );
  lib[2] = dlopen("libwrap1.so", );
  lib[3] = dlopen("libmpi.so", );

For each MPI function an array of function pointers would exist, one for each library:
  int (*(fptr_MPI_Send[MAX_LEVELS]))();
Some of the entries would be null, but the bottom level should always be defined.

And a thread-local global would define the calldepth for the thread, initially 0.  The calldepth needs to be thread-local data so the implementation can be thread-safe, since different threads would have unrelated call stacks.

Model implementation of MPI_Send and PMPI_Send:

int
MPI_Send(void* buf, int count, MPI_Datatype dt, int to, int tag, MPI_Comm comm)
{
  int rval;
  int entrylev, nextlev;
  int *calldepth;
  calldepth = (int*) pthread_getspecific(depthkey); // thread-local global
  entrylev = *calldepth;
  nextlev = entrylev;  // (or nextlev = entrylev + 1 for PMPI_Send)
  if (nextlev >= nwrapper_levels) { --nextlev; }
  while (!fptr_MPI_Send[nextlev] && nextlev<nwrapper_levels) { ++nextlev; }
  if (nextlev >= nwrapper_levels) {
    printf("Fatal Error: unable to find symbol at level>=%d for %s\\n",
      entrylev, "MPI_Send");
    exit(1);
  }
  *calldepth = nextlev;
  rval = fptr_MPI_Send[nextlev](buf, count, dt, to, tag, comm);
  *calldepth = entrylev;
  return(rval);
}

If code like the above is in a library called libmpiprofilesupport.so and an MPI app is linked against that library, then an example sequence of events might be

- app.x calls MPI_Send, the linker resolves it in libmpiprofilesupport.so
- above code sees calldepth=0, calls fptr_MPI_Send[0] == MPI_Send in libwrap1.so
- libwrap1.so's MPI_Send calls PMPI_Send, it resolves into libmpiprofilesupport.so
- above code increments calldepth to 1, calls fptr_MPI_Send[1] == MPI_Send
  in libwrap2.so

And so on.  At each MPI/PMPI call, the linker/loader resolves to the symbol in libmpiprofilesupport.so, and that library decides which function pointer to go into next.

----------------------------------------------------------------
Fortran

I believe Open MPI makes the profiling design choice that wrapping the C-language MPI_Send() automatically results in Fortran mpi_send() being wrapped.

The "calldepth" behavior from above isn't a perfect match for this situation.  For comparison, the traditional behavior for an application linked against a single libwrap.so would go as follows.  An "ldd" on the application might show dependencies
    libwrap.so
    libmpi_mpifh.so
    libmpi.so
and an application call to fortran mpi_send() would find the first definition in libmpi_mpifh.so.0 where it would call MPI_Send() which would go back up to libwrap.so which might call PMPI_Send() which would then go down to libmpi.so.  The linker/loader starts at the top in its search for each symbol.

The layered profiling approach with a "calldepth" always goes down the list of libraries.  If fortran is at the bottom of the list, those symbols would end up not being intercepted.

I think the easiest way to maintain the desired behavior is to re-order the libraries in the dynamic case as
  lib[0] = dlopen("libmpi_mpifh.so", RTLD_NOW|RTLD_GLOBAL);
  lib[1] = dlopen("libwrap.so",      RTLD_NOW|RTLD_GLOBAL);
  .. other wrapper libraries ..
  lib[n] = dlopen("libmpi.so",       RTLD_NOW|RTLD_GLOBAL);
So the fortran wrapper is always first, and libmpi.so is always last, with all the wrapper libraries inbetween.

Internally the -mca prrte_tools_entry feature produces a list of entrypoint levels with three types:
  1. wrapper libraries like libwrap.so
  2. base MPI implementation which is two libraries libmpi.so and libmpi_mpifh.so
  3. just the fortran calls from the base MPI implementation
and allows "fort" to be manually included in the list of libraries.

So if one had a library libwrap.so that defined MPI_Send and called PMPI_Send, then the syntax
    -mca prrte_tools_entry fort,libwrap.so
would produce a list of entry levels as
  level[0] = fortran symbols from libmpi_mpifh.so
  level[1] = libwrap.so
  level[2] = all the base MPI symbols from libmpi.so and libmpi_mpifh.so
and if an application called fortran mpi_send, the call sequence would be
- app.x calls mpi_send, the linker resolves it in libmpiprofilesupport.so
- calldepth=0, calls fptr_mpi_send[0] == mpi_send in libmpi_mpifh.so
- libmpi_mpifh.so's mpi_send calls PMPI_Send, which resolves into libmpiprofilesupport.so
- calldepth=1, calls fptr_MPI_Send[1] == MPI_Send in libwrap.so
- libwrap.so's MPI_Send calls PMPI_Send, which resolves into libmpiprofilesupport.so
- calldepth=2, calls fptr_MPI_Send[2] == MPI_Send in libmpi.so
so including the OMPI fortran wrappers in the list in front of libwrap.so enabled automatic
wrapping of mpi_send by only redefining MPI_Send. If this behavior is not desired, it's possible
tp use the syntax
    -mca prrte_tools_entry libwrap.so,fort
which puts the fortran symbols at the bottom of the list where all the base MPI symbols are.

----------------------------------------------------------------
Performance

On a machine where OMPI pingpong takes 0.22 usec, the -mca prrte_tools_entry option slowed the pingpong to 0.24 usec.

This is enough of an impact I wouldn't consider putting the code from libmpiprofilesupport.so into the main libmpi.so library. But as long as the code is isolated in its own libmpiprofilesupport.so and LD_LIBRARY_PATH is used to point at a stub library there is no performance impact when the feature isn't activated.

----------------------------------------------------------------
Weaknesses

I think the main weakness is that the above design only handles MPI_* calls.  If a wrapper library wanted to intercept both MPI_* calls and all malloc()/free() calls for example, the above would result in the malloc()/free() not being intercepted (the app isn't linked against libwrap.so, it's only linked against libmpiprofilesupport.so which has a dlopen() of libwrap.so).

This commit includes a "preload" feature (that can be included in the -mca prrte_tools_entry list) that would be needed with such libraries. Then the malloc/free/etc from libwrap.so would be used. There wouldn't be any kind of layering for the non-MPI symbols if multiple libraries were trying to intercept malloc/free for example.

Another minor weakness is that MPI_Pcontrol(level, ...)'s variable argument list is impossible to maintain as far as I know.  The proper behavior would be that if an application calls MPI_Pcontrol(1, "test", 3.14), we should call MPI_Pcontrol with those arguments in every wrapper library in the list.  But there's no way in stdarg.h to extract and re-call with an unknown list of arguments.  The best we can do is call MPI_Pcontrol(level) at each library.

----------------------------------------------------------------
Documentation for this feature:

  [Dynamic MPI Profiling interface with layering]

    The MCA option -mca prrte_tools_entry <list> can be used to enable interception
    from profiling libraries at runtime.

    In traditional MPI usage a wrapper library "libwrap.so" can be built
    that redefines selected MPI calls, and using that library would
    require relinking the application with -lwrap to enable the
    interception.

    But this feature allows such interception to be enabled at runtime
    without relinking by using the mpirun option
      -mca prrte_tools_entry libwrap.so

    The -mca prrte_tools_entry feature can be used as any of
      -mca prrte_tools_entry /path/to/libwrap.so
      -mca prrte_tools_entry libwrap.so  (if the library will be found via
                             LD_LIBRARY_PATH)
      -mca prrte_tools_entry wrap        (shortcut for libwrap.so)

    Note that this feature doesn't automatically set LD_LIBRARY_PATH so
    that "libwrap.so" can be found. That could be done by using the
    additional option
      -x OMPI_LD_LIBRARY_PATH_PREPEND=<dir>:<dir>:...

    To layer multiple libraries, a comma separted list can be used:
      -mca prrte_tools_entry libwrap1.so,libwrap2.so

    A few other keywords can be added into the -mca prrte_tools_entry list:
      -mca prrte_tools_entry v       : verbose option, list the opened libraries
      -mca prrte_tools_entry preload : cause the wrapper libraries to be added to an
                           LD_PRELOAD. This could be needed if a library
                           redefine non-MPI symbols
      -mca prrte_tools_entry fortran : a layer that minimally wraps the Fortran MPI
                           calls on top of the C calls, eg defining
                           mpi_foo and calling PMPI_Foo
      -mca prrte_tools_entry fort    : short for fortran above

    By default Fortran wrappers are placed at the top of the list and
    the base product is always placed at the bottom, so
      -mca prrte_tools_entry libwrap.so
    would be equivalent to
      -mca prrte_tools_entry fortran,libwrap.so
    and would produce a layering as
      level[0] = fortran symbols defining mpi_foo and calling PMPI_Foo
      level[1] = libwrap.so
      level[2] = MPI from the base product

    In this way if libwrap.so defined MPI_Send and an application used
    Fortran mpi_send the MPI_Send call in libwrap.so would be triggered.
    If that behavior was not desired, the fortran wrappers can be
    essentially disabled by moving them to the bottom of the list, eg
      -mca prrte_tools_entry libwrap.so,fortran

----------------------------------------------------------------
Signed-off-by: Mark Allen <[email protected]>

Author: markalle

config/ompi_config_files.m4

@@ -6,6 +6,7 @@
 # Copyright (c) 2018      Los Alamos National Security, LLC. All rights
 #                         reserved.
 # Copyright (c) 2018      FUJITSU LIMITED.  All rights reserved.
+# Copyright (c) 2020      IBM Corporation. All rights reserved.
 # $COPYRIGHT$
 #
 # Additional copyrights may follow
@@ -58,5 +59,7 @@ AC_DEFUN([OMPI_CONFIG_FILES],[
         ompi/tools/wrappers/mpijavac.pl
         ompi/tools/mpisync/Makefile
         ompi/tools/mpirun/Makefile
+        ompi/tools/pmpi_dynamic_layering/stub/Makefile
+        ompi/tools/pmpi_dynamic_layering/real/Makefile
     ])
 ])

ompi/runtime/ompi_mpi_params.c

@@ -19,6 +19,7 @@
  *                         All rights reserved.
  * Copyright (c) 2016-2019 Research Organization for Information Science
  *                         and Technology (RIST). All rights reserved.
+ * Copyright (c) 2020      IBM Corporation. All rights reserved.
  * $COPYRIGHT$
  *
  * Additional copyrights may follow
@@ -83,6 +84,8 @@ static bool show_file_mca_params = false;
 static bool show_enviro_mca_params = false;
 static bool show_override_mca_params = false;
 static bool ompi_mpi_oversubscribe = false;
+static char *entry_string = NULL;
+static char *entry_base_string = NULL;
 
 int ompi_mpi_register_params(void)
 {
@@ -340,6 +343,24 @@ int ompi_mpi_register_params(void)
                                  MCA_BASE_VAR_SCOPE_READONLY,
                                  &ompi_mpi_spc_dump_enabled);
 
+    /* -mca tools_entry <libraries-and-options> */
+    (void) mca_base_var_register("ompi", "tools", NULL, "entry",
+                                  "options for dynamic PMPI wrapper layering",
+                                  MCA_BASE_VAR_TYPE_STRING, NULL,
+                                  0, 0,
+                                  OPAL_INFO_LVL_3,
+                                  MCA_BASE_VAR_SCOPE_READONLY,
+                                  &entry_string);
+
+    /* -mca tools_entry_base <libraries> */
+    (void) mca_base_var_register("ompi", "tools", NULL, "entry_base",
+                                  "base libraries for dynamic PMPI wrapper layering",
+                                  MCA_BASE_VAR_TYPE_STRING, NULL,
+                                  0, 0,
+                                  OPAL_INFO_LVL_3,
+                                  MCA_BASE_VAR_SCOPE_READONLY,
+                                  &entry_base_string);
+
     return OMPI_SUCCESS;
 }
 

ompi/tools/Makefile.am

@@ -11,7 +11,7 @@
 # Copyright (c) 2004-2005 The Regents of the University of California.
 #                         All rights reserved.
 # Copyright (c) 2014      Intel, Inc. All rights reserved.
-# Copyright (c) 2020      IBM Corporation.  All rights reserved.
+# Copyright (c) 2019-2020 IBM Corporation. All rights reserved.
 # $COPYRIGHT$
 #
 # Additional copyrights may follow
@@ -25,10 +25,14 @@ SUBDIRS += \
 	tools/mpirun \
 	tools/ompi_info \
 	tools/wrappers \
+	tools/pmpi_dynamic_layering/stub \
+	tools/pmpi_dynamic_layering/real \
         tools/mpisync
 
 DIST_SUBDIRS += \
 	tools/mpirun \
 	tools/ompi_info \
 	tools/wrappers \
+	tools/pmpi_dynamic_layering/stub \
+	tools/pmpi_dynamic_layering/real \
         tools/mpisync

ompi/tools/pmpi_dynamic_layering/parse_mpi_standard/README.txt

@@ -0,0 +1,38 @@
+A portion of the MPI standard is checked in here:
+    binding-tool/*.py
+
+Those are the tools that make sense of blocks like
+    function_name("MPI_Send")
+
+    parameter(
+        "buf", "BUFFER", desc="initial address of send buffer", constant=True
+    )
+    parameter(
+        "count", "POLYXFER_NUM_ELEM", desc="number of elements in send buffer"
+    )
+    parameter(
+        "datatype", "DATATYPE", desc="datatype of each send buffer element"
+    )
+    parameter("dest", "RANK", desc="rank of destination")
+    parameter("tag", "TAG", desc="message tag")
+    parameter("comm", "COMMUNICATOR")
+that are scattered throughout the standard's *.tex file.
+
+The *.tex files from the MPI standard are not checked in here, but an
+extraction tool and the result of running it are:
+    extract.py
+    generated_mpistandard_parsesets.py
+
+The "extract.py" tool does a find of *.tex and prints all the
+python blocks it finds to stdout.  The results in
+"generated_mpistandard_parsesets.py" are what happens if this is
+run in a check out of the MPI standard (with Jeff Squyres'
+pythonization checked in).
+
+A user of this MPI standard data can use calls like
+    sys.path.append("path/to/parse_mpi_standard")
+    from generated_mpistandard_parsets import \
+        run_everything_from_the_standard, MPI_FN_DATA
+    run_everything_from_the_standard()
+    # At this point the dict MPI_FN_DATA is populated
+    PARSESET = MPI_FN_DATA['MPI_Send']

ompi/tools/pmpi_dynamic_layering/parse_mpi_standard/binding-tool/binding.py

@@ -0,0 +1,129 @@
+'''
+
+'''
+
+
+from typing import Mapping, MutableMapping
+import copy
+import logging
+from collections import defaultdict
+
+
+import common
+from userfuncs import *
+
+
+DEFAULT_ATTRIBUTES = {
+        # validity attributes
+        'lis_expressible': True,
+        'c_expressible': True,
+        'f08_expressible': True,
+        'f90_expressible': True,
+
+        # TODO
+        'f90_use_colons': False,
+        # weird index overloading
+        'f90_index_overload': None,
+
+        # callback attributes
+        'callback': False,
+        'predefined_function': None,
+
+        # semantic attributes
+        'deprecated': False,
+        'execute_once': False,
+
+        # control dependencies
+        # 'needs_any': ['MPI_Init', 'MPI_Init_thread'],
+        # 'needs_all': [],
+        # 'leads_any': ['MPI_Finalize'],
+        # 'leads_all': [],
+    }
+
+
+DEFAULT_TEMPORARIES = {
+        'has_ierror': True,
+
+        # whether this binding is from a definition
+        'reference': None,
+
+        # which expressible languages should be rendered
+        # at definition mpi-binding
+        'renders': ['lis', 'c', 'f08', 'f90'],
+    }
+
+
+def reset_parseset() -> None:
+    """
+    Resets the global variable PARSESET.
+    """
+
+    # global PARSESET
+    # PARSESET = dict()
+    PARSESET.clear()
+
+    # name -> LIS/C/F08, name_f90 -> F90
+    # these must be unique, otherwise you have big problems
+    PARSESET['name'] = None
+    PARSESET['name_f90'] = None
+
+    # API attributes
+    PARSESET['parameters'] = list()
+    PARSESET['return_kind'] = None
+    PARSESET['attributes'] = copy.deepcopy(DEFAULT_ATTRIBUTES)
+
+    # not needed in apis.json
+    PARSESET['temporaries'] = copy.deepcopy(DEFAULT_TEMPORARIES)
+
+
+def parsing_done() -> None:
+    """
+    This hook is run after all the parsing is done for a given binding.
+    """
+
+    if PARSESET['temporaries']['has_ierror']:
+        parameter('ierror',
+                  'ERROR_CODE',
+                  optional=True,
+                  direction='out',
+                  suppress='c_parameter lis_parameter')
+
+    # default to an INT return type
+    if PARSESET['return_kind'] is None:
+        # MR this is not strictly true, Fortran just doesn't have a return type
+        # since they are subroutines
+        return_type('ERROR_CODE')
+
+
+def execute_binding(binding: str, require_definition: bool = False) -> Mapping:
+    '''
+    Executes the content of a matched binding block.
+    '''
+
+    reset_parseset()
+
+    logging.info("Rendering mpi-binding to LaTeX:")
+    for line in binding.split('\n'):
+        if line:
+            logging.info('>> %s', line)
+
+    # if definition required
+    #     then ignore bindings without function_name
+    if require_definition and 'function_name' not in binding:
+        return None
+
+    # execute the lines read from the mpi-binding
+    # pylint: disable=exec-used
+    exec(common.remove_unexpected_indentation(binding))
+
+    parsing_done()
+
+    # write to dataset
+    logging.debug('parsed %s', PARSESET)
+    parseset = copy.deepcopy(PARSESET)
+
+    # clear parsed information
+    logging.debug('clearing PARSESET: %s',
+                  str(PARSESET))
+
+    return parseset