[Beowulf] compilers vs mpi?

[Mark Hahn]

> we do exactly what you describe: compile OpenMPI with the gcc suite
> and then use it with gcc, intel and open64 compilers.

nice.

> This works out-of-the-box, almost.
> The problem is the f90 module mpi.mod. This is (usually) a binary
> file and specific to the f90 compiler that was used to compile
> OpenMPI. But there is a way to solve this problem:

ah, .mod files.  we have for the most part ignored them entirely.
(what are we losing by doing that?)

> 1. compile openmpi using the gcc compilers, i.e., gfortran as the Fortran
>   compiler and install it in /usr/local/openmpi

this is perhaps a tangent, but we install everything we support
under /opt/sharcnet/$packagebasename/$ver.  for openmpi, we've had 
to bodge the compiler flavor onto that (/opt/sharcnet/openmpi/1.4.2/intel).

> 2. move the Fortran module to the directory
>   /usr/local/openmpi/include/gfortran. In that directory
>   create softlinks to the files in /usr/local/openmpi/include.
> 3. compile openmpi using ifort and install the Fortran module (and only
>   the Fortran module!) in /usr/local/openmpi/include/intel. In that
>   directory create softlinks to the files in /usr/local/openmpi/include.

I guess I'm surprised this works - aren't you effectively assuming that 
the intel and gfortran interfaces are compatible here?  that is, the app
compiles with the compiler-specific module, which basically promises 
a particular type-safe interface (signature) for MPI functions, but 
then the linker just glues them together without any way to verify 
the signature compatibility...

am I misunderstanding?


> 4. in /usr/local/openmpi/bin create softlinks mpif90.ifort
>   and mpif90.gfortran pointing to opal_wrapper. Remove the
>   mpif90 softlink.
> 5. Move /usr/local/openmpi/share/openmpi/mpif90-wrapper-data.txt
>   to /usr/local/openmpi/share/openmpi/mpif90.ifort-wrapper-data.txt.
>   Change the line includedir=${includedir} to:
>   includedir=${includedir}/intel
>   Copy the file to
>   /usr/local/openmpi/share/openmpi/mpif90.gfortran-wrapper-data.txt
>   and change the line includedir=${includedir} to
>   includedir=${includedir}/gfortran
> 6. Create a wrapper script /usr/local/openmpi/bin/mpif90:
>
> #!/bin/bash
> OMPI_WRAPPER_FC=`basename $OMPI_FC 2> /dev/null`
> if [ "$OMPI_WRAPPER_FC" = 'gfortran' ]; then
>   exec $0.gfortran "$@"
> else
>   exec $0.ifort "$@"
> fi

this is a tangent, but perhaps interesting.  we don't use the wrappers 
from the MPI package, but rather our own single wrapper which has some
built-in intelligence (augmented by info from the compiler's (environment)
module.)

> The reason we use gfortran in step 1 is that otherwise you get those
> irritating error messages from the Intel libraries, cf.
> http://www.open-mpi.org/faq/?category=building#intel-compiler-wrapper-compiler-warnings

hmm.  we work around those by manipulating the link arguments in our wrapper.

thanks!
-mark