[Beowulf] compilers vs mpi?

[Martin Siegert]

On Tue, Jul 20, 2010 at 02:25:32PM -0400, Mark Hahn wrote:
>> we do exactly what you describe: compile OpenMPI with the gcc suite
>> and then use it with gcc, intel and open64 compilers.
>
> nice.
>
>> This works out-of-the-box, almost.
>> The problem is the f90 module mpi.mod. This is (usually) a binary
>> file and specific to the f90 compiler that was used to compile
>> OpenMPI. But there is a way to solve this problem:
>
> ah, .mod files.  we have for the most part ignored them entirely.
> (what are we losing by doing that?)

For starters, a program with a "use mpi" statement won't compile, if
you don't have mpi.mod.

>> 1. compile openmpi using the gcc compilers, i.e., gfortran as the Fortran
>>   compiler and install it in /usr/local/openmpi
>
> this is perhaps a tangent, but we install everything we support
> under /opt/sharcnet/$packagebasename/$ver.  for openmpi, we've had to bodge 
> the compiler flavor onto that (/opt/sharcnet/openmpi/1.4.2/intel).

(I simplified the description: we install in /usr/local/openmpi-version
and then create a softlink /usr/local/openmpi that points to the current
default version)

>> 2. move the Fortran module to the directory
>>   /usr/local/openmpi/include/gfortran. In that directory
>>   create softlinks to the files in /usr/local/openmpi/include.
>> 3. compile openmpi using ifort and install the Fortran module (and only
>>   the Fortran module!) in /usr/local/openmpi/include/intel. In that
>>   directory create softlinks to the files in /usr/local/openmpi/include.
>
> I guess I'm surprised this works - aren't you effectively assuming that the 
> intel and gfortran interfaces are compatible here?  that is, the app
> compiles with the compiler-specific module, which basically promises a 
> particular type-safe interface (signature) for MPI functions, but then the 
> linker just glues them together without any way to verify the signature 
> compatibility...
>
> am I misunderstanding?

The (default) name mangling scheme of gfortran, ifort, openf90 is the same:
append a single underscore. As soon as a user uses -fno-underscoring or
-fsecond-underscore nothing would work anymore. So, don't do that.
We have had not a single user who tried to change the default name mangling
scheme, thus this is not really a problem. We no longer support
g77 (which does have a different default).

As far as I understand the type checking with respect to mpi.mod is done
at compile time. That's why you need to have the correct fortran module
when compiling. At link time the linker only needs to find the library
routines like mpi_send_ which is a C wrapper routine anyway that just calls
MPI_Send.

>> 4. in /usr/local/openmpi/bin create softlinks mpif90.ifort
>>   and mpif90.gfortran pointing to opal_wrapper. Remove the
>>   mpif90 softlink.
>> 5. Move /usr/local/openmpi/share/openmpi/mpif90-wrapper-data.txt
>>   to /usr/local/openmpi/share/openmpi/mpif90.ifort-wrapper-data.txt.
>>   Change the line includedir=${includedir} to:
>>   includedir=${includedir}/intel
>>   Copy the file to
>>   /usr/local/openmpi/share/openmpi/mpif90.gfortran-wrapper-data.txt
>>   and change the line includedir=${includedir} to
>>   includedir=${includedir}/gfortran
>> 6. Create a wrapper script /usr/local/openmpi/bin/mpif90:
>>
>> #!/bin/bash
>> OMPI_WRAPPER_FC=`basename $OMPI_FC 2> /dev/null`
>> if [ "$OMPI_WRAPPER_FC" = 'gfortran' ]; then
>>   exec $0.gfortran "$@"
>> else
>>   exec $0.ifort "$@"
>> fi
>
> this is a tangent, but perhaps interesting.  we don't use the wrappers from 
> the MPI package, but rather our own single wrapper which has some
> built-in intelligence (augmented by info from the compiler's (environment)
> module.)

Again, I simplified. We use, e.g., compiler env modules to set env. variables
like OMPI_FC.

>> The reason we use gfortran in step 1 is that otherwise you get those
>> irritating error messages from the Intel libraries, cf.
>> http://www.open-mpi.org/faq/?category=building#intel-compiler-wrapper-compiler-warnings
>
> hmm.  we work around those by manipulating the link arguments in our wrapper.

Does that work when using gfortran to link with libraries compiled with
icc/ifort? Anyway, it appeared to be an unnecessary complication as the
fortran compiler has no affect on the performance of the MPI distribution;
it is only really needed for compiling the f90 module (and for configure
to determine the name mangling scheme).

- Martin