[Beowulf] The Case for an MPI ABI

[Ashley Pittman]

On Thu, 2005-02-24 at 16:45 +0100, Joachim Worringen wrote:
> My experience is that such sites want to use their (expensive, 
> commercial, Fortran) compiler to optimize the binaries to their 
> platform. In some cases, this requires source-code changes (parameters) 
> anyway. It's not about PMB.

The difference here is between "want" and "need".  If they want to do it
then well done, congratulations, it is often the right thing to do for
performance reasons.  In terms of setup time to get a working cluster
though there is a difference, having things work out the box is a good
thing (tm).

Of course there is a potential downside that if if works out of the box
then they won't play with compilers (why bother - it works?) and not
even try to get the extra performance.  This isn't a valid reason not to
have a ABI though.

> > You also make the assumption that it's the high-performance vendors who
> > do things differently, I don't believe this is the case.  Quadrics for
> > example (my employer) happen to use whatever ABI MPICH (1.2.x) provides
> [...]
> 
> I can't see where I made this assumption.

I was referring to this quote from earlier.  I thought you were implying
that code should come pre-linked with TCPIP MPICH (in effect a de-facto
standard) and us "exotic" people would be on our own.  In this context
there is nothing exotic or unusual about our MPI.  There is no
relationship between the performance of a MPI stack and how much it's
interface varies.  Sounds like we are on the same wavelength here
though.

>> For open source software packages alone, an ABI is not of critical
>> importance as people with a tcp/ip cluster can use pre-linkked
>> packages, and people with a high-perfomance interconnect cluster
>> typically have  enough competence to compile the software themselves.

> Indeed, most interconnect 
> vendors (Quadrics, various Infiniband, Myrinet, ...) happily plug their 
> low-level stuff into the latest MPICH and are done. So, it's most often 
> the cross-interconnect MPI vendors which create their own ABI for some 
> reason. Other cases are vendors (like us) who started to provide MPI-2 
> when there was no open-source MPI-2 around. We had to do our own 
> definitions then.

Hhmm.  Does this mean that the only reasons for not having a ABI are
historical, purely because we have never had one and there isn't the
inertia to change this?  Are there valid technical/performance reasons
for the need to change mpi.h?

> But this doesn't really matter after all; what matters is if there are 
> enough parties to take part in this effort, and to understand the 
> related issues as much as possible and as early as possible.

Plus the fact that someone (everyone?) has to take the hit of breaking
binary compatibility with all their previous MPI releases when they make
the jump to being compliant.  Any volunteers?  This might not actually
be so bad with the shared library number versioning scheme, in fact it
might be possible to avoid it completely at the cost of a bit more
effort in the packaging.

Ashley,