[Beowulf] Re: Re: Home beowulf - NIC latencies

[Patrick Geoffray]

Joachim,

Joachim Worringen wrote:
> Patrick Geoffray wrote:
> 
>> While we are at it, here is my wish list for the next MPI specs:
>>
>> a) only non-blocking calls. If there are no blocking calls, nobody 
>> will use them.
> 
> 
> While this makes sense technically, nobody will probably offer an MPI 
> implementation without MPI_Send for the next 20 years for compatibility 
> reasons, so we can just forget about it.

Throw away compatibility. If you keep the legacy API, you have no 
incentive for change. I don't want MPI-3, I want MPI-light. We are 
against a wall because the MPI spec was too rich and developers took the 
lazy path.

The weight of legacy will make shared memory paradigms the only proposal 
for the next step. If you believe we have to support the whole MPI 
semantics in the next message passing standards, then we are doomed.

>> c) ban of the ANY_SENDER wildcard: a world of optimization goes away 
>> with this convenience.
> 
> 
> I think this could best be achieved with an assertion like those for 
> one-sided and I/O. There are situations where ANY_SENDER is needed, or 
> at least avoids large programming overheads.

It's used because it's there, there is no other reason. If you don't 
know who sends you what in a message passing application, then you 
cannot get either performance or robustness. If really you cannot do 
otherwise (and I don't believe that), you can always use unexpected 
messages (post the receive after Probe()ing), That's ugly, but you get 
what you deserved :-)

>> d) throw away the user defined datatypes, or at least restrict it to 
>> regular strides.
> 
> 
> This is nonsense: user-defined datatypes do not cause any overhead if 
> you don't use them, there are ways to implemenent them very efficiently, 
> and you can't do without in many situations (like MPI-IO).

I know this item would itch, you spend a lot of time working on that.

If you don't use user-defined datatypes, then you don't need it and it 
should not be there in the first place. It's a temptation, it's too 
easy. No, there is no ways to implement them efficiently unless they are 
regular, and this is what I am willing to keep: strided types with long 
segments. Everything else leads to memory copies. The developer should 
wipe his own bottom instead of asking the message passing interface to 
work around bad data layout. Sending a column of blocs, yes, that's 
regular stride and it makes a lot of sense. Sending non-contiguous 
irregular structure ? As we used to say in France, $100 and a chocolate 
bar with that ?

Oh, BTW, I would gut MPI-IO and make a separate interface. Only a small 
subset of applications use it and the core semantics are quite different 
that pure message passing. Man, it's not MPI, it's emacs...

>> e) get rid of one-sided communications: if someone is serious about 
>> it, it uses something like ARMCI or UPC or even low level vendor 
>> interfaces.
> 
> 
> Instead, I propose to rework the MPI one-sided communications for a more 
> simple and flexible semantic. The current definition does not match 
> todays network capabilities, but was designed to allow a simple 
> implemenentation for slow/non-RDMA networks.

I don't know about that. I just would took it out of the Message Passing 
Interface because it's not message passing. There would certainly be a 
need for a pure RMA interface, and there is already a lot of existing 
work and experience to build upon.

>> Rob, you are politically connected, could you make it happen, please ?
>> :-)
> 
> 
> One person alone can't do this. The best place to discuss such things is
> the MPI users group meeting (EuroPVM/MPI, this year in Capri/Italy).

Nothing that radical would ever come out of EuroPVM/MPI (I heard that 
Capri is a really nice place, I will definitively beg my boss) or any 
other users group.

> Also, adding mpi.h to the standard to define an ABI is a good thing.

Just achieving that would be beyond my greatest expectations. It would 
certainly be fun to watch. We could organize fist fights on the beach in 
Capri...

Patrick
-- 

Patrick Geoffray
Myricom, Inc.
http://www.myri.com