[Beowulf] Re: Re: Home beowulf - NIC latencies

[Rob Ross]

On Mon, 14 Feb 2005, Joachim Worringen wrote:

> Rob Ross wrote:
> > Making a sequence of MPI_Isends followed by a MPI_Wait go faster than a 
> > sequence of MPI_Sends isn't hard, particularly if the messages are to 
> > different ranks.  I would guess that every implementation will provide 
> > better performance in the case where the user tells the implementation 
> > about all these concurrent operations and then MPI_Waits on the bunch.
> 
> In this case, the user should think about MPI_Alltoall(v) - there are
> MPI implementations which do this in a smarter way than
> Isend/Irecv/Waitall to achieve much better performance than using the
> naive approach. Especially if you go to large process numbers, some
> coordination can help a lot, even for a full bisection network like a
> single-stage full crossbar...

Yes!  We don't see nearly enough of this I think.

> Generally, collectives are there to let the library know what kind of 
> communication is coming next. All speculations in the library based on 
> monitoring and predicting non-collective communication will probably 
> only do good in the matching micro-benchmark (my personal experience).

I agree.  Which is why we don't tend to try to figure out what the user is 
trying to do, and instead just implement an algorithm to get things done 
as quickly as we can.

Rob