[Beowulf] MPI programming question: Interleaved MPI_Gatherv?
Robert G. Brown
rgb at phy.duke.edu
Thu Mar 3 05:20:57 PST 2005
On Thu, 3 Mar 2005, Joachim Worringen wrote:
> Michael Gauckler wrote:
> > Dear List,
> > I would like to gather the data from several processes.
> > Instead of the comonly used stride, I want to interleave
> > the data:
> > Rank 0: AAAAA -> ABCDABCDABCDABCDABCD
> > Rank 1: BBBBB ----^---^---^---^---^
> > Rank 2: CCCCC -----^---^---^---^---^
> > Rank 3: DDDDD ------^---^---^---^---^
> > Since the stride of the receive type is indicated
> > in multpiles of its mpi_type, no interleaving is
> > possible (the smallest striping factor leads to
> > AAAAABBBBBBCCCCCDDDDD).
> > Is there a way to achieve this behaviour in an
> > elegant way, as MPI_Gather promises it? Or do
> > I need to do Send/Recv with self-aligned offsets?
What about RMA-like commands? MPI_Get in a loop? Since that is
controlled by the gatherer, one would presume that it preserves call
order (although it is non-blocking).
Or of course there are always raw sockets... where you have complete
control. Depending on how critical it is that you preserve this strict
> Actually, I don't see an 'elegant' way to do this, either. The decision
> between multiple MPI_Gatherv() calls and a Irecv/Send/Waitall construct
> depends on the quality of the MPI implementation you use (MPI_Gatherv
> can be optimized well for small amounts of data), the characteristics of
> you interconnect (high latency gives more room for optimization) and the
> number of processes you use. For small process numbers, you wont see
> much of a difference anyway.
> You could also try to gather all data on the root in separate buffers,
> and then let this process send/recv to itself using the proper datatypes.
> Finally, if this communication is not a significant part of your
> runtime, you shouldn't spend much time optimizing it anyway.
Robert G. Brown http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
More information about the Beowulf