[Beowulf] Why is communication so expensive for very small messages?

Jonathan Boyle j.boyle at manchester.ac.uk
Tue Apr 24 08:54:49 PDT 2007

I apologise if this is a naive question, but I'm new to this world of 

I'm using C++/mpi, to get a feel for communication costs I ran tests using 
mpptest and my own programs.

For 2 processor blocking calls, mpptest indicates a latency of about 30 

However when I measure communication times in my own program using a loop as 
start = MPI_Wtime();
for (unsigned t=1; t<=5000; t++)
 if (my_rank==0)
  MPI_Send(data, size, MPI_INT, 1, tag, MPI_COMM_WORLD);
  MPI_Recv(data, size, MPI_INT, 0, tag, MPI_COMM_WORLD, &status);
end = MPI_Wtime();

for size>=4, I get a latency of about 30 microseconds as expected, however for 
size<4, communication costs increase massively, and latency now appears to be 

Firstly, I assume this isn't normal?

Secondly, can anyone suggest what's going on, or where I can go for more 

Many thanks.

We're using mpich.

Processors are Intel(R) Xeon(TM) CPU 3.60GHz.

Interconnects are Dell PowerConnect 5324 24-port gigabit switches.

More information about the Beowulf mailing list