[Beowulf] copying big files

[Ashley Pittman]

On Fri, 2008-08-08 at 11:59 -0400, Joe Landman wrote:
> > The transmission of a single package is fast. The time of passing a particular
> > package through the whole chain of nodes is short compared with time of the 
> > entire copying process. E.g., using jumbo frames a package can have the size of ca 10kB.
> > In Gb network the transmission time of a single package between nodes is 
> > of the order of 0.1 ms.  Even in a cluster with 1024 nodes it takes
> > in an ideal case just 0.1s to pass a package from node n1 through all nodes to n1024.
> > 
> > On each node the package is stored and, in the end, one reassembles the file.
> > For big files (size >> 10Mb) the required time is approximately 
> > the same as one needs for copying the file between two nodes plus 0.1s.
> > 
> > One needs basically a daemon which handles copying requests and establishes 
> > the connection to next node in the chain.
> > 
> > Has somebody written such a tool?
> 
> I saw something like this several years ago.

I've written a couple of them over the years, one as a way of copying a
new OS between nodes and I started one more recently as a general
purpose copy tool.

I've always thought that something like this should be written in MPI
and simply make use of MPI_Bcast() and MPI_Addreduce() to do the hard
part, it's very easy to write a version that copies a single file,
slightly harder to do multiple files but still not rocket science.

Basically efficient broadcast isn't as easy to make generic as it seems,
why waste time even trying when you can get MPI to do all tricky bits
like work out toplogy/starting deamons/security and in all probability
do it better to boot?

Ashley Pittman.