[Beowulf] copying big files

[Lombard, David N]

On Fri, Aug 08, 2008 at 08:37:13AM -0700, Henning Fehrmann wrote:
> Hi everybody,
> 
> Coping a big file onto all nodes in a cluster is a rather common problem.
> I would have thought that there might be a standard tool for
> distributing the files in an efficient way. So far, I haven't found one.
> 
> Assuming one has a network design which allows non blocking full duplex
> wire-speed connections between N/2 pairs of nodes where N is the number
> of nodes in the cluster. It is basically a non blocking coreswitch.
> 
> In this case the following scheme would be convenient and rather simple:
> 
> The file is placed on node n1 and one builds a chain of nodes n1 , n2 .... nN.

See Brent Chen's pcp at <http://www.theether.org/work.html>
You'll want pcp, authd, and libe. Get gexec while you're at it...

-- 
David N. Lombard, Intel, Irvine, CA
I do not speak for Intel Corporation; all comments are strictly my own.