[Beowulf] copying big files

[Bernard Li]

Hi:

On Tue, Aug 12, 2008 at 10:10 AM, Lombard, David N
<dnlombar at ichips.intel.com> wrote:

> See Brent Chen's pcp at <http://www.theether.org/work.html>
> You'll want pcp, authd, and libe. Get gexec while you're at it...

Dave!!  You beat me to mention pcp! :-)

I really wished somebody would pick up the project and make it better.
 I have tested it a few years back and it was really fast.  The
general idea is that the host that has the original file will send to
the neighbour and before that transfer completes the neighbour will
send to its neighbour and so on and so forth.  But it has some
shortcomings:

1) If one host in the path goes down, you need to start over again
2) The command option/interface is a little bit awkward, if I remember
correctly you need to specify all the hosts manually in the command
line
etc.

And yes, I agree BitTorrent is a simple solution that works well, that
is why we integrated BitTorrent as part of the distribution mechanism
of OS images in SystemImager.  However, it would be nice if someone
actually wrote an application which eliminates the manual setup of the
tracker, seed, etc. -- better yet, code something from scratch as
BitTorrent cannot handle user/file permissions, and thus the way
around it is to tar up the files you wanted transfer, and untar it
after, which adds additional overhead.

I look forward to trying out XGET, though.

Cheers,

Bernard