[Beowulf] copying big files

[Jan Heichler]

Hallo Henning,

Freitag, 8. August 2008, meintest Du:

HF> Hi everybody,


HF> One needs basically a daemon which handles copying requests and establishes
HF> the connection to next node in the chain.

Why a daemon? Just MPI that starts up the processes on the remote nodes during programm startup. Advantage is that you can use any high-speed-interconnect which you have an MPI for. 

HF> Has somebody written such a tool?

I wrote a tool in Java in 2004 or 2005 during an internship for IBM at the IPK in Gatersleben that implemented your strategy. Worked good but had some flaws since i was using some Java specific remote procedure calls and daemons. Implementing it with C/C++ and MPI can't take more than a couple of hours and should be easy to do. 

You could use tar and pipe the datastream to copy whole directories without worrying to much about file attributes. I always wanted to implement it myself but i can't find time with my current job :-(

Could be a good project for a student to do.

Regards,
Jan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.scyld.com/pipermail/beowulf/attachments/20080808/f862522b/attachment.html