[Beowulf] copying big files
jan.heichler at gmx.net
Fri Aug 8 08:52:40 PDT 2008
Freitag, 8. August 2008, meintest Du:
HF> Hi everybody,
HF> One needs basically a daemon which handles copying requests and establishes
HF> the connection to next node in the chain.
Why a daemon? Just MPI that starts up the processes on the remote nodes during programm startup. Advantage is that you can use any high-speed-interconnect which you have an MPI for.
HF> Has somebody written such a tool?
I wrote a tool in Java in 2004 or 2005 during an internship for IBM at the IPK in Gatersleben that implemented your strategy. Worked good but had some flaws since i was using some Java specific remote procedure calls and daemons. Implementing it with C/C++ and MPI can't take more than a couple of hours and should be easy to do.
You could use tar and pipe the datastream to copy whole directories without worrying to much about file attributes. I always wanted to implement it myself but i can't find time with my current job :-(
Could be a good project for a student to do.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf