[Beowulf] copying big files (Henning Fehrmann)
atchley at myri.com
Sun Aug 10 05:02:52 PDT 2008
On Aug 10, 2008, at 7:57 AM, Scott Atchley wrote:
> You may want to look at http://loci.cs.utk.edu. If you need to
> distribute large files within a cluster or across the WAN, you can
> use the LoRS tools to stripe the file over multiple servers and the
> clients then try pulling blocks off of each server in parallel.
> Using Internet2 and one client at Vanderbilt and a couple servers at
> Univ of Tennessee, they were able to saturate UT's ~400 Mb/s I2 link
> (much to the disbelief of the Vandy IT staff). I have seen ~5 Gb/s
> within a cluster using good 10G NICs. :-)
I forgot to mention LoRS optionally uses MD5 for checksums and AES-128
for encryption (you can use either, both or neither).
The stored file is represented by a XML file called an exNode. If you
want to share the data, you can email the exNode to someone and they
can then download the data. You control the download offset and length
so that you can extract just the parts of the file that you want. I
believe there is a NetCDF version that can use exNodes and there may
be a HDF5 version as well.
More information about the Beowulf