[Beowulf] copying big files (Henning Fehrmann)

[Scott Atchley]

On Aug 10, 2008, at 7:57 AM, Scott Atchley wrote:

> You may want to look at http://loci.cs.utk.edu. If you need to  
> distribute large files within a cluster or across the WAN, you can  
> use the LoRS tools to stripe the file over multiple servers and the  
> clients then try pulling blocks off of each server in parallel.  
> Using Internet2 and one client at Vanderbilt and a couple servers at  
> Univ of Tennessee, they were able to saturate UT's ~400 Mb/s I2 link  
> (much to the disbelief of the Vandy IT staff). I have seen ~5 Gb/s  
> within a cluster using good 10G NICs. :-)
>
> Scott

I forgot to mention LoRS optionally uses MD5 for checksums and AES-128  
for encryption (you can use either, both or neither).

The stored file is represented by a XML file called an exNode. If you  
want to share the data, you can email the exNode to someone and they  
can then download the data. You control the download offset and length  
so that you can extract just the parts of the file that you want. I  
believe there is a NetCDF version that can use exNodes and there may  
be a HDF5 version as well.

Scott