[Beowulf] copying big files (Henning Fehrmann)

[jitesh dundas]

Hi,

We could try and implement this functionality of resuming broken
downloads like in some softwares like Download Accelerator and
bit-torrent.

I hope my views can help, so here goes:-

When a file is being downloaded, we can keep a stack of all of these
downloads in progress at a centralized repository, preferably where
the user has kept his file hosted for download or on the machine where
the download is to be done.

Next, we can keep the track of the point at which the download stopped
and store it in the repository. Next, if the user tries to start the
download it again, we can again retrieve it back from the data and get
the end point of the previous download.

The end point for each file can include the file details in terms of
bits and bytes( 0 & 1) or even in percentages or pieces..Next time we
can break our file based on pieces or percentages( as needed) and
start the download from the nearest point that is best suited for the
user.

I hope this helps...
I request your feedback...

Thanks,
Jitesh Dundas
Mobile- +91-9860925706
http://jiteshbdundas.blogspot.com


On 8/9/08, Carsten Aulbert <carsten.aulbert at aei.mpg.de> wrote:
> Hi
>
> Perry E. Metzger wrote:
>
>> Is there a reason bittorrent isn't suited to this application?
>>
>
> Our investigations so far showed that bittorrent is only good if the
> files to be transferred fit well into main memory. If you exceed about
> 90-95% of the RAM your disks will be accessed a lot and the performance
> breaks down a lot (we have seen close to wirespeed in the beginning and
> in the end we were crawling with mere few 10 kByte/s).
>
> Cheers
>
> Carsten
>
> --
> Dr. Carsten Aulbert - Max Planck Institute for Gravitational Physics
> Callinstrasse 38, 30167 Hannover, Germany
> Phone/Fax: +49 511 762-17185 / -17193
> http://www.top500.org/system/9234 | http://www.top500.org/connfam/6/list/31
>