data transfer and condor

[Thomas R Boehme]

Hi,

Sorry, I guess I didn't read your post good enough. Reading rgb's answer, I
realized that the throughput is probably not your actual problem (well, with
sever 100s of GB, I guess it will still be a problem).

If your are only looking for a way to hand out the chunks to the jobs and
the throughput is not the problem, you might want to take a look at
MW for Condor http://www.cs.wisc.edu/condor/mw/ It is a C++ class library
which is designed for Master-Worker applications and uses condor.
It should make it fairly easy to write a Master-Worker application the way
you need it.

Bye, Thommy

> -----Original Message-----
> From: Steven Berukoff [mailto:steveb at aei-potsdam.mpg.de]
> Sent: Friday, August 03, 2001 8:36 AM
> To: beowulf at beowulf.org
> Subject: data transfer and condor
> 
> 
> Hi all,
> 
> We're looking at using Condor on our cluster for its checkpointing and
> job-handling abilities, as the routines we're running don't require much
> in the way of internode communication.  We have an NFS file server which
> contains our entire fileset (something on the order of 100s of GB), a
> master node for the cluster, and several nodes.  Outside of Condor, our
> algorithm requires that each of the nodes get some subset of the data (on
> the order of perhaps 100MB) and runs the analysis code on this data
> segment.  Obviously, each node must gather its share of the data from the
> NFS file server; this of course requires a large amount of network
> traffic.
> 
> Does anyone have a clever idea about scheduling the data
> transfers so that it is accomplished in a reasonable fashion?  We were
> hoping Condor provides this functionality to some degree, but it doesn't
> seem to.
> 
> Thanks
> Steve
> 
> 
> 
> =====
> Steve Berukoff					tel: 49-331-5677233
> Albert-Einstein-Institute			fax: 49-331-5677298
> Am Muehlenberg 1, D14477 Golm, Germany
email:steveb at aei.mpg.de
> 
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf