[Beowulf] Parallel Programming Question

[Bogdan Costescu]

On Tue, 30 Jun 2009, Gus Correa wrote:

> My answers were given in the context of Amjad's original questions

Sorry, I missed somehow the context for the questions. Still, the 
thoughts about I/O programming are general in nature, so they would 
apply in any case.

> Hence, he may want to follow the path of least resistance, rather 
> than aim at the fanciest programming paradigm.

Heh, I have the impression that most scientific software is started 
like that and only if it's interesting enough (f.e. survives the first 
generation of PhD student(s) working on/with it) and gets developed 
further it has some "fancyness" added to it. ;-)

> Nevertheless, what if these 1000 jobs are running on the same 
> cluster, but doing "brute force" I/O through each of their, say, 100 
> processes? Wouldn't file and network contention be larger than if 
> the jobs were funneling I/O through a single processor?

The network connection to the NFS file server or some uplinks in an 
over-subscribed network would impose the limitation - this would be a 
hard limitation: it doesn't matter if you divide a 10G link between 
100 or 100000 down-links, it will not exceed 10G in any case; in 
extreme cases, the switch might not take the load and start dropping 
packets. Similar for a NFS file server: it certainly makes a 
difference if it needs to serve 1 client or 100 simultaneously, but 
beyond that point it won't matter too much how many there are (well, 
that was my experience anyway, I'd be interested to hear about a 
different experience).

> Absolutely, but the emphasis I've seen, at least for small clusters 
> designed for scientific computations in a small department or 
> research group is to pay less attention to I/O that I had the chance 
> to know about. When one gets to the design of the filesystems and 
> I/O the budget is already completely used up to buy a fast 
> interconnect for MPI.

That's a mistake that I have also done. But one can learn from own 
mistakes or can learn from the mistakes of others. I'm now trying to 
help others understand that the cluster is not only about CPU or MPI 
performance, but about the whole, including storage. So, spread the 
word :-)

>> >  [ parallel I/O programs ] always cause a problem when the number of 
>> >  processors is big.
> Sorry, but I didn't say parallel I/O programs.

No, that was me trying to condense your description in a few words to 
allow for more clipping - I have obviously failed...

> The opposite, however, i.e., writing the program expecting the 
> cluster to provide a parallel file system, is unlikely to scale well 
> on a cluster without one, or not?

I interpret your words (maybe again mistakenly) as a general remark 
and I can certainly find cases where the statement is false. If you 
have a well thought-out network design and a NFS file server that can 
take the load, a good scaling could still be achieved - please note 
however that I'm not necessarily referring to a Linux-based NFS file 
server, an "appliance" (f.e. from NetApp or Isilon) could take that 
role as well although at a price.

> If you are on a shoestring budget, and your goal is to do parallel 
> computing, and your applications are not particularly I/O intensive, 
> what would you prioritize: a fast interconnect for MPI, or hardware 
> and software for a parallel file system?

A balanced approach :-) It's important to have a clear idea of what 
"not particularly I/O intensive" actually means and how much value the 
users give for the various tasks that would run on the cluster.

> Hopefully courses like yours will improve this. If I could, I would 
> love to go to Heidelberg and take your class myself!

Just to make this clearer: I wasn't teaching myself; based on Uni 
Heidelberg regulations, I'd need to hold a degree (like Dr.) to be 
allowed to do teaching. But no such restrictions are present for the 
practical work, which is actually the part that I find most 
interesting because it's more "meaty" and a dialogue with students can 
take place ;-)

-- 
Bogdan Costescu

IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8240, Fax: +49 6221 54 8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de