Hi,<br>Gus--thank you.<br>You are right. I mainly have to run programs on a small cluster (GiGE) dedicated for my job only; and sometimes I might get some opportunity to run my code on a shared cluster with hundreds of nodes.<br>
<br>My parallel CFD application involves (In its simplest form):<br>1) reading of input and mesh data from few files by the master process (I/O)<br>2) Sending the full/respective data to all other processes (MPI Broadcast/Scatter)<br>
3) Share the values at subdomains (created by Metis) boundaries at the end of each iteration (MPI Message Passing)<br>4) On converge, send/Gather the results from all processes to master process<br>5) Writing the results to files by the master process (I/O).<br>
<br>So I think my program is not I/O intensive; so the Funneling I/O through the master process is sufficient for me. Right?<br><br>But now I have to parallelize a new serial code, which plots the results while running (online/live display). Means that it shows the plots of three/four variables (in small windows) while running and we see it as video (all progress from initial stage to final stage). I assume that this time much more I/O is involved. At the end of each iteration result needs to be gathered from all processes to the master process. And possibly needs to be written in files as well (I am not sure). Do we need to write it on some file/s for online display, after each iteration/time-step? <br>
<br>I think (as serial code will be displaying result after each iteration/time step), I should display result online after 100 iterations/time-steps in my parallel version so less "I/O" and/or "funneling I/O through master process" will be required.<br>
Any opinion/suggestion?<br><br>regards,<br>Amjad Ali.<br><br><br><br><div class="gmail_quote">On Wed, Jul 1, 2009 at 5:06 AM, Gus Correa <span dir="ltr"><<a href="mailto:gus@ldeo.columbia.edu">gus@ldeo.columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Bogdan, list<br>
<br>
Oh, well, this is definitely a peer reviewed list.<br>
My answers were given in the context of Amjad's original<br>
questions, and the perception, based on Amjad's previous<br>
and current postings, that he is not dealing with a large cluster,<br>
or with many users, and plans to both parallelize and update his code from F77 to F90, which can be quite an undertaking.<br>
Hence, he may want to follow the path of least resistance,<br>
rather than aim at the fanciest programming paradigm.<br>
<br>
In the edited answer that context was stripped off,<br>
and so was the description of<br>
"brute force" I/O in parallel jobs.<br>
That was the form of concurrent I/O I was referring to.<br>
An I/O mode which doesn't take any precautions<br>
to avoid file and network contention, and unfortunately is more common<br>
than clean, well designed, parallel I/O code (at least on the field I work).<br>
<br>
That was the form of concurrent I/O I was referring to<br>
(all processors try to do I/O at the same time using standard open/read/write/close commands provided by Fortran or another language,<br>
not MPI calls).<br>
<br>
Bogdan seems to be talking about programs with well designed parallel I/O instead.<div class="im"><br>
<br>
Bogdan Costescu wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Wed, 24 Jun 2009, Gus Correa wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
the "master" processor reads... broadcasts parameters that are used by all "slave" processors, and scatters any data that will be processed in a distributed fashion by each "slave" processor.<br>
...<br>
That always works, there is no file system contention.<br>
</blockquote>
<br>
I beg to disagree. There is no file system contention if this job is the only one doing the I/O at that time, which could be the case if a job takes the whole cluster. However, in a more conventional setup with several jobs running at the same time, there is I/O done from several nodes (running the MPI rank 0 of each job) at the same time, which will still look like mostly random I/O to the storage.<br>
<br>
</blockquote>
<br></div>
Indeed, if there are 1000 jobs running,<br>
even if each one is funneling I/O through<br>
the "master" processor, there will be a large number of competing<br>
requests to the I/O system, hence contention.<br>
However, contention would also happen if all jobs were serial.<br>
Hence, this is not a problem caused by or specific from parallel jobs.<br>
It is an intrinsic limitation of the I/O system.<br>
<br>
Nevertheless, what if these 1000 jobs are running on the same cluster,<br>
but doing "brute force" I/O through<br>
each of their, say, 100 processes?<br>
Wouldn't file and network contention be larger than if the jobs were<br>
funneling I/O through a single processor?<br>
<br>
That is the context in which I made my statement.<br>
Funneling I/O through a "master" processor reduces the chances of file<br>
contention because it minimizes the number of processes doing I/O,<br>
or not?<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Another drawback is that you need to write more code for the I/O procedure.<br>
</blockquote>
<br>
I also disagree here. The code doing I/O would need to only happen on MPI rank 0, so no need to think for the other ranks about race conditions, computing a rank-based position in the file, etc.<br>
<br>
</blockquote>
<br></div>
>From what you wrote,<br>
you seem to agree with me on this point, not disagree.<br>
<br>
1) Brute force I/O through all ranks takes little programming effort,<br>
the code is basically the same serial,<br>
and tends to trigger file contention (and often breaks NFS, etc).<br>
2) Funneling I/O through the master node takes a moderate programming<br>
effort. One needs to gather/scatter data through the "master" processor, which concentrates the I/O, and reduces contention.<br>
3) Correct and cautious parallel I/O across all ranks takes a larger programming effort,<br>
due to the considerations you pointed out above.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
In addition, MPI is in control of everything, you are less dependent on NFS quirks.<br>
</blockquote>
<br>
... or cluster design. I have seen several clusters which were designed with 2 networks, a HPC one (Myrinet or Infiniband) and GigE, where the HPC network had full bisection bandwidth, but the GigE was a heavily over-subscribed one as the design really thought only about MPI performance and not about I/O performance. In such an environment, it's rather useless to try to do I/O simultaneously from several nodes which share the same uplink, independent whether the storage is a single NFS server or a parallel FS. Doing I/O from only one node would allow full utilization of the bandwidth on the chain of uplinks to the file-server and the data could then be scattered/gathered fast through the HPC network. Sure, a more hardware-aware application could have been more efficient (f.e. if it would be possible to describe the network over-subscription so that as many uplinks could be used simultaneously as possible), but a more balanced cluster design would have been even better...<br>
<br>
</blockquote>
<br></div>
Absolutely, but the emphasis I've seen, at least for small clusters<br>
designed for scientific computations in a small department or research group is to pay less attention to I/O that I had the chance to know about.<br>
When one gets to the design of the filesystems and I/O the budget is already completely used up to buy a fast interconnect for MPI.<br>
I/O is then done over Gigabit Ethernet using a single NFS<br>
file server (often times a RAID on the head node itself).<br>
For the scale of a small cluster, with a few tens of nodes or so,<br>
this may work OK,<br>
as long as one writes code that is gentle with NFS<br>
(e.g. by funneling I/O through the head node).<br>
<br>
Obviously the large clusters on our national labs and computer centers<br>
do take into consideration I/O requirements, parallel file systems, etc. However, that is not my reality here, and I would guess it is<br>
not Amjad's situation either.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
[ parallel I/O programs ] always cause a problem when the number of processors is big.<br>
</blockquote>
<br>
</blockquote>
<br></div>
Sorry, but I didn't say parallel I/O programs.<br>
I said brute force I/O by all processors (using standard NFS,<br>
no parallel file system, all processors banging on the file system<br>
with no coordination).<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I'd also like to disagree here. Parallel file systems teach us that a scalable system is one where the operations are split between several units that do the work. Applying the same knowledge to the generation of the data, a scalable application is one for which the I/O operations are done as much as possible split between the ranks.<br>
</blockquote>
<br></div>
Yes.<br>
If you have a parallel file system.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
IMHO, the "problem" that you see is actually caused by reaching the limits of your cluster, IOW this is a local problem of that particular cluster and not a problem in the application. By re-writing the application to make it more NFS-friendly (f.e. like the above "rank 0 does all I/O"), you will most likely kill scalability for another HPC setup with a distributed/parallel storage setup.<br>
<br>
</blockquote></div>
Yes, that is true, but may only be critical if the program is I/O<br>
intensive (ours are not).<br>
One may still fare well with funneling I/O through one or a few<br>
processors, if the program is not I/O intensive,<br>
and not compromise scalability.<br>
<br>
The opposite, however, i.e.,<br>
writing the program expecting the cluster to<br>
provide a parallel file system,<br>
is unlikely to scale well on a cluster<br>
without one, or not?<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Often times these codes were developed on big iron machines, ignoring the hurdles one has to face on a Beowulf.<br>
</blockquote>
<br>
Well, the definition of Beowulf is quite fluid. Nowadays is sufficiently easy to get a parallel FS running with commodity hardware that I wouldn't associate it anymore with big iron.<br>
<br>
</blockquote>
<br></div>
That is true, but very budget dependent.<br>
If you are on a shoestring budget, and your goal is to do parallel<br>
computing, and your applications are not particularly I/O intensive,<br>
what would you prioritize: a fast interconnect for MPI,<br>
or hardware and software for a parallel file system?<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
In general they don't use MPI parallel I/O either<br>
</blockquote>
<br>
Being on the teaching side in a recent course+practical work involving parallel I/O, I've seen computer science and physics students making quite easily the transition from POSIX I/O done on a shared file system to MPI-I/O. They get sometimes an index wrong, but mostly the conversion is painless. After that, my impression has become that it's mostly lazyness and the attitude 'POSIX is everywhere anywhere, why should I bother with something that might be missing' that keeps applications at this stage.<br>
<br>
</blockquote>
<br></div>
I agree with your considerations about laziness and the POSIX-inertia.<br>
However, there is still a long way to make programs and programmers<br>
at least consider the restrictions imposed by network and file systems,<br>
not to mention to use proper parallel I/O.<br>
Hopefully courses like yours will improve this.<br>
If I could, I would love to go to Heidelberg and take your class myself!<br>
<br>
Regards,<br><font color="#888888">
Gus Correa</font><div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
</div></div></blockquote></div><br>