[Beowulf] What now?

Wed Aug 18 16:25:25 PDT 2004

My imaging software is very simple and embarassingly parallel.  It is 
mostly of the type one would use on landsat imagery.
I have convolution filters , histograms, difference filters, constrast 
range enhancement filters, and still need to work out my image 
resampling code and  a cannny filter.  I also plan to write a JPG filter 
to remove the annoying 8X8 pixilization inherent in JPG. This is an 
ongoing thing. No real schedule, no particluar motivation, other than 
fun.  I just kind of do it as I feel the need. There were two instances 
where I wanted to take a better look at some images from one of the Mars 
Rovers.  In these two cases , I used my filters in combination with 
shell scripts to enhance these images to better see the details I was 
interested in. Other than serving as a testbed, this is where the mini 
cluster has earned it's keep.  It did such a good job of it that I was a 
bit surprised when I saw the results, and spent the rest of the night 
fiddling with the filters out of sheer fascination.

 One of the things I studied in college was remote sensing, so I had 
written a bit of software. I wanted a simple project to get myself 
started in parallel programming, so I rewrote my old code in C, using 
MPI, in a very coarse-grained fashion.My programs split the processing 
of the data set between the cpu's, then reassembles the separately 
processed sections after filtering.  I got a good speedup, and delighted 
myself in being able to chew through large data sets much faster than I 
can with my "old" single cpu P4 machine.

Not only did I get a speedup, but writing my own parallel code taught me 
that parallel programming is not as scarey and difficult as it may seem 
on the surface. Once you get used to the idea of it, doing things in 
parallel is not more difficult conceptually than "ordinary" programming. 
 There are pitfalls, and oddities, but you pick up on them really fast. 
Parallel programming is really kind of neat.

Glen

Benjamin W. Mayer wrote:

>I am a grad student in CS at the University of Minnesota. I have just
>gotten into digital photography and namely making large panoramas. Being
>that I work at a HPC center I am thinking about making this into a
>parallel operation.
>
>I am wondering what you do with your imaging software, or if it something
>more along the lines of POV-RAY.
>
>Ben
>
>
>On Wed, 18 Aug 2004, Glen Gardner wrote:
>
>  
>
>>Most of the commercially available software is aimed at numeric problem
>>solving (modelling).  Matlab, IDL, Mathcad, and a host of others.
>>Usually, after a beowulf cluster is isntalled, a burn-in and testing
>>phase begins in order to get an understanding of what that particular
>>machine is good at, and what problems remain to be solved in order to
>>get the most out of it. Some people have had some fun running a cluster
>>as a pile of pc's using seti at home.
>>
>>For many people, the machine itself is frequently the object of
>>attention, more than the software it runs.  In that case you might want
>>to spend your time tying differnet hardware and system configurations.
>> For others, speed is where it is at. I might suggest acquiring several
>>benchmarks and running them on the computer and writing some parallel
>>programs in C or Fortran to learm MPI and to get a feel for what the
>>machine is (and is not) good for.
>>
>>If you don't program in C,  C++, Fortran, Java , or Python, then you
>>probably need to warm up to at least one of them as your favorite
>>programming language so you can put your machine to work. Learning about
>>the importance, sillyness, and annyoing inconsistencies of popular
>>compilers and benchmarks on a Beowulf is not trivial , and can be
>>entertaining, to say the least.
>>
>>Right now, there is a lot of interest in gridded applications for
>>parallel machines , so installing globus toolkit might prove useful and
>>interesting if you are into that sort of thing. Alternativly, don't rule
>>out writing your own gridding and networking software, as the "rules"
>>for gridded applications have yet to be written. Be inventive, and have
>>fun. In any case you will be writing some code and some shell scripts.
>>It is standard fare on clusters.
>>
>> There is a great need for good console and graphical user interfaces
>>for MPI on Linux/FreeBSD, with console mode being the current "best"
>>popular user interface for Beowulf running MPI.  If you like to code and
>>have an idea, you should pursue it. People are interested in parallel
>>computing, but the lack of a sane user interface, and the custom nature
>>of beowulf means that we have a very long way to go before parallel
>>computing becomes a mainstream platform. This also means that there is a
>>huge niche open to those who wish to fill it.
>>
>>I had a conversation with a young fellow who told me he had a cluster,
>>of nearly 30 old pc's in the basement of his parents house, which had
>>been calculating pi for the last 6 months. He designed the project, and
>>managed to scrounge up all the hardware and make it work for next to
>>zero dollars invested.  A remarkable thing , and quite an achevement
>>(kudos to the very supportive parents of that youngster).
>>
>>I have been in a similar situation as you. I built a 12 node beowulf for
>>the purpose of learning parallel computing, and once I had it built, I
>>spent a short time wondering what I'd do with it.  The machine was the
>>goal, but once I was there, I wanted to put it to work.  I began testing
>>the cluster and identified a couple of bottlenecks I felt could be
>>easily fixed. I eventually upgraded the hardware to 14 nodes and added
>>improved networking and a separate I/O node to address these issues, and
>>improved the performance measurably. After a while I decided it was time
>>to stop "fixing" things and just put the machine to work (tweaking a
>>beowulf is like peeling an onion, it never really ends).
>>
>>Now my little cluster spends time as a test platform for things I don't
>>want to annoy users with on the larger clusters at work. When it is not
>>doing that , it runs  imaging software which I have written for my own
>>use. Overall, it has proven to be quite useful for me in terms of
>>learning, work, and just having fun with computing.
>>
>>Whatever it is that fascinates you about parallel computing, you should
>>pursue it.  Just follow your interests.
>>
>>
>>Glen Gardner
>>
>>
>>Jack C wrote:
>>
>>    
>>
>>>Hello all,
>>>
>>>I'm sorry to ask a question like this, but I'm seriously stuck. I'ver
>>>installed MPI just because the idea of clustering excites me. But...
>>>what now? What do all of you "veteran" Beowulf people do with yours? I
>>>mean, are there GPL/etc packages, or code that anyone knows of? Mabye
>>>primes, anything. (I tried to write some small things, but it's going
>>>to take me many more months before I can effectivly design an
>>>efficient program to utilize MPI)
>>>
>>>Thanks for you input!
>>>
>>>-Jack C
>>>jack {at} crepinc.com
>>>http://www.crepinc.com
>>>
>>>_______________________________________________
>>>Beowulf mailing list, Beowulf at beowulf.org
>>>To change your subscription (digest mode or unsubscribe) visit
>>>http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>>      
>>>
>>--
>>Glen E. Gardner, Jr.
>>AA8C
>>AMSAT MEMBER 10593
>>
>>
>>
>>http://members.bellatlantic.net/~vze24qhw/index.html
>>
>>
>>
>>_______________________________________________
>>Beowulf mailing list, Beowulf at beowulf.org
>>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>>
>>    
>>
>
>  
>

-- 
Glen E. Gardner, Jr.
AA8C
AMSAT MEMBER 10593

http://members.bellatlantic.net/~vze24qhw/index.html