[Beowulf] Digital Image Processing via HPC/Cluster/Beowulf - Basics

[Lux, Jim (337C)]

From: CJ O'Reilly <supaiku at gmail.com<mailto:supaiku at gmail.com>>
Date: Saturday, November 3, 2012 3:47 PM
To: Mark Hahn <hahn at mcmaster.ca<mailto:hahn at mcmaster.ca>>
Cc: "beowulf at beowulf.org<mailto:beowulf at beowulf.org>" <beowulf at beowulf.org<mailto:beowulf at beowulf.org>>
Subject: Re: [Beowulf] Digital Image Processing via HPC/Cluster/Beowulf - Basics


Thanks, infoative: p
I'll consider your advice.

If i read correctly, it seems the answer to the question about programming was: yes, a program must be written to accommodate a cluster. Did i get you right?


>> You got that right…  But bear in mind that for your task (whatever it is), someone might have written most of the pieces you need already.  If you're using some computationally intensive utility (finite element modeling or raytraced graphics, for instance) as the underpinnings of your problem may already be cluster-aware.

But Mark's comments are very true.. In general, there is NO turnkey solution and whatever is out there will be fine for some parts of your problem and a pain for others.  So spending a bit of time figuring out what it is you are trying to do, and what the parallelization/HPC parts are is worth it.  No point in a flexible multi-user resource allocation system with fancy schedulers and job pre-emption if you're the only user of the box, for instance.

It might be worth building a "toy" cluster with, say, 4 nodes working against a file server, and fooling around a bit with workloads like the one you are planning to get a feel for it.  Don't go for performance, but try to understand how your workload can be divided up, and what the information flows are (lots of node to node, or very little? .. Shared disk gets hit all the time?)

There are a variety of cluster in a box things out there to get started (I hesitate to suggest any, because they may not exist any more)  (back when, I tried ClusterMatic, and Rocks.)   It really doesn't matter what you use, because as Mark points out, it probably is pretty clunky in some ways, but by experiencing the clunkyness, you'll instantly become more expert.  And worst case, you've spent a week of your life doing it.

Really, a week's playing around can be invaluable.  (I wonder if people offer short courses on this.. It might be useful for people where the manager comes in and says, my boss said we should look at putting X on a cluster, can you write up a white paper in a month to lay it all out)


http://www.clustermonkey.net/  might be a decent resource on putting together a low end cluster

Check out their projects and getting started sections..


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20121104/0eae6682/attachment.html>