! type fast indeed:p<br><br>Thank you for the detailed explanation.<br><br>I'll have to look more into the processing we're doing and it's requirements before proceeding:)<br>Your information has be extremely helpful:) <br>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Nov 4, 2012 at 7:42 AM, Mark Hahn <span dir="ltr"><<a href="mailto:hahn@mcmaster.ca" target="_blank">hahn@mcmaster.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks, infoative: p<br>
I'll consider your advice.<br>
<br>
If i read correctly, it seems the answer to the question about programming<br>
was: yes, a program must be written to accommodate a cluster. Did i get you<br>
right?<br>
</blockquote>
<br></div>
it depends what you mean. if you have a program which is written<br>
so that it can be run from a script, then a cluster can immediately<br>
let you run lots of them. if you're expecting a cluster to speed<br>
up a single instance, then you'll probably be disappointed.<br>
<br>
in short, clustering doesn't speed up any of the computers in the cluster.<br>
it just makes it more convenient to get multiple computers working.<br>
if you want multiple computers to work on the same program, then someone has to make it happen: divide up the work so each computer<br>
and put together the results.<br>
<br>
suppose you're trying to detect a particular face in all your images.<br>
you could have once machine searching an image, then going onto the next.<br>
basically, that one node is running a simple scheduler that runs jobs:<br>
lookforface face.png image0.png<br>
lookforface face.png image1.png<br>
lookforface face.png image2.png<br>
...<br>
<br>
if you want, you can divide up the work - send every other image to a second machine. in general, this would mean that a scheduler reads from that same list and dispatches one line (job) at a time to any<br>
node that isn't already busy. when a job completes, that node gets another job, and eventually all the work is done.<br>
<br>
"embarassingly parallel" just means you have enough images to keep all your<br>
machines busy this way.<br>
<br>
if you don't have that many images, you might want to try to get more than<br>
one machine working on the same image. a simple way to do that would be to (imaginarily) divide each image into, say, quadrants, so 4 machines can<br>
work on the same image (each getting a quarter of the image - with some<br>
overlap so targets along the border don't get missed.) to be specific,<br>
your list of jobs could be like this:<br>
lookforface face.png image0.png 0<br>
lookforface face.png image0.png 1<br>
lookforface face.png image0.png 2<br>
lookforface face.png image0.png 3<br>
lookforface face.png image1.png 0<br>
lookforface face.png image1.png 1<br>
...<br>
where 'lookforface' only looks for the face in the specified quadrant of the input image. the most obvious problem with this approach is that<br>
1-quadrant search may take too little time relative to the overhead of setting up each job. which includes accessing face.png and image0.png,<br>
even if only a quadrant of the latter is used. in general, this kind of issue is called "load balance", and is really the single most fundamental issue in HPC.<br>
<br>
if you wanted to pursue this direction, you could optimize by reducing<br>
the cost of distributing the images. if image0.png is quite large,<br>
then access through a shared filesystem might be efficient (if the FS block size is comparable to 1/2 the width of one image row.) if image0.png<br>
is smaller, then you could distribute that information "manually" by running<br>
a job which reads the image one one node and distributes quadrants to other nodes. the obvious way to do this would be via MPI, which is pretty friendly to matrices like decompressed images. this could even<br>
operate on pieces smaller than a quadrant - in fact, you could divide the work however finely you like. though as before, divide it too fine, and the<br>
per-chunk overhead dominates your cost, destroying efficiency.<br>
<br>
note that this refinement has merely changed who/how the work is being divided and data being communicated. in the simple case, work was divided<br>
at the command/job/scheduler level and data transmitted by file. the more fine-grained approach has subsumed some scheduling into your program, and<br>
is communicating the data explicitly over MPI.<br>
<br>
basically, someone has to divide up work, and data has to flow to where it's<br>
used. you could take this further: a single MPI program that runs on all nodes of the cluster at once and distributes work among MPI ranks. this<br>
would be the most programming effort, but would quite possibly be the most efficient. often, the amount of time needed to perform one unit of work<br>
is not constant - this can cause problems if your division of labor is too<br>
rigid. (consider the MPI-searches-4-quadrants approach: if one quadrant takes very little time, then the CPU associated with that quadrant will be twiddling its thumbs while the other quadrants get done.)<br>
<br>
I have, of course, completely fabricated this whole workflow. it becomes more interesting when the work has other dimensions - for instance, if you<br>
are searching 1M images for any of 1k faces. or if you are really hot to use a convolution approach so will be fourier-transforming all the images<br>
before performing any matching. or if you want to use GPUs, etc.<br>
<br>
TL;DR it's a good thing I type fast ;)<br>
<br>
in any case, your first step should be to look at the time taken to get<br>
inputs to a node, and then how long it takes to do the computation.<br>
life is easy if setup is fast and compute is long. that stuff is far more<br>
important than choosing a particular scheduler or cluster package.<br>
<br>
regards, mark hahn.<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
? 2012-11-4 ??6:11?"Mark Hahn" <<a href="mailto:hahn@mcmaster.ca" target="_blank">hahn@mcmaster.ca</a>>???<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
I am currently researching the feasibility and process of establishing a<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
relatively small HPC cluster to speed up the processing of large amounts<br>
of<br>
digital images.<br>
<br>
</blockquote>
<br>
do you mean that smallness is a goal? or that you don't have a large<br>
budget?<br>
<br>
After looking at a few HPC computing software solutions listed on the<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Wikipedia comparison of cluster software page (<br>
</div><a href="http://en.wikipedia.org/wiki/**Comparison_of_cluster_software" target="_blank">http://en.wikipedia.org/wiki/*<u></u>*Comparison_of_cluster_<u></u>software</a><<a href="http://en.wikipedia.org/wiki/Comparison_of_cluster_software" target="_blank">http://en.wikipedia.<u></u>org/wiki/Comparison_of_<u></u>cluster_software</a>>) I still have<div class="im">
<br>
only a rough understanding of how the whole system works.<br>
<br>
</div></blockquote><div class="im">
<br>
there are several discrete functionalities:<br>
- shared filesystem (if any)<br>
- scheduling<br>
- intra-job communication (if any; eg MPI)<br></div>
- management/provisioning/**<u></u>monitoring of nodes<div><div class="h5"><br>
<br>
IMO, anyone who claims to have "best practices" in this field is lying.<br>
there are particular components that have certain strengths, but none of<br>
them are great, and none universally appropriate. (it's also common<br>
to conflate or "integrate" the second and fourth items - for that matter,<br>
monitoring is often separated from provisioning.)<br>
<br>
1. Do programs you wish to use via HPC platforms need to be written to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
support HPC, and further, to support specific middleware using parallel<br>
programming or something like that?<br>
<br>
</blockquote>
<br>
"middleware" is generally a term from the enterprise computing environment.<br>
it basically means "get someone else to take responsibility for hard bits",<br>
and is a form of the classic commercial best practice of CYA. from an HPC<br>
perspective, there's the application and everything else. if you really<br>
want, you can call the latter "middleware", but doing so is uninformative.<br>
<br>
HPC covers a lot of ground. usually, people mean jobs will execute in a<br>
batch environment (started from a commandline/script). OTOH HPC sometimes<br>
means what you might call "personal supercomputing", where an interactive<br>
application runs in a usually-dedicated cluster (shared clusters tend to<br>
have scheduling response times that make interactive use problematic.)<br>
(shared clusters also give rise to the single most important value of<br>
clusters: that they can interleave bursty demand. if everyone in your<br>
department shares a cluster, it can be larger than any one group can<br>
afford, and therefore all groups will be able to burst to higher capacity.<br>
this is why large, shared clusters are so successful. and, for that<br>
matter,<br>
why cloud services are successful.)<br>
<br>
you can do HPC with very little overhead. you will generally want a shared<br>
filesystem - potentially just a NAS box or existing server. you may not<br>
bother with scheduling at all - let users pick which machine to run on,<br>
for instance. that sounds crazy, but if you're the only one using it, why<br>
bother with a scheduler? HPC can also be done without inter-job<br>
communication - if your jobs are single-node serial or threaded, for<br>
instance. and you may not need any sort of management/provisioning,<br>
depending on the stability of your nodes, environment, expected lifetime,<br>
etc.<br>
<br>
in short, slapping linux onto a few boxes, set up ssh keys or hostbased<br>
trust, have one or more of them NFS out some space, and you're cooking.<br>
<br>
OR<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Can you run any program on top of the HPC cluster and have it's workload<br>
effectively distributed? --> How can this be done?<br>
<br>
</blockquote>
<br>
this is a common newbie question. a naive program (probably serial or<br>
perhaps<br>
multithreaded) will see no benefit from a cluster. clusters are just plain<br>
old machines. the benefit comes if you want throughput (jobs per time) or<br>
specifically program for distributed computation (classically with MPI).<br>
it's common to use infiniband to accelerate this kind of job (as well as<br>
provide the fastest possible IO.)<br>
<br>
2. For something like digital image processing, where a huge amount of<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
relatively large images (14MB each) are being processed, will network<br>
<br>
</blockquote>
<br>
the main question is how much work a node will be doing per image.<br>
<br>
suppose you had an infinitely fast fileserver and gigabit connected nodes:<br>
transferring the image would take 10-15ms, so you would ideally spend<br>
about the same amount of time processing an image. but in this case, you<br>
should probably ask whether you can simply store images on the nodes in the<br>
first place. if you haven't thought about where the inputs are and how<br>
fast they<br>
can be gotten, then that will probably be your bottleneck.<br>
<br>
speed, or processing power be more of a limiting factor? Or would a<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
gigabit<br>
network suffice?<br>
<br>
</blockquote>
<br>
how long does a prospective node take to complete one work unit,<br>
and how long does it take to transfer the files for one?<br>
your speedup will be limited by whatever resource saturates first<br>
(possibly your fileserver.)<br>
<br>
3. For a relatively easy HPC platform what would you recommend?<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
</blockquote>
<br>
they are all crap. you should try not to spend on crap you don't need,<br>
but ultimately it depends on how much expertise you have and/or how much<br>
you value your time. any idiot can build a cluster from scratch using<br>
fundamental open-source components, eventually. but if said idiot has to<br>
learn filesystems, scheduling, provisioning, etc from scratch, it could<br>
take quite a while. when you buy, you are buying crap, but it's crap<br>
that may save you some time.<br>
<br>
don't count on commercial support being more than crappy.<br>
<br>
you should probably consider using a cloud service - this is just<br>
commercial<br>
outsourcing - more crap, but perhaps of value if, for instance, you don't<br>
want to get your hands dirty hosting machines (amazon), etc.<br>
<br>
anything commercial in this space tends to be expensive. the license to<br>
cover a crappy scheduler for a few hundred nodes, for instance will be<br>
pretty<br>
close to an FTE-year. renting a node from a cloud provider for a year<br>
costs<br>
about as much as buying a new node each year, etc.<br>
<br>
Again, I hope this is an ok place to ask such a question, if not please<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
</blockquote>
<br>
this is the place. though there are some fringe sects of HPC who tend to<br>
subsist on more and/or different crap (such as clusters running windows.)<br>
beowulf tends towards the low-crap end of things (linux, open packages.)<br>
<br>
regards, mark hahn.<br>
<br>
</div></div></blockquote>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
operator may differ from spokesperson. <a href="mailto:hahn@mcmaster.ca" target="_blank">hahn@mcmaster.ca</a><br>
</font></span></blockquote></div><br><br clear="all"><br>-- <br><img src="http://img.photobucket.com/albums/v202/CrashOveride/sijir-1.gif"><br><div></div><div></div><div></div><br>
</div>