[Beowulf] choosing a high-speed interconnect

[Bill Broadley]

I'm glossing over many details, but in general I've found the below
mentioned strategies a good first order approximation.

I'd suggest taking several representative production runs and try graph
the performance on 1,2,4,8,16,32 processors or whatever is feasible for
your jobs and cluster.

If you see good scaling I.e. each jump gets almost twice as much work
done you very likely will not benefit from a faster interconnect.

If you do not see good scaling then you might be bottlenecked by latency
or bandwidth, or possibly other factors like a faster than linear increase
in work with extra nodes, and disk I/O performance among others.

Hope for the first, it will save you money, time and effort.  If it's
the later then it would be worth your while to try to find out exactly
why your code isn't scaling.  Even the simplest measures can help,
for instance recording the before and after packet counts as reported
by ifconfig.  Graphing how the number of packets increases with N and
how the performance scales with N might provide valuable insight.

Another dirty hack can be to force your interfaces to 100 Mbit and
see how the performance changes.  If it's minimal it's likely to be
either latency (100 Mbit and GigE usually don't vary by much) or not
bandwidth constrained.

Also something like ganglia can provide you with a significant amount of
additional info for a run, so you can watch memory, network, load, memory
used, buffers used etc.  See how these variables change with the timestep
and with the number of nodes can be very helpful for getting a general
idea of how your job is behaving.  One particular job I was running had
network traffic increasing with each iteration, above a certain point
the wall clock time per timestep increased.  Calculations showed I was
getting 30% of peak GigE performance, it is likely that between the
packet overhead and MPI overhead that was as fast as I was likely to see.

Certainly none of the above will give you as good as an idea as a source
code analysis or a fully profiled run, but they can help steer you in
the right direction.

-- 
Bill Broadley
Computational Science and Engineering
UC Davis