Instrumenting a parallel code

[Jared Hodge]

We are working on instrumenting a parallel finite element code we have
been working on.  We want to analyze how the system is being utilized,
and produce reliable, and quantifiable results so that in the future
we'll know if clusters designed for this code should be designed to
optimize network bandwidth, CPU speed, memory size, or other variables. 
Basically what we want to do is measure performance degradation as each
of these decreases (and vise versa).  This won't give us absolute
numbers for all variables, since there will obviously be plateaus in
performance, but it's a start.  Here's what I've got in mind for each of
these areas, please let me know if you have any suggestions.

Memory
	This one is pretty easy as far as memory size.  We can just launch
another application that will allocate a specific amount of memory and
hold it (with swap off of course).  I'm not sure if adjusting and
measuring memory latency is feasible or too great a concern.


Network
	We're writing a series of wrapper functions for the MPI calls that we
are using that will time their execution.  This will give us a good
indication of the blocking nature of communication in the program.

CPU usage
	I'm really not sure how we can decrease this one easily other than
changing the bus multiplier in hardware.  A timeline of CPU usage would
at least give us a start (like capturing top's output), but this would
alter the performance too (invasive performance monitor).  We could just
use the network measurements and assume that whenever a node is not
communicating or blocked for communication, it's "computing", but that
is definitely an over simplification.

Any useful comments or suggestions would be appreciated.  Thanks.
-- 
Jared Hodge
Institute for Advanced Technology
The University of Texas at Austin
3925 W. Braker Lane, Suite 400
Austin, Texas 78759

Phone: 512-232-4460
Fax: 512-471-9096
Email: Jared_Hodge at iat.utexas.edu