[Beowulf] Three notes from ISC 2006

[Kevin Ball]

> - *If* you feel you need to use such a new metric for whatever reason, you 
> should at least publish the benchmark that is used to gather these numbers to 
> allow others to do comparative measurements. This goes to Greg.

This has been done.  You can find the benchmark used for message rate
measurements at
http://www.pathscale.com/performance/InfiniPath/mpi_multibw/mpi_multibw.html

It is essentially a bandwidth test that can be run on multiple processes
per node simultaneously, and happens to report Message Rate as well as
bandwidth.

It also dynamically determines the N/2 bandwidth point, the other
benchmarking number PathScale has been pushing.

This is another area where we would be happy to see results on other
interconnects.  Feedback on ways to improve the benchmark would be good.

  In essence, we're trying to find ways that improve upon the ways that
clusters are evaluated, such that customers can better predict the
performance they will get.  All of these various metrics that we're
pushing (application performance, message rate, n/2) are attempts to
improve on the traditional 0-byte latency and 4 MB message bandwidth
metrics.

  Unfortunately, you can't always test *your* application ahead of time,
especially if its still under development;  hence microbenchmarks and
their continuing survival and broad usage.

  We're not (at least, Greg and I aren't... can't speak for Marketing)
convinced that we've found the best metrics out there in Message rate
and N/2... however, we do believe they add significant information about
interconnect performance, and give end users a better insight into what
they are buying.  If anyone on this list has suggestions for better
metrics, I'd be happy to hear them.  

-Kevin

> 
> But I don't think that Greg's "Real Appliation Performance" white paper is 
> infamous.  It states where the data comes from, you have to trust him for his 
> own numbers, and it does not directly link the differences in the application 
> performance to the messaging rate. Of course, it does not offer a scientific 
> analysis, and you can not compare it to papers like the ones from Leonid Oliker. 
> But I don't think it's unfair, and surely stimulates the competition for better 
> technical solutions or better white papers.