[Beowulf] SPEC CPU 2006 released

[Joe Landman]

Robert G. Brown wrote:

> This, in turn, was supposed to stimulate a discussion on whether or not
> benchmarks of this sort "need" to be licensed and controlled (I would
> argue a resounding "no") and if so controlled by whom, to prevent vendor

I agree.  Benchmarks, and all data around them are an example of a need 
for transparency.  Without such transparency, well, you never really 
know what you have.  Unfortunately benchmark design and reporting is 
rather suspect as well.  Anyone advocating taking a large 
multi-dimensional space and reducing it to a zero-dimensional quantity 
by means of a non-intuitive averaging scheme ought to think hard about 
the vast quantities of information being destroyed by such a process. 
IMO that is.

> abuse or benchmark drift or the healthy discussion, criticism, and
> modification that results in a true open source process.  This might
> have lead to discussing e.g. the top500 rankings and linpack and other
> (more open) macro benchmarks, whether the "member organizations" of SPEC

Heh... some of us have gotten together to try to get something 
reasonable (http://www.scalableinformatics.com/bbs), and completely open 
source created for LS benchmarking.  All tests, test cases, and code is 
OSS, and specifically the stuff being used by researchers is used.  We 
have had lots of people pull it down (over 1000) in the 2 years it has 
been out, but alas there seems to be little interest in the community to 
contributing test cases and code snippets, and lots of interest in 
simply consuming baseline results.  So work on the subsequent versions 
takes a long time as it is done in one persons "copious" spare time.

> are above reproach and have no self-interest to promote (Oh wait!  The
> member list, who doubtless pony up the actual cash that runs the
> non-profit, is made up of all the top COMPANIES that sell HARDWARE?  No
> hint of self-interest there...;-).

Hmmm.... we recently responded to a government RFP where they "require" 
runs on the hardware from the spec suite.

[soap box]

  This is counter productive IMO as the spec suite really doesn't do 
much for you in terms of meaningful performance measurements.  Does 
about the same as Linpack/HPL.  Yeah, some people argue otherwise, and 
compiler vendors and hardware OEMs/resellers work really hard to put out 
great benchmark data.  I don't believe such data is terribly meaningful, 
and customers who rely upon it for purchase decisions may wind up making 
decisions in part based upon "data" which has little intrinsic value to 
the tasks at hand.


The most important benchmark is the one that uses the same code you use 
in the way you are going to use it.  Anything else is an entropy 
generator.

[/soap box]


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615