[Beowulf] SPEC CPU 2006 released

[Robert G. Brown]

On Sat, 26 Aug 2006, Vincent Diepeveen wrote:

> Find me 1 site that 'tests' hardware that's objective. Spec is the best 
> compromise.

I don't know about site testing hardware, but there are many objective
hardware tests on sites.  In particular, lmbench is an excellent and
unbiased toolset (in my personal belief, having communicated fairly
extensively with Larry and Carl, knowing e.g. that Linus uses lmbench
routinely to test and tune the linux kernel).  benchmaster (my own
microbenchmark suite, ex. cpu_rate) isn't finished, really, but it is
definitely objective.  netpipe is quite objective.  I've never heard
bonnie accused of being biased.  Stream is objective, and damn simple
code at that.  I really dunno about the new code being developed for
top500 testing, but I do trust to SOME extent the folks developing it --
the bias if any will be in that it is focussed on HPC and perhaps
certain classes of code WITHIN HPC.

Macro benchmarks are much tougher, as they are really in the category of
"is this program "like" my program" for the component(s) of the suite or
tool.  They also tend to be relatively "rich" code environments, where
compilers are tested as much or more than just the hardware.  This is
always true, I supposed, but it makes it difficult to separate
comparative analysis of different hardware from the compiler even when
the same compiler is used, especially across architectures.  Leaving
aside the issue of whether or not one SHOULD try to differentiate
compiler from the hardware.  Microbenchmarks are probably better for
that purpose as in many cases the core code fragments they time are
small enough that there isn't that much variation in their assembler
implementation, at any rate.

I am a longstanding, fairly passionate advocate of totally open (GPL)
benchmarking code.  I was one of the folks who talked John into opening
up lmbench and liberalizing its fairly restrictive original license,
pointing out that the synergy obtained by freeing the code was of
greater importance than the control he was trying to maintain to prevent
vendor abuse.  In the end, I don't think anyone has abused lmbench in
part because it has never become a major marketing tool (unlike certain
other benchmark suites I can name).

One of the things that has LONG irritated me is that SPEC isn't open
source, isn't extensible, isn't freely available.  I'm certain that
there are reasons for this -- they could even be "good" reasons.  That
doesn't mean I have to like it, or think that the world doesn't need a
truly open alternative.  However, this needs at least one human to "own"
the project of creating the alternative.  I myself cannot do it -- I
already own a languishing microbenchmark suite, a cluster monitoring
toolset, a random number tester, and an XML-based flashcard presentation
program, and none of them get enough attention as it is.  If somebody
DID put one together and own it, though, I'd contribute, help, clap
loudly, cheer.

    rgb

>
>
> Vincent
>
>
> ----- Original Message ----- From: "Geoff Jacobs" <gdjacobs at gmail.com>
> To: <beowulf at beowulf.org>
> Sent: Friday, August 25, 2006 9:01 PM
> Subject: Re: [Beowulf] SPEC CPU 2006 released
>
>
>> Mark Hahn wrote:
>>>> The new SPEC cpu suite has been released, and initial results
>>>> are posted to:
>>> 
>>> is this one still stupidly not open-source?
>> 
>> I don't think SPEC has any open source benchmarks.
>> 
>> As far as I can tell, SPEC groups derive income by licensing their
>> benchmarks to integrators and such (although I know SpecViewPerf is free
>> to use). I would think they also have issues with proprietary 3rd party
>> software within their benchmarks.
>> 
>> I think they could be supported in the same manner as OSDL, but they are
>> not, so no open source from SPEC.
>> 
>> -- 
>> Geoffrey D. Jacobs
>> 
>> Go to the Chinese Restaurant,
>> Order the Special
>> 
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit 
>> http://www.beowulf.org/mailman/listinfo/beowulf
>> 
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu