[Beowulf] SPEC CPU 2006 released

[Vincent Diepeveen]

If you want to objectively test yourself, you download some open source 
software,
go compile it for the different processors and benchmark how fast the 
programs are.

Keep into account then that you need to really be objective as some programs 
depend a lot
upon some simple driver of the graphics systems, to give a trivial example. 
Then you need
to get it deterministically to work for 1 core, which happens for very few 
software applications
too. Then you need to know something about compilers, how to compile 
something such that
you obtain a fast executable. Knowledge on how to get things done with 
compilers and the
word 'tester' are 2 that usually do not go hand in hand.

But more importantly, find me 1 tester that wants to do that much effort.

ZERO will do it.
ZERO out of all the billion persons on planet earth.

For those cpu manufacturers buying a license from spec is quite easy.

I'm not aware how much they need to pay. Couple of tens of thousands or so?

Spec pays out like 2000 dollar a few times to motivate developers.

In case of Sjeng i didn't catch that money, but GCP did (we appointed it 
like that
as he started the program and did do the majority of the work on the 
program).

Bottomline is that spec has something the important guys (the manufacturers) 
can test for and compare themselves with
and cpu designers can design new cpu's for, and we all can objectively look 
at the results they produce and see what is clearly
fastest.

Vincent

----- Original Message ----- 
From: "Geoff Jacobs" <gdjacobs at gmail.com>
To: "Vincent Diepeveen" <diep at xs4all.nl>
Cc: <beowulf at beowulf.org>
Sent: Saturday, August 26, 2006 1:01 AM
Subject: Re: [Beowulf] SPEC CPU 2006 released


> Vincent Diepeveen wrote:
>> The programs inside spec are open source.
>>
>> For example 458.sjeng is the free sjeng 11.2 version,
>> which we gave a 150MB hashtable and fixed a few old bugs which the
>> compiler guys noticed past years when testrunning specint2006.
>>
>> Lucky there were more bugs in compilers than in the code to be fixed.
>>
>> It's simply nearly impossible to license for example my diep's source
>> code to
>> spec. Not because i would mind those compiler guys have my diep's source
>> code,
>> as it will only have a positive effect that their compiler gets fast for
>> my code.
>>
>> But competitors will be able to simply buy a cheap spec license and see
>> my source code.
>>
>> As a result only non-commercial code will show up in spec.
>
> Man, that's what lawyers are for.
>
>> That's the only disadvantage of spec, and it is a non-solvable problem
>> in gametree search.
>>
>> So apart from this single problem, and the 2 year unnecessary delay to
>> create the testsuite,
>> what they do is pretty good idea in terms of benchmarking.
>>
>> Someone without ties to a company must make a benchmark for systems, and
>> let's face it.
>> 95% of all persons on this list are totally subjective, some because
>> they are paid by some company
>> and therefore will only see that company or one of its business partners
>> as best, and again
>> others are still believing too much in the past where the hardware field
>> advances quickly.
>>
>> So someone must do objective testing with software of what is faster.
>>
>> Then you get a compiler problem. So you need source code.
>>
>> In all that spec is doing a reasonable good job compared to others.
>>
>> Find me 1 site that 'tests' hardware that's objective. Spec is the best
>> compromise.
>
> I should have been more specific and said that I do not believe any
> integrated benchmarks packaged and shipped by SPEC are in any WSOF Open
> Source or Free Software. Apparently some software included in SPECCpu
> 2006 is in fact licensed under the GPL, including sjeng. Possibly some
> other Free Software and OSS licenses are represented as well.
>
> To use the benchmark itself requires a license from SPEC.
>
> -- 
> Geoffrey D. Jacobs
>
> Go to the Chinese Restaurant,
> Order the Special
>