[Beowulf] Cell

[Vincent Diepeveen]

At 05:41 PM 4/27/2005 -0400, Robert G. Brown wrote:
>> If you can deliver 1 processor that can do 1 tflop, there is no need for
>> bandwidth anymore, everything happens on that chip in such a case :)
>
>???
>
>Perhaps I'm highly confused, but I thought vector units operated on
>DATA, and that that data had to live in MEMORY.
>
>Memory, in turn, has some fairly stringent limits on the rate at which
>it can be accessed -- very definitely finite in terms of both latency
>and bandwidth.  Furthermore, for streaming vector-like operations in at
>least NORMAL PC-like architectures (as opposed to supercomputer
>architectures where a Lot Of Money (LOM) is spent feeding the processor
>I would expect memory bandwidth to be the fundamental bottleneck in
>streaming/vector operations.
>
>To be more explicit -- if you run stream on most reasonably modern
>systems, stream "copy" takes a time that is very comparable to to stream
>"scale".  For example, on a P4 I have handy:
>
>rgb at lucifer|B:1100>./benchmaster -t 2 -s 1000000 -i 1 -n 10
>avg full = 1.497222e+07 min = 1.492613e+07  max = 1.513106e+07
>Content-Length: 1067
>
><?xml version="1.0"?>
><benchml>
>  <version>Benchmaster 1.1.2</version>
>  <hostinfo>
>    <hostname>lucifer</hostname>
>    <vendor_id> GenuineIntel</vendor_id>
>    <CPU name> Intel(R) Pentium(R) 4 CPU 1.80GHz</CPU name>
>    <CPU clock units="Mhz"> 1804.520</CPU clock>
>    <l2cache units="KB"> 512 KB</l2cache>
>    <memtotal units="KB">515840</memtotal>
>    <memfree units="KB">105536</memfree>
>    <nanotimer>cpu cycle counter nanotimer</nanotimer>
>    <nanotimer_granularity units="nsec">98.158</nanotimer_granularity>
>  </hostinfo>
>  <benchmark>
>    <name>stream copy</name>
>    <command>./benchmaster</command>
>    <args>-t 2 -s 1000000 -i 1 -n 10</args>
>    <description>d[i] = a[i] (standard is -s 1000000 -i 1 -n
>10)</description>
>    <iterations>1</iterations>
>    <size>1000000</size>
>    <stride>1</stride>
>    <time units="nsec">1.50e+01</time>
>    <time_stddev units="nsec">1.80e-02</time_stddev>
>    <min_time units="nsec">1.49e+01</min_time>
>    <max_time units="nsec">1.51e+01</max_time>
>    <rate units="10e+6">1.07e+03</rate>
>  </benchmark>
></benchml>
>
>
>rgb at lucifer|B:1104>./benchmaster -t 3 -s 1000000 -i 1 -n 10
>avg full = 1.489518e+07 min = 1.486376e+07  max = 1.494210e+07
>Content-Length: 1075
>
><?xml version="1.0"?>
><benchml>
>  <version>Benchmaster 1.1.2</version>
>  <hostinfo>
>    <hostname>lucifer</hostname>
>    <vendor_id> GenuineIntel</vendor_id>
>    <CPU name> Intel(R) Pentium(R) 4 CPU 1.80GHz</CPU name>
>    <CPU clock units="Mhz"> 1804.520</CPU clock>
>    <l2cache units="KB"> 512 KB</l2cache>
>    <memtotal units="KB">515840</memtotal>
>    <memfree units="KB">105536</memfree>
>    <nanotimer>cpu cycle counter nanotimer</nanotimer>
>    <nanotimer_granularity units="nsec">98.118</nanotimer_granularity>
>  </hostinfo>
>  <benchmark>
>    <name>stream scale</name>
>    <command>./benchmaster</command>
>    <args>-t 3 -s 1000000 -i 1 -n 10</args>
>    <description>d[i] = xtest*d[i] (standard is -s 1000000 -i 1 -n
>10)</description>
>    <iterations>1</iterations>
>    <size>1000000</size>
>    <stride>1</stride>
>    <time units="nsec">1.49e+01</time>
>    <time_stddev units="nsec">8.47e-03</time_stddev>
>    <min_time units="nsec">1.49e+01</min_time>
>    <max_time units="nsec">1.49e+01</max_time>
>    <rate units="10e+6">1.07e+03</rate>
>  </benchmark>
></benchml>
>
>Note that these times are already identical.  The "scale" operation is
>done in parallel with I/O and takes "zero time".  Clearly the
>memory-to-CPU I/O is the bottleneck, not the processor, which is
>"infinitely fast" as far as this operation is concerned.
>
>Comparing triad:
>
>
>
>rgb at lucifer|B:1099>./benchmaster -t 5 -s 1000000 -i 1 -n 10
>avg full = 1.982203e+07 min = 1.980236e+07  max = 1.985822e+07
>Content-Length: 1081
>
><?xml version="1.0"?>
><benchml>
>  <version>Benchmaster 1.1.2</version>
>  <hostinfo>
>    <hostname>lucifer</hostname>
>    <vendor_id> GenuineIntel</vendor_id>
>    <CPU name> Intel(R) Pentium(R) 4 CPU 1.80GHz</CPU name>
>    <CPU clock units="Mhz"> 1804.520</CPU clock>
>    <l2cache units="KB"> 512 KB</l2cache>
>    <memtotal units="KB">515840</memtotal>
>    <memfree units="KB">105144</memfree>
>    <nanotimer>cpu cycle counter nanotimer</nanotimer>
>    <nanotimer_granularity units="nsec">98.244</nanotimer_granularity>
>  </hostinfo>
>  <benchmark>
>    <name>stream triad</name>
>    <command>./benchmaster</command>
>    <args>-t 5 -s 1000000 -i 1 -n 10</args>
>    <description>d[i] = a[i] + xtest*b[i] (standard is -s 1000000 -i 1
>-n 10)</description>
>    <iterations>1</iterations>
>    <size>1000000</size>
>    <stride>1</stride>
>    <time units="nsec">1.98e+01</time>
>    <time_stddev units="nsec">6.08e-03</time_stddev>
>    <min_time units="nsec">1.98e+01</min_time>
>    <max_time units="nsec">1.99e+01</max_time>
>    <rate units="10e+6">1.21e+03</rate>
>  </benchmark>
></benchml>
>
>shows a slowdown of roughly 1/3 relative to copy (2 ns per loop pass
>instead of 1.5 ns per loop pass). Again, on-chip parallelism and the
>efficiency of the memory interface are clearly exhibited.
>
>In other words, if you stuck a "1 TFLOP" vector processor into this
>particular MEMORY architecture and ran stream, I wouldn't expect to see
>much speedup.  The bulk of the time required for the actual numerical
>operations is already hidden by the memory bottleneck and design
>parallelism.  It isn't clear how much a larger cache would help, either,
>on streaming code that doesn't reuse any memory location -- there comes
>a point of diminishing returns where you're stuck at fundamental
>bandwidth again, waiting for the cache to refill, when the CPU is much
>faster.  I tend to think of large caches as being useful to people with
>lots of contexts to juggle more than a necessary advantage to people
>running LARGE vector operations, although they are an obvious advantage
>to folks whose vectors can fit into cache:-)
>
>> "If you were plowing a field, which would you rather use? Two strong oxen
>> or 1024 chickens?"
>
>I'd take the chickens any day, if each chicken could plow 1/1024th of
>the field faster than the two oxen could do the whole thing.
>
>And they only cost chickenfeed...;-)

The fundamental problem is you need a $2 million network for your 1024
chickens.

All i need with the virtual 1 tflop processor is some of that XDR memory,
add a few memory banks at the mainboard which are each independantly
connected to one of the 4 sides of the chip. 

Even if that XDR rambuss grossely expensive at say $1000 for 2 gigabyte, 

then i put in 4 dimms of it at each independant memory controller of the
chip, 
delivering a total bandwidth of 100 gigabyte a second.

Even a medium engineer can add on die memory controllers to a chip.

The extreme expensive mainboard i buy for $2500 and i feel ripped off.
The extreme expensive cell processor i buy for $3000 and i feel ripped off,

But even then at a total price of say $10k the system is eating your $3
million 1024 chicken system alive.

See my point?

Whatever price the 1024 chickens have, your network costs several million.

A 12288 chicken system from IBM costs to be precise 8 million euro,
delivering 27.5 tflop. Thanks to the network it is not a chicken system but
europese fastest supercomputer.

By the way, my chessprogram doesn't output 2 bytes, but it outputs 4 bytes
each few minutes. 

A chess move.

If you have a processor that can run it, just outputting 4 bytes each few
minutes, at a speed equal to a 1 Thz opteron processor, then i can sell
100000 machines of that hands down if its price is that of a normal PC.

At a 1 thz machine it will output d2d4 anyway the first move. In
correspondence notation 'd2' is the field 42 and 'd4' is 44.

So it's output is 42 anyway.

Vincent

>   rgb
>
>>   Seymour Cray
>> 
>> Vincent
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
>> 
>
>-- 
>Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
>Duke University Dept. of Physics, Box 90305
>Durham, N.C. 27708-0305
>Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
>
>
>
>