[Beowulf] X5500

[Bill Broadley]

Vincent Diepeveen wrote:
> I wouldn't bet at registered-ecc DDR3 ram to become cheaper.
> To be honest i misjudged that for DDR reg-ecc ram also,
> it still is relative spoken expensive.

I've heard this a dozen times, seems repeated quite often.  Yet when I
actually look I see it either so small as to be inconsequential or even negative.

6GB ram for a dual nehalem system:
http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Order=BESTMATCH&Description=DDR3-1066+ECC&x=0&y=0
$96.99 / 6 = $16.165 per GB

4 GB ram
http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Order=BESTMATCH&Description=2GB+DDR2-800+ECC&x=0&y=0
$109.99 / 4 = $27.49 per GB

Sure the later is fully buffered and the former is not, but registered isn't
required for up to 6 dimms in a dual socket.

At silicon mechanics a dual nehalem upgrade from 12GB of DDR-1333 -> 24GB of
(swapping 2GB dimms for 4GB
DDR-1333 = $348 / 12 = $29 per GB.

For an opteron system to swap 2GB dimms for 4 = $760 /16 = $47.50 per GB.

Where is the horror of the high DDR3 prices?

> DDR2 ecc ram on other hand is so dirt cheap,

Previous comparisons for desktops showed about a $25 difference for your
average new highend desktop with 4-8GB ram.  Hardly something worth noting IMO.

> that you really can make nodes with a lot of ram really cheap.

Indeed, a dual xeon E5530 with 24GB ram is around $3k, which I think is very
reasonable.

> Both amd as well as intel have boards where you can put in easily 64GB ram,
> some even you could go to 128GB ram.

Indeed, the above mentioned xeon E5530 with 48GB ram is $3700.

> Latency of DDR3 is not really good cmopared to DDR2. Of course this
> depends upon how you use
> the RAM.

Not really good?  I've seen a decent improvement using the normal, my plat, or
lmbenchs.

> That i7 can deliver the weird amount of 192 bytes at a time or so,
> versus dual channel ddr2 can deliver 64 at a time.

Er, well the newer AMD chips can return 64 at a time on 2 channels, and the
new intel is 64 at a time from 3.  Are you arguing aesthetics or performance?
 Considering that often the intel get's a fair bit more performance with 8
threads vs 4 it seems pretty reasonable to add another channel to maintain the
performance vs memory bandwidth or IMO more important performance per random
memory reference.

> Now that's a huge bandwidth difference of course, some i7's even get to
> 18GB/s bandwidth, versus 10GB/s for the DDR2

I've seen 22GB/sec on a single socket, it did require using pathscale's
compiler.  The dual sockets are of course better... unlikely the previous
generation dual socket intels which were often SLOWER than the desktops.

> equivalent; yet that means that the latency of the DDR2 quadcores is
> simply better. More bytes at a time comes at a big latency

I don't follow, do you have numbers?

> price simply.
> 
> If you're just having software that streams and hardly has something to
> do for the cpu's to calculate while you stream,
> then maybe consider rewriting the algorithm to something more complex
> that needs to stream less and can do more calculations,
> as doing massive calculations at just a few gigabytes of RAM is what
> gpu's are genius at and will scale perfectly for, maybe
> even outperforming the law of more there for a short period of time in
> number of instructions a cycle you can push through in total
> at a single gpu node.

Certainly performance vs the previous generation varies, but it seems that
even cache friendly stuff is often 5-10% faster and that's before you consider
running 8 threads instead of 4.  If you are doing random lookups or anything
bandwidth intensive it's even better.  Once you look at dual sockets (xeon
55xx) vs intel the difference is even bigger.

So when you look at the performance, the price, and the power usage it's a
pretty compelling improvement vs the previous intel.  I'm not sure if the AMD
systems have price adjusted as a result yet but I expect them to.  Now it's
AMDs turn, maybe they will ship some CPUs that take more than 75 watts now
that Intel seems to have finally shipped something competitive.