[Beowulf] 8 ranks of DDR400 and Opterons (was: Tyan 2882)

[serguei.patchkovskii at sympatico.ca]

Mark Hahn wrote:
: > * Mem: 8*1GB PC3200 (DDR 400) ECC reg.; Corsair/Samsung CM72SD1024RLP-3200/SB: >  ( 12 nodes have 8*2GB)
: 
: this dimm is 2-rank, I believe; corsair's datasheet is pretty lame. 
: that means that each bank of memory is 4x2=8 ranks. that's definitely
: pushing the limit; I'm sure it can be done in some cases, 

Mark is absolutely right. Running 8 ranks of DDR RAM per CPU at DDR400 is definitly pushing the limit, and could only be done at all on recent Opteron revisions. We had a very similar problem with a 4-socket Supermicro board and 8 ranks per socket of PNY memory - apparently random panics, lock-ups, and application errors.

It turned out that the DDR400 profile on the DIMMs was programmed with CAS 2.5 timing, and the BIOS was trusting enough to use it - even though Opterons are only spec'ed to support CAS 3.0 for DDR400 ECC modules. Fortunately for us, Supermicro BIOS supports very fine-grained control over memory timings - from just choosing the specific SPD profile to completely overriding the SPD. Either one of: a) downclocking the DIMMS to 183MHz; or b) overriding the CAS to 3.0 gave us a rock-stable system - but for a while this was extremely puzzling ....

Serguei