[Beowulf] Optimal BIOS settings for Tyan K8SRE
ballen at gravity.phys.uwm.edu
Mon Sep 4 17:49:28 PDT 2006
PS: be sure to use the 'mcelog' utility and package to monitor for ECC
errors. If you have a large number of nodes this will help to identify
flaky memory and cpus with cache memory issues.
On Mon, 4 Sep 2006, stephen mulcahy wrote:
> Hi Bruce,
> Do you have any idea what the performance impact from enabling scrubbing
> is on your systems? did you do any before/after benchmarking?
> Bruce Allen wrote:
>> On Sun, 3 Sep 2006, Mark Hahn wrote:
>>>> ECC Features
>>>> ECC Enabled
>>>> ECC Scrub Redirection Enabled
>>>> Dram ECC Scrub CTL Disabled
>>>> Chip-Kill Disabled
>>>> DCACHE ECC Scrub CTL Disabled
>>>> L2 ECC Scrub CTL Disabled
>> You can find our systems BIOS/ECC/Scrub settings here:
>> Our systems are Supermicro H8SSL-i motherboards, with a
>> Serverworks/Broadcom HT1000 chipset and a single Opteron 175 (dual core,
>> 2.2 GHz).
>> The ECC part is:
>> DRAM ECC Enable = Enabled
>> MCA DRAM ECC Logging = Enabled
>> DRAM Scrub Redirect = Enabled
>> DRAM BG Scrub = 2.62ms
>> L2 Cache BG Scrub = 84.00ms
>> Data Cache BG Scrub = 84.00ms
>> Scrubbing is done one cache line (64) bytes at a time. Thus with 2GB of
>> memory and DRAM background scrub interval of 2.62ms we will scrub the
>> entire memory in approximately:
>> 2 GB/64 Bytes * 2.62 ms = 2^31 / 2^6 * 2.62 ms = 87912 secs
>> So our choices correspond to one complete scrub of DRAM per day. Our
>> settings scrub the L2 cache more often: about once every half hour.
>> Just modify the calculation above, using 1MB instead of 2GB, and 84 ms
>> instead of 2.62 ms. One finds that the L2 cache is scrubbed about once
>> every 1376 seconds (every 23 minutes).
More information about the Beowulf