[Beowulf] Optimal BIOS settings for Tyan K8SRE

[stephen mulcahy]

Hi,

Sure. We have a head node which acts as an NFS server for the diskless
compute nodes in the cluster. On the head node, I had to use the
following BIOS settings to ensure the OS could see the full 4GB of
physical memory installed,


Main
	Installed O/S Linux

        Memory Hole
                4GB Memory Hole Adjust  Auto
                4GB Memory Hole Size    1024 MB
                IOMMU   Enabled
                Memhole mapping Software

        MTRR Mapping    Discrete

On the diskless nodes, if I used the above settings - the driver for the
network card wasn't loaded (I guess the probing failed due to the
network device not being visible with the above memory hole config - but
I'm a bit fuzzy on the details of what happens here and whether this is
h/w or driver dependent). Some further experimentation gave the
following configuration which sees 4GB of memory available, *and* a
working network driver (which is pretty useful for diskless compute
nodes :)

Main
	Installed O/S Linux

        Memory Hole
                4GB Memory Hole Adjust  Manual
                4GB Memory Hole Size    768 MB
                IOMMU   Enabled
                Size    32 MB
                Memhole mapping Hardware

        MTRR Mapping    Discrete

I'm not sure what the performance difference is between the 2 above and
whether there would be an advantage to changing the head node to also
use hardware memhole mapping.

The kernel in both cases are stock Debian kernels (mostly using standard
options, some config changes to allow diskless NFS booting but nothing
memory related).

-stephen

Ivan Paganini wrote:
> Can you post what were the tweaking that you had undergone to access the
> 4GB?  Thank you.
> 
> On 8/31/06, *stephen mulcahy* < smulcahy at aplpi.com
> <mailto:smulcahy at aplpi.com>> wrote:
> 
>     Hi,
> 
>     I'm maintaining a 20-node cluster of Tyan K8SREs (4GB RAM, dual Opteron
>     270s) which are being used primarily for Oceanographic modelling (MPICH2
>     running on Debian/Linux 2.6 kernel).
> 
>     I had to make some tweaks to make all 4GB of RAM visible to the OS.
> 
>     I'm now at the point where I'm considering a pass at performance tuning
>     the system. Before I start on OS level tuning, I'm trying to figure out
>     whether there are any performance improvements to be had from tweaking
>     BIOS settings, particularly those relating to Memory and ECC. I have a
>     reasonable conceptual understanding of what the various settings are
>     doing
>     (and have glanced at the AMD BIOS developers guide for reference)
>     but I am
>     very unclear on what the potential performance impact of any of these
>     settings are. Does anyone have any general advice or pointers to good
>     reference information on this?
> 
>     My current settings are,
> 
>     Hammer Configuration
>             HT-LDT Frequency        Auto
>             Dual-Core Enable        Enabled
>             ECC Features
>                     ECC     Enabled
>                     ECC Scrub Redirection   Enabled
>                     Dram ECC Scrub CTL      Disabled
>                     Chip-Kill       Disabled
>                     DCACHE ECC Scrub CTL    Disabled
>                     L2 ECC Scrub CTL        Disabled
> 
>             Memory Hole
>                     4GB Memory Hole Adjust  Manual
>                     4GB Memory Hole Size    768 MB
>                     IOMMU   Enabled
>                     Size    32 MB
>                     Memhole mapping Hardware
> 
>             Memory Config
>                     Swizzle Memory Banks    Enabled
>                     DDR clock jitter        Disabled
>                     DDR Data Transfer Rate  Auto
>                     Enable all memory clocks        Populated
>                     Controller config mode  Auto
>                     Timing config mode      Auto
>             AMD PowerNow!   Disabled
>             Node Memory Interleave  Auto
>             Dram Bank Interleave    Auto
>             GART Error Reporting    Disabled
>             MTRR Mapping    Discrete
> 
> 
>     Any comments on those welcome.
> 
>     Thanks,
> 
>     -stephen
> 
> 
>     --
>        Stephen Mulcahy    Applepie Solutions Ltd   http://www.aplpi.com
>       Unit 30, Industry Support Centre, GMIT, Dublin Rd, Galway, Ireland.
> 
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org <mailto:Beowulf at beowulf.org>
>     To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
> 
> 
> 
> 
> -- 
> -----------------------------------------------------------
> Ivan S. P. Marin
> Laboratório de Física Computacional
> lfc.ifsc.usp.br <http://lfc.ifsc.usp.br>
> Instituto de Física de São Carlos - USP
> ----------------------------------------------------------

-- 
Stephen Mulcahy, Applepie Solutions Ltd, Innovation in Business Center,
   GMIT, Dublin Rd, Galway, Ireland.      http://www.aplpi.com