[Beowulf] Optimizations for smaller scale linux clusters

[Christopher Samuel]

On 20/07/15 23:47, Trevor Gale wrote:

> I am building a small scale cluster (most likely around 8-10 nodes).
> I was wondering if anyone has any general optimizations that they
> like to implement on their clusters.

These are the sysctl settings we push out to our x86 clusters, one set
for GPFS tuning (set by our on-site IBM staff over the past few years),
one set to disable the OOM killer by getting malloc() to fail when
there's no memory for it to work and one to disable zone_reclaim_mode
for reasons previously discussed on the list (though checking it looks
like RHEL6 disables it by default these days):

::::::::::::::
gpfs.sysctl
::::::::::::::
#
# These are changes to the kernel configuration recommend by GPFS
#
# Memory
#
vm.min_free_kbytes = 370560
#
# Communications
#
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 262144 8388608
net.ipv4.tcp_wmem = 4096 262144 8388608
net.core.netdev_max_backlog = 2500
net.ipv4.tcp_mtu_probing = 1

::::::::::::::
oom.sysctl
::::::::::::::
# Stop nodes OOM'ing
vm.overcommit_memory = 2
vm.overcommit_ratio = 99

::::::::::::::
zone_reclaim_mode.sysctl
::::::::::::::
# Disable zone_reclaim_mode as it can cause massive
# performance problems with mmap()'d IO on GPFS
# (amongst other things).  See RT#10793.

vm.zone_reclaim_mode = 0


-- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci