[Beowulf] Large Dell, odd IO delays

[Gus Correa]

On 02/15/2018 02:04 AM, John Hearns via Beowulf wrote:
> Hmmm...  I will also chip in with my favourite tip
> Look at the sysctl for min_free_kbytes    It is often set very low.
> Increase this substantially. It will do no harm to your system (unless 
> you set it ti an absurd value!)
> 
> You should be looking at the vm dirty ratios etc. also

+1
vm.dirty_background_bytes
vm.dirty_bytes
(or the corresponding _ratios)
vm.min_free_kbytes
Defaults are low.
Increasing them improved a lot our compute nodes IO.
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-tunables

> 
> On 15 February 2018 at 00:44, Kilian Cavalotti 
> <kilian.cavalotti.work at gmail.com 
> <mailto:kilian.cavalotti.work at gmail.com>> wrote:
> 
>     On Wed, Feb 14, 2018 at 2:26 PM, David Mathog <mathog at caltech.edu
>     <mailto:mathog at caltech.edu>> wrote:
>     > Checked the hugepage settings and found a difference there.  The two systems
>     > that don't do this have  /sys/kernel/mm/redhat_transparent_hugepage/defrag
>     >
>     > always madvise [never]
>     >
>     > whereas the system with the issue has:
>     >
>     > [always] madvise never
> 
>     THP defragmentation is definitely something that has bitten us in the
>     past, when under memory pressure, and we now default to [madvise]
>     pretty much everywhere (we're too timid to disable it entirely).
> 
>     A good way to see if that's really the issue is to "echo never >
>     /sys/kernel/mm/redhat_transparent_hugepage/defrag" while the problem
>     is happening, while simultaneously monitoring the processes with htop,
>     for instance.
>     It's usually pretty instant:  if the issue is really with THP defrag,
>     then CPU usage for your stalling process should drop pretty much
>     immediately and things go back to normal.
> 
>     Cheers,
>     --
>     Kilian
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>     <http://www.beowulf.org/mailman/listinfo/beowulf>
> 
> 
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>