[Beowulf] Large Dell, odd IO delays
gus at ldeo.columbia.edu
Thu Feb 15 08:29:12 PST 2018
On 02/15/2018 02:04 AM, John Hearns via Beowulf wrote:
> Hmmm... I will also chip in with my favourite tip
> Look at the sysctl for min_free_kbytes It is often set very low.
> Increase this substantially. It will do no harm to your system (unless
> you set it ti an absurd value!)
> You should be looking at the vm dirty ratios etc. also
(or the corresponding _ratios)
Defaults are low.
Increasing them improved a lot our compute nodes IO.
> On 15 February 2018 at 00:44, Kilian Cavalotti
> <kilian.cavalotti.work at gmail.com
> <mailto:kilian.cavalotti.work at gmail.com>> wrote:
> On Wed, Feb 14, 2018 at 2:26 PM, David Mathog <mathog at caltech.edu
> <mailto:mathog at caltech.edu>> wrote:
> > Checked the hugepage settings and found a difference there. The two systems
> > that don't do this have /sys/kernel/mm/redhat_transparent_hugepage/defrag
> > always madvise [never]
> > whereas the system with the issue has:
> > [always] madvise never
> THP defragmentation is definitely something that has bitten us in the
> past, when under memory pressure, and we now default to [madvise]
> pretty much everywhere (we're too timid to disable it entirely).
> A good way to see if that's really the issue is to "echo never >
> /sys/kernel/mm/redhat_transparent_hugepage/defrag" while the problem
> is happening, while simultaneously monitoring the processes with htop,
> for instance.
> It's usually pretty instant: if the issue is really with THP defrag,
> then CPU usage for your stalling process should drop pretty much
> immediately and things go back to normal.
> Beowulf mailing list, Beowulf at beowulf.org
> <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf