[Beowulf] Slow RAID reads, no errors logged, why?

[David Mathog]

Found the problem.  Well, sort of.

The issue is that when a long SMART test runs on any disk on the system 
(A) which has this problem the IO goes down to 30Mb/s.  It doesn't 
matter which disk is running the test.  The system we have which is most 
like it (C) does not have this issue.

            A       C
Centos     6.7     6.9
RAM        512     512 Gb
CPUs       56      40  (actually threads)
PowerEdge  T630    T630
Xeon       E5-2695 E5-2650 (both v3)
speed      2.30GHz 2.30Ghz
cpufreq?   yes     no
PERC       H730    H730P
SAS disk   ST2000NM0023
SAS disk           ST4000NM0005

There are a bunch of small differences between the two systems so it is
hard to say for sure which is the actual culprit.

I will put this out on the smartmontools list and see if anybody has 
seen it before.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech