[Beowulf] Building new cluster - estimate

[Joe Landman]

Chris Samuel wrote:
> ----- "Bogdan Costescu" <Bogdan.Costescu at iwr.uni-heidelberg.de> wrote:
> 
>> On Tue, 29 Jul 2008, Chris Samuel wrote:
>>
>>> 1) Use a mainline kernel, we've found benefit of that
>>> over stock CentOS kernels.
>> Care to comment on this statement ?
> 
> a) We found that we got better performance out of
> the mainline kernels than the CentOS ones; we guess
> because they handle newer hardware better (RHEL is
> meant to aim for stability over performance)

This mirrors our experience, though RHEL stability under intense loads 
is questionable IMO (talking about the kernel BTW).  We find that the 
missing drivers, the omitted drivers, the backported drivers along with 
some odd and often useless "features" (4k stacks anyone?) render the 
RHEL default kernels (and by definition the Centos kernels) less useful 
for HPC and storage tasks than what we build.  Our current standard is a 
2.6.23.14 kernel which is rock solid under load.  Working on a 2.6.26 
based version now (even though I am on vacation/holiday, I just updated 
it to 2.6.26.1 to address an observed crashing issue with the RDMA server)

> b) We can use XFS for scratch space rather than being
> tied to the RHEL One True Filesystem (ext3) which
> (in our experience) can't handle large amounts of disk
> I/O.

Combine this with the small upper limit of ext3 partition sizes, the 
file size limits in ext3, the serialization in the journaling code (ext4 
is extents based to help deal with this), ext3 just doesn't make much 
sense in a storage/HPC system (apart from possibly boot/root file system 
where performance is less critical).  Yeah I have seen studies from 
folks whom had done 1E6 removes, file creates, and other things who 
claim xfs is slower than ext3.  Yeah, those are bad benchmarks in that 
they really don't touch on real end user use cases for the most part 
(apart from possible large scale mail servers and other things like that).

> 
> YMMV!

Always ... and wish gas in ~$4USD region, you need to conserve .  Having 
been in London a few months ago, seeing almost $10USD/gallon (3.75 
liters), I am gonna stop complaining about our price over here.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615