[Beowulf] Big storage

[Leif Nixon]

Jakob Oestergaard <jakob at unthought.net> writes:

> I'm looking at getting some big storage. Of all the parameters, getting as low
> dollars/(month*GB) is by far the most important. The price of acquiring and
> maintaining the storage solution is the number one concern.

Even at the price of reliability.

> One setup I was looking at, is simply using SunFire X4500 systems (you can put
> 48 standard 3.5" SATA drives in each 4U system). Assuming I can buy them with
> 1T SATA drives shortly, I could start out with 3 systems (12U) and grow the
> entire setup to 1P with 22 systems in little over two full racks.

They are nice, I understand. And seem like a good approach. The
vendors will likely want to sell you SAN-ified systems with cluster
file systems and lots of Windows-only management software. It can take
some convincing to get them to understand that that is actually not
what you are looking for.

> Any better ideas? Is there a way to get this more dense without
> paying an arm and a leg? Has anyone tried something like this with
> HSM?

That is probably not worth the bother.

But something that you have to be prepared for when going to that
storage volume is that you *will* suffer data corruption at some
point, and you need to plan for it. See for example

  http://cern.ch/Peter.Kelemen/talk/2007/kelemen-2007-C5-Silent_Corruptions.pdf

It's quite possible (though unlikely) for a hard disk to suddenly
return corrupted data without signalling a read error, and this is a
possibility that raid controllers typically just ignores. And then you
have the usual crop of software, firmware and hardware errors that can
trash your data more or less silently.

At a minimum, make sure you keep checksums of all files so you can
verify their integrity.

-- 
Leif Nixon                       -            Systems expert
------------------------------------------------------------
National Supercomputer Centre    -      Linkoping University
------------------------------------------------------------