[Beowulf] Roadrunner picture

[John Hearns]

On Thu, 2008-07-17 at 04:42 +1000, Andrew Robbie (GMail) wrote:
> On Tue, Jul 15, 2008 at 12:35 AM, Josip Loncaric <josip at lanl.gov> wrote:
> >
> > Another good link:
> >
> > http://www.lanl.gov/orgs/hpc/roadrunner/rrtechnicalseminars2008.shtml
> 
> As I was reading the slides, one question leap out at me: they have a
> huge IB network connecting every 'node', but instead of connecting
> direct to storage, this connects to an IB-to-10GigE bridge/IO
> processor board. Why? If they used avoided the protocol conversion
> going on that would be inherently simpler, and I've seen nothing to
> indicate that 10GigE is faster or cheaper (certainly not cheaper!).
> 
> Is this (ethernet) a Panasas requirement?
Yes indeed it is.
I'll stick my head up above the parapet as someone who cares for and
feeds several Panasas installations, though I don't work for the
company.

For installations using Infiniband, Panasas will advise on how to
implement Infiniband to (10gig) Ethernet storage routers. 
I take it here that Roadrunner is using a blade in the Infiniband
switch, rather than a discreet router. This sounds a very good idea to
me, and I looked into it for a particular project in the UK, though we
didn't go for that approach in the end.

To answer your question more directly, Panasas is a storage cluster to
complement your compute cluster. Each storage blade is connected into a
shelf (chassis) with an internal ethernet network. Each shelf is then
connected to your ethernet switch with at least 4Gbps of bandwidth.
It might look on the front like a big RAID array - and hence the
questions as to why you don't have a fibrechannel or native Infiniband
connector on it. But its not really a RAID array - its a storage
cluster. Files are RAIDed over the filesystem and your client stripes IO
over several storage blades at any one time.

And please lets not start any Infiniband good/ethernet bad wars here,
or bandwidth willy-waving. Panasas have made some damned good
engineering decisions and their system scales like crazy - just what you
need for something like Roadrunner.