[Beowulf] how large can we go with 1GB Ethernet? / Re: how large of an installation have people used NFS, with?

[Greg Lindahl]

On Wed, Sep 09, 2009 at 03:23:30PM -0400, psc wrote:

> I wonder what would be the sensible biggest cluster possible based on
> 1GB Ethernet network.

People build multi-thousand node clusters like that. In the HPC world,
oil-and-gas and some other industrial applications don't need more
than 1 gbit. Of couse these days they use 10 gbit to connect 1gbit
switches.

In the non-HPC world, companies with huge datacenters often have
thousands to many thousands of nodes in a single layer-2 network with
1gbit to the nodes. That's limited only by the size of the mac addr
tables in your switches. When you have to split your cluster into
layer-3 chunks, the bandwidth between chunks sucks, but most clusters
in the non-HPC world are limited to 2000-4000 nodes anyway by various
software limitations.

-- greg