[Beowulf] how large can we go with 1GB Ethernet? / Re: how large of an installation have people used NFS, with?

[Skylar Thompson]

Greg Lindahl wrote:
> On Wed, Sep 09, 2009 at 03:23:30PM -0400, psc wrote:
>
>   
>> I wonder what would be the sensible biggest cluster possible based on
>> 1GB Ethernet network.
>>     
>
> People build multi-thousand node clusters like that. In the HPC world,
> oil-and-gas and some other industrial applications don't need more
> than 1 gbit. Of couse these days they use 10 gbit to connect 1gbit
> switches.
>
> In the non-HPC world, companies with huge datacenters often have
> thousands to many thousands of nodes in a single layer-2 network with
> 1gbit to the nodes. That's limited only by the size of the mac addr
> tables in your switches. When you have to split your cluster into
> layer-3 chunks, the bandwidth between chunks sucks, but most clusters
> in the non-HPC world are limited to 2000-4000 nodes anyway by various
> software limitations.
>
>   

How do people with large layer-2 networks deal with broadcast storms?
I've been in situations even on small networks where a node goes haywire
and starts spewing broadcast traffic and slows everything in its
broadcast domain down. The probability and impact of that goes up with
larger networks, and it seems like even the baseline chatter from ARP
could be significant.

-- 
-- Skylar Thompson (skylar at cs.earlham.edu)
-- http://www.cs.earlham.edu/~skylar/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://www.scyld.com/pipermail/beowulf/attachments/20090909/f56d24fd/signature.bin