[Beowulf] Network considerations for new generation cheap beowulfcluster

[Vincent Diepeveen]

hi,

Thanks for your reaction.

Ethernet is of course too slow in latency.

the cheapest cable i see is 1 meter and $70



Cheapest card i see is $715

So the node price starts at $765, which is already way way more than the 
total price of 1 node.
Now we didn't discuss the switches yet. Switches and routing of a network is 
important.

The problem of myrinet nowadays is already that it is way too expensive when 
compared to the node price.

I also tend to remember a few years ago that a myrinet card was like far 
under $500.
Now cheapest card of myri i see is $715, and i didn't see the huge price of 
switches
yet that will add up to node price.

More interesting than paying a $1000 a node for 10 gigabit MPI, is having 
some older card say 3 gbit/s,
which uses MPI and is DMA low latency with a bit older switch for say $300 a 
node.

Then you've got a good low latency network for a small price, yet still 
making price of a node more expensive,
from $450 to $750.

Vincent

----- Original Message ----- 
From: "Jess Cannata" <jac67 at georgetown.edu>
Cc: <beowulf at beowulf.org>
Sent: Friday, May 18, 2007 10:44 PM
Subject: Re: [Beowulf] Network considerations for new generation cheap 
beowulfcluster


> While I can't foresee the future, I do think that we are going to a lot 
> more low latency 10 Gb/s cards that use standard 10 Gb switches and cables 
> such as Myricom's 10 Gb Myrinet/Ethernet card and NetEffect's 10 Gb 
> Ethernet card.
>
> http://www.myricom.com/Myri-10G/product_list.html
> http://www.neteffect.com/ne020-features.html
>
> Jess
>
> Vincent Diepeveen wrote:
>> hi All of you,
>>  Now that developments go fast in CPU land, by 22 july or so, intel drops 
>> price of its quad core to $266 more or less.
>> Hopefully AMD's quadcore chip releases soon too for a decent price.
>>  As intels memory subsystem is real weak, not to mention the extra price 
>> that AMD and intel ask for dual socket/quad socket capable chips, the 
>> optimal node is a single socket node.
>>  4 cores is already a lot anyway for 1 highend network card.
>>  That means in short that you can produce for quite little money, far 
>> under $500, a node with 4 cores,
>> or considering the far higher taxrates in Europe, far under 500 euro in 
>> Europe.
>>  Basically what a node needs is a mainboard, a bit of RAM, and a cpu with 
>> cooler. That keeps a node tiny and it's easier coolable. With some wood 
>> then you can build a great case that holds many nodes. Booting of course 
>> diskless over the gigabit network. Of course interesting to know secondly 
>> is whether putting in ECC-reg ram is interesting, considering its 
>> scandaleous high price always.
>>  What are opinions here?
>>  Of course now the question is how to get a reasonable low latency 
>> highend network with a reasonable bandwidth (latency bigger priority than 
>> bandwidth of course) and of course being capable of reading in memory 
>> without writing. Of course the switch/routing prices + cable prices need 
>> to be included in those price considerations.
>>  Perhaps some bit older generation card gets sold very cheap now. What 
>> are the options the coming years there, any manufacturer keeping up with 
>> the dropped price of a single quad core node?
>>  Gigabit ethernet is not an option of course, that just works for 
>> embarrassingly parallel software, it's usually interrupting bigtime the 
>> cpu and has an ugly one-way pingpong latency, especially when there is 
>> several threads simultaneously shipping messages.
>>  What are the options for the network in the future?
>>  Vincent
>>  ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit 
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf
>