[Beowulf] Cheap SDR IB

[Gilad Shainer]

Richard Walsh wrote:
>> With more cores on a single node, the IB benefits are seen in much
lower number 
>> of nodes. I am testing some applications on a new cluster that I have
(dual 
>> sockets quad core Barcelona), and my first results are with Fluent
new 
>> benchmarks. I will have the numbers posted soon, so you all can take
a look. For 
>> 2 nodes, IB shows an average of 15-20% higher performance then GigE,
and this 
>> gap gets bigger with cluster size. At 4 nodes the difference was
40-50%. Even 
>> more important, 3 nodes results with IB were higher then 8 nodes with
GigE, and 
>> GigE stop scaling after 3-4 nodes (performance numbers were flat
after 3-4 
>> nodes). 
 
> Interesting. I am guessing this is with ConnectX adapters and PCIe
Gen2. When does
> the InfiniBand technology used in this test flatten?  How about the
less expensive SDR
> technology referred to in Jeff's article?  Can you provide the curves
up to 8 nodes for
> both IB and GE in this test? 
>
> A quick analysis, limited to this Fluent test, suggests that 4 nodes
plus ConnectX 
> technology roughly equals 8 nodes plus on-board GE technology.  Can
you provide
> the system per node price differences?  Then we can rougly determine
the cost 
> benefit relationship.

In my testing I have used ConnectX and PCIe Gen1. I will have the
numbers posted soon and will send a note to this list once done. The
major take was that GigE did not scale beyond 3 servers, so adding more
nodes did not provide more performance (Fluent rating stayed the same
from 3 nodes with GigE and above). The IB shows almost linear scaling as
we have added more nodes. The less expensive SDR technology should
follow the same lines, as latency was more important in those
benchmarks, so the numbers should be in the same range. 

For 2 nodes only the performance gap was 15-20%. If I am taking the safe
approach and assuming 10-15% gap with the low cost SDR. The servers will
cost you ~$8K, the IB part (for 2 nodes you don't need a switch)
according to the pricing in Jeffs is <$300 (2 adapters and one cable)
which is less than 4% added cost. Of course when you scale beyond 2
nodes, the cost/performance advantage with IB increases dramatically (as
GigE performance is flat beyond 3 servers).