[Beowulf] Cheap SDR IB

[Joe Landman]

Mark Hahn wrote:

> sure, and these are very fat nodes for which a fat interconnect is
> appropriate for almost any workload that's not embarassing.  but really
> I wasn't suggesting that plain old Gb (bandwidth in particular) was
> adequate for all possible clusters.  I was questioning whether IB was a 
> panacea for small, cost-sensitive ones...

Cheap gigabit is fine for small cost sensitive clusters.  You can get 
cheap (not great, but ok) gigabit switches with 48 ports for under $700 
today.  They are not as fast as the higher cost ones from HP and others, 
but they are great for inexpensive clusters.

At the small cluster side of things, the cost per core and cost per node 
(fully burdened with switches, cables, OS, compilers, etc) is very 
important.  At these prices for some small clusters, the cost to add IB 
is no longer completely prohibative.  But, at the same time, the benefit 
needs to outweigh the costs.  I would argue that the more interesting 
small clusters with IB probably won't be used for message passing, but 
for storage using NFSoverRDMA to move large chunks of data back and 
forth.  There you get 5-8x better performance on your data xfer from 
storage than you get with gigabit.  For *some* workloads (life science 
"database" analysis , large image processing, ...) this could be quite 
important.

Most of the benchmark results (life science codes, chemistry codes, 
engineering codes, ...) I have seen/worked on don't show a huge 
difference between gigabit and IB until you get north of 32 cores.  Of 
course, that is 2-4 nodes these days ...



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615