[Beowulf] QDR InfiniBand interconnect architectures ... approaches ...

[Tom Ammon]

Another thing to remember with chassis switches is that you can also 
build them in an oversubscribed model by removing spine cards. Most 
chassis' have at least 3 spine modules so you lose some granularity in 
oversubscription, but you can still cut costs. You don't have to go with 
fully nonblocking in a chassis if you want to save money.

Tom

On 04/09/2010 03:16 AM, Peter Kjellstrom wrote:
> On Thursday 08 April 2010, Greg Lindahl wrote:
>    
>> On Thu, Apr 08, 2010 at 04:13:21PM +0000, richard.walsh at comcast.net wrote:
>>      
>>> What are the approaches and experiences of people interconnecting
>>> clusters of more than128 compute nodes with QDR InfiniBand technology?
>>> Are people directly connecting to chassis-sized switches? Using
>>> multi-tiered approaches which combine 36-port leaf switches?
>>>        
>> I would expect everyone to use a chassis at that size, because it's cheaper
>> than having more cables. That was true on day 1 with IB, the only question
>> is "are the switch vendors charging too high of a price for big switches?"
>>      
> Recently we've (swedish academic centre) got offers using 1U 36-port switches
> not chassis from both Voltaire and Qlogic reason given: lower cost. So from
> our point of view, yes, "switch vendors [are] charging too high of a price
> for big switches" :-)
>
> One "pro" for many 1U switches compared to a chassi is that it gives you more
> topological flexibility. For example, you can build a 4:1 over subscribed
> fat-tree and that will obviously be cheaper than a chassi (even if they were
> more reasonably priced).
>
> /Peter
>    

-- 
--------------------------------------------------------------------
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu