<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Arial; font-size: 12pt; color: #000000'><br>On Thursday, April 8, 2010 2:14:11 PM Greg Lindahl wrote:<div><div><div><br>>> What are the approaches and experiences of people interconnecting <br>>> clusters of more than128 compute nodes with QDR InfiniBand technology? <br>>> Are people directly connecting to chassis-sized switches? Using multi-tiered <br>>> approaches which combine 36-port leaf switches?<br>><br>>I would expect everyone to use a chassis at that size, because it's cheaper<br>>than having more cables. That was true on day 1 with IB, the only question is<br>>"are the switch vendors charging too high of a price for big switches?"</div><div><br></div><div>Hey Greg,</div><div><br></div><div>I think my target is around 192 compute nodes, with room for a head node(s),</div><div>and ports to a Lustre file server. So, 216 ports looks like a reasonable number</div><div>to me (6 x 36). The price for an integrated chassis model solution should not exceed</div><div>the price for a multi-tiered solution using 36-port (or some other switch smaller</div><div>than 216) plus the cabling costs. Reliability and labor would also have to factored</div><div>in with an advantage going to the chassis I assume based also on fewer cables?</div><div>Looks like the chassis options are between $375 and $400 a port, while the 36</div><div>port options are running at about $175 to $200 a port (but you need more ports and</div><div>cables). </div><div><br>>> I am looking for some real world feedback before making a decision on <br>>> architecture and vendor. <br>><br>>Hopefully you're planning on benchmarking your own app -- both the<br>>HCAs and the switch silicon have considerably different application-<br>>dependent performance characteristics between QLogic and Mellanox<br>>silicon.<br><br></div><div>Yes, I assume that people would also recommend matching NIC and switch</div><div>hardware. </div><div><br></div><div>Thanks for your input ...</div><div><br></div><div>rbw<br><br><br><br>_______________________________________________<br>Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing<br>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf<br></div></div></div></div></body></html>