[Beowulf] Infiniband modular switches

andrew holway andrew at moonet.co.uk
Thu Jun 12 08:51:11 PDT 2008

+1 for the 24 port flextronics switches. They are very cost effective
for half bisectional networks upto 32 ports. It starts to get messy
after that.

I wonder how long we will be waiting for switches based on the 36p asic?

On Thu, Jun 12, 2008 at 4:08 PM, Don Holmgren <djholm at fnal.gov> wrote:
> Ramiro -
> You might want to also consider buying just a single 24-port switch for your
> 22 nodes, and then when you expand either replace with a larger switch, or
> build a distributed switch fabric with a number of leaf switches connecting
> into a central spine switch (or switches).  By the time you expand to the
> larger cluster, switches based on the announced 36-port Mellanox crossbar
> silicon will be available and perhaps per port prices will have dropped
> sufficiently to justify the purchase delay and the disruption at the time of
> expansion.
> If your applications can tolerate some oversubscription (less than a 1:1
> ratio of leaf-to-spine uplinks to leaf-to-node connections), a distributed
> switch fabric (leaf and spine) has the advantage of shorter (and cheaper)
> cables between the leaf switches and your nodes, and relatively fewer longer
> cables from the leaves back to the spine, compared with a single central
> switch.
> We have many Flextronics switches - SDR and DDR, 24-port and 144-port - on a
> pair of large clusters (520 nodes, and 600 nodes) built in 2005 and 2006. No
> complaints.  But, we have been self-supporting, and I would guess you would
> have very different support structures with Voltaire or Qlogic.  With the
> Flextronics
> switches you will definitely be using the OFED stack, and you will have to
> run
> a subnet manager on one of your nodes (dedicated is probably best).  You
> could
> optionally buy an embedded subnet manager on the Voltaire or Qlogic
> switches,
> depending upon model, though I believe for a large fabric an external subnet
> manager is still recommended.
> Don Holmgren
> Fermilab
> On Tue, 10 Jun 2008, Ramiro Alba Queipo wrote:
>> Hello everybody:
>> We are about to build an HPC cluster with infiniband network starting
>> from 22 dual socket nodes with AMD QUAD core processors and in a year or
>> so we will be having about 120 nodes. We will be using infiniband both
>> for calculation as for storage.
>> The question is that we need a modular solution and we are having 3
>> candidates:
>> a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)->
>> seems very good and well supported, but very expensive.
>> b) Qlogic SilverStorm 9120 (144 ports) -> no price and support
>> information yet
>> c) Flextronics 10U 144 Port Modular-> very good at price but little
>> support => risky option?.
>> I am in a mess. What is your opinion about this matter? Are you using
>> any of this products.
>> Regards
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list