[Beowulf] Multirail Clusters: need comments

[john leidel]

Greg, I've had some experience working with multirail Infiniband 
clusters.  The main issue in our implementation was complexity.  Our 
specific implementation involves a dual-rail, 3D Torus networking 
topology, so the physical cabling can become interesting.   Once you can 
wrap your mind around the topology, the issue of complexity isn't so 
bad. On the issue of drivers, we're using the Infiniband Gold Stack 
(with OSU's mvapich) from Mellanox. They seem to work just fine.  The 
answer to your performance question is, of course, a bit loaded.  You 
will only see a drastic increase in performance when sending large 
messages across the fabric.  The message size must reach a certain 
threshold (forgot the exact number off-hand) in order to split the 
message across both rails.  When running apps with large messages, you 
should see quite a jump in bandwidth.

cheers
john

Greg Anderson wrote:

> I would like to know the views of the cluster gurus on multi-rail 
> clusters, esp, quad rail networks. Are quad rail networks practical to 
> implement ( say InfiniBand, Quadrics, Myrinet,  (GE, maybe not?)).
>
> - What are the issues?
> - Are there any quad rail HPC clusters? (me and google couldn't find any)
> - Are drivers an issue?
> - Does the performance increase significantly to justify the cost and 
> complexity?
>
> The assumption is that the plan is to solve comm or BW crippled apps, 
> on large SMP nodes (say 8+ CPUs).
>
> Note: I found just one line in a CLRC Daresbury lab presentation about 
> quadrail Quadrics on Alpha (probably QsNet-1?) Any update on QsNet2/Eagle?
>
> Thanks!
> Greg
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>  
>