[Beowulf] Multirail Clusters: need comments

[duncan.roweth at quadrics.com]

Greg

Quadrics libraries (MPI and Shmem) provide transparent support 
for multiple rails. If you send lots of small messages they will 
be posted to alternate rails. Larger ones will be striped over the 
available rails. This is an important part of providing scalable 
comms for high CPU count SMPs, where you want the bandwidth and 
issue rate (numbers of MPI messages per microsecond) to rise 
with the number of CPUs per node. We have tested 2,3,4 and 8 rails 
on machines with 8-32 CPUs per node. 

Best Wishes
Duncan Roweth

Quadrics Ltd                www.quadrics.com
Tel: +44 117 9075384        duncan at quadrics.com  
Fax: +44 117 9075395 





-----Original Message-----
From: Greg Anderson [mailto:greg.s.anderson at gmail.com]
Sent: 29 November 2005 13:38
To: beowulf at beowulf.org
Subject: [Beowulf] Multirail Clusters: need comments


I would like to know the views of the cluster gurus on multi-rail clusters, esp, quad rail networks. Are quad rail networks practical to implement ( say InfiniBand, Quadrics, Myrinet,  (GE, maybe not?)).

- What are the issues? 
- Are there any quad rail HPC clusters? (me and google couldn't find any)
- Are drivers an issue?
- Does the performance increase significantly to justify the cost and complexity? 

The assumption is that the plan is to solve comm or BW crippled apps, on large SMP nodes (say 8+ CPUs). 

Note: I found just one line in a CLRC Daresbury lab presentation about quadrail Quadrics on Alpha (probably QsNet-1?) Any update on QsNet2/Eagle?

Thanks!
Greg