[Beowulf] Infiniband modular switches

[Patrick Geoffray]

Gilad Shainer wrote:
>> The injection rate is irrelevant 
> 
> The injection rate is super relevant. If your injection rate is 10% of

The injection rate is absolutely irrelevant for contention due to 
Head-of-Line Blocking. You will have the same fabric efficiency under 
contention if your link rate is 1 Mb/s or 100 Gb/s. For example, the 
efficiency of a single-channel *full* crossbar under random traffic is 
58.6%, whatever is the link rate.

Google "HOL blocking" or read a book.

> you referred to, when doing the same testing on IB DDR we got much
> better results with IB versus Quadrics. your theory does not really meet

The routing efficiency of 3-year old Quadrics QsNetII is greater than 
latest Mellanox IB in these tests. For IB SDR link rate, Quadrics has 
more throughput because of the better efficiency, as their link rate is 
similar. For DDR throughput, it is possible that the increased link rate 
compensate for the bad routing efficiency.

The only comparison I have seen with IB DDR was by Woven Systems at 
Sandia last year, and 10 Gb/s Ethernet (higher link rate than QsNetII) 
was beating IB DDR for a similar benchmark.

> reality. Also, when you do look on adaptive routing, make sure it is
> real time solution...  Not some synthetic testing .... 

AlltoAll of large messages is not a useless synthetic benchmark IMHO.

> What Panda did is using multiple static routs, which can be chosen
> according to different algorithms. It is an interesting idea, and used
> in several cases with nice results. This is not adaptive routing. 

That's right, this is not adaptive routing, it's lip stick. And I am 
sorry but those are not nice results compared to real adaptive routing.

Patrick