[Beowulf] IB in the real world

[Patrick Geoffray]

Hi Josh,

Josh England wrote:
> On Thu, 2005-05-12 at 14:32 -0700, Bill Broadley wrote:

>>6.  What real world latencies and bandwidths are you observing on production
>>    clusters with MPI?  How much does that change when all nodes are running
>>    the latency or bandwidth benchmark?
> 
> 
> I don't have the numbers offhand, but I recall about 7-9ns latency and
> ~=800MB/s on PCI-X and ~=1200MB/s on PCIe.

~1200 MB/s on a single IB 4x link ?

The IB 4x link is 4 links at 2.5 Gb/s signal rate, so that 10 Gb/s 
signal rate total. After the 8b/10b encoding, that's 8 Gb/s data rate. 
Even without counting the efficiency of the protocol (ratio payload size 
/ packet size wich should cost ~3% conservatively assuming a header of 
64 Bytes and a max payload of 2KB), there is no way to get more than 1 
GB/s *on paper*.

How did you get 1200 MB/s ?

Patrick
-- 

Patrick Geoffray
Myricom, Inc.
http://www.myri.com