[Beowulf] IB in the real world
jjengla at sandia.gov
Wed May 25 18:05:20 PDT 2005
On Wed, 2005-05-25 at 15:47 -0400, Patrick Geoffray wrote:
> Hi Josh,
> Josh England wrote:
> > On Thu, 2005-05-12 at 14:32 -0700, Bill Broadley wrote:
> >>6. What real world latencies and bandwidths are you observing on production
> >> clusters with MPI? How much does that change when all nodes are running
> >> the latency or bandwidth benchmark?
> > I don't have the numbers offhand, but I recall about 7-9ns latency and
> > ~=800MB/s on PCI-X and ~=1200MB/s on PCIe.
> ~1200 MB/s on a single IB 4x link ?
> The IB 4x link is 4 links at 2.5 Gb/s signal rate, so that 10 Gb/s
> signal rate total. After the 8b/10b encoding, that's 8 Gb/s data rate.
> Even without counting the efficiency of the protocol (ratio payload size
> / packet size wich should cost ~3% conservatively assuming a header of
> 64 Bytes and a max payload of 2KB), there is no way to get more than 1
> GB/s *on paper*.
> How did you get 1200 MB/s ?
I should have been more explicit. I don't mean the dual port cards.
This is only on a single port.
The latency is around 3.5us and bw is around 900MB/s unidirectional and
1600MB/s bidirectional on PCIe.
PCI-X is 6.0-7.0us for latency and 750-850MB/s (uni), 750-875 (bi).
These numbers may be a bit off, but I think they're close.
More information about the Beowulf