[Beowulf] Infiniband modular switches
Shainer at mellanox.com
Sun Jun 15 14:46:05 PDT 2008
Joe Landman wrote:
> On a different note ...
> Gilad Shainer wrote:
> > routing capabilities. And if you are bandwidth bounded,
> using IB QDR
> > will help. You will be able to drive more than 3GB/s from
> each server.
> This is theoretical max, correct? What sort of real observed
> DMA bandwidth are people seeing with the QDR adapters? That
> is, how quickly can I copy 32MB from main memory to the PCIe
> (even gen2) based card? Or how quickly (e.g. what is the
> real observed data rate) for copying streaming data, say from
> a nice NFS over RDMA session, to the card?
No, this is the actual number. IB QDR theoretical max is 4GB/s
> Please correct me if I am wrong, but I suspect that this will
> be one of rate limiting factors at least in the near term.
> Not that this is a bad thing, 3GB/s is quite good, but if the
> practical best case memory->PCIe bandwidth only let us get
> 2GB/s, or even 1.5 GB/s, this is important to know.
PCIe x8 Gen1 is maxed in around 1.5-1.6GB/s, and PCIe x8 Gen2 in around
3.3-3.4GB/s. There are multiple PCIe Gen2 servers out there from
multiple vendors. I am using Supermicro based for my testing, but there
are other solutions.
Here is some non-tuned MPI results bandwidth (IB QDR on PCIe x8 Gen2
Bytes PingPong (MB/s) SendRecv (MB/s)
512 695.71 707.2
1024 1160.13 1178.18
2048 1664.83 1972.34
4096 1957.76 2597.41
8192 2165.76 3025.58
16384 2642.91 4187.19
32768 2962.5 5076.31
65536 3165.03 5693.37
131072 3274.05 6078.59
262144 3332.28 6287.49
524288 3362.12 6399.46
1048576 3377.28 6458.3
2097152 3385.08 6489.52
4194304 3388.02 6457.68
> If you or someone out there with QDR could measure this, I
> think a few of us (at least on the storage side) would like
> to know this. If someone wants to loan us cards and cables
> (cough cough), we would be happy to do this measurement
> between pairs of JackRabbits.
I might be able to help you here.
More information about the Beowulf