[Beowulf] Infiniband modular switches

[Joe Landman]

On a different note ...

Gilad Shainer wrote:

> routing capabilities. And if you are bandwidth bounded, using IB QDR
> will help. You will be able to drive more than 3GB/s from each server. 

This is theoretical max, correct?  What sort of real observed DMA 
bandwidth are people seeing with the QDR adapters?  That is, how quickly 
can I copy 32MB from main memory to the PCIe (even gen2) based card?  Or 
how quickly (e.g. what is the real observed data rate) for copying 
streaming data, say from a nice NFS over RDMA session, to the card?

Please correct me if I am wrong, but I suspect that this will be one of 
rate limiting factors at least in the near term.

Not that this is a bad thing, 3GB/s is quite good, but if the practical 
best case memory->PCIe bandwidth only let us get 2GB/s, or even 1.5 
GB/s, this is important to know.

If you or someone out there with QDR could measure this, I think a few 
of us (at least on the storage side) would like to know this.  If 
someone wants to loan us cards and cables (cough cough), we would be 
happy to do this measurement between pairs of JackRabbits.

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615