[Beowulf] Lowered latency with multi-rail IB?

[Håkon Bugge]

On Mar 27, 2009, at 6:20 , Nifty Tom Mitchell wrote:

> On Thu, Mar 26, 2009 at 09:03:30PM -0700, Greg Lindahl wrote:
>> On Thu, Mar 26, 2009 at 11:32:23PM -0400, Dow Hurst DPHURST wrote:
>>
>>> We've got a couple of weeks max to finalize spec'ing a new  
>>> cluster.  Has
>>> anyone knowledge of lowering latency for NAMD by implementing a
>>> multi-rail IB solution using MVAPICH or Intel's MPI?
>>
>> Multi-rail is likely to increase latency.

In genetral true, but it depends. With a heavy all-to-all  
communication pattern, I have seen intelligent use of two HCAs (not  
muiltirail per se in order to increase bandwidth). I have observed  
almost 2x performance increase on SPEC MPI2007 pop2 application when  
going from one to two HCAs. This is a typical many-small-packets  
application.

I am not saying the same is true for NAMD, but I wouldn't rule it out.

>> BTW, Intel MPI usually has higher latency than other MPI
>> implementations.

Intel MPI and MVAPICH is not what I would call top-of-the-line  
performers (but I am biased as I work for a company delivering an MPI  
product). You might find it useful to look at pop2 performance for  
different interconnects and MPI implementations at ttp://www.spec.org/mpi2007/results/mpi2007.html



Thanks, Håkon