<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META http-equiv=Content-Style-Type content=text/css>
<STYLE type=text/css>BODY {
MARGIN: 5px; BACKGROUND-COLOR: #ffffff
}
HR {
COLOR: #000000
}
BODY {
FONT-WEIGHT: normal; FONT-SIZE: 9pt; COLOR: #000000; FONT-STYLE: normal; FONT-FAMILY: 'Courier New'; TEXT-DECORATION: none
}
TABLE {
FONT-WEIGHT: normal; FONT-SIZE: 9pt; COLOR: #000000; FONT-STYLE: normal; FONT-FAMILY: 'Courier New'; TEXT-DECORATION: none
}
SPAN.rvts1 {
FONT-WEIGHT: bold; FONT-SIZE: 10pt; COLOR: #0000ff; FONT-FAMILY: 'Arial'
}
SPAN.rvts2 {
FONT-WEIGHT: bold; FONT-SIZE: 10pt; COLOR: #000080; FONT-FAMILY: 'Arial'
}
SPAN.rvts3 {
FONT-SIZE: 10pt; COLOR: #800000; FONT-STYLE: italic; FONT-FAMILY: 'Arial'
}
A.rvts4 {
FONT-SIZE: 10pt; COLOR: #008000; FONT-FAMILY: 'Arial'; TEXT-DECORATION: underline
}
SPAN.rvts4 {
FONT-SIZE: 10pt; COLOR: #008000; FONT-FAMILY: 'Arial'; TEXT-DECORATION: underline
}
A.rvts5 {
FONT-SIZE: 10pt; COLOR: #008000; FONT-FAMILY: 'Arial'; TEXT-DECORATION: underline
}
SPAN.rvts5 {
FONT-SIZE: 10pt; COLOR: #008000; FONT-FAMILY: 'Arial'; TEXT-DECORATION: underline
}
SPAN.rvts6 {
FONT-SIZE: 10pt
}
P {
PADDING-RIGHT: 0px; PADDING-LEFT: 0px; PADDING-BOTTOM: 0px; MARGIN: 0px; TEXT-INDENT: 0px; PADDING-TOP: 0px; TEXT-ALIGN: left
}
UL {
PADDING-RIGHT: 0px; PADDING-LEFT: 0px; PADDING-BOTTOM: 0px; MARGIN: 0px; TEXT-INDENT: 0px; PADDING-TOP: 0px; TEXT-ALIGN: left
}
OL {
PADDING-RIGHT: 0px; PADDING-LEFT: 0px; PADDING-BOTTOM: 0px; MARGIN: 0px; TEXT-INDENT: 0px; PADDING-TOP: 0px; TEXT-ALIGN: left
}
.rvps1 {
TEXT-ALIGN: center
}
</STYLE>
<META content="MSHTML 6.00.6000.16608" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=813395521-12062008>So you're concerned with
the gap between the 2.63 us that OSU measured and your 3.07 us you
measured. I wouldn't be too concerned.</SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008>MPI latency can be quite
dependent on the systems you use. OSU used dual-processor 2.8 Ghz
processors. Such as system has ~60 ns latency to local memory.
On your 4-socket Opteron system, your local memory latency is probably in
the 90-100 ns range. </SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008>Assuming you are also
using MVAPICH2, this is probably the main difference for the latency shortfall
you are seeing.</SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008>Another possibility is
that the CPU you are running the MPI test on is not the closest CPU to the PCIe
chipset. Thus, you may be taking some HT hops on the way to the PCIe bus
and adapter card.</SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=813395521-12062008>-Tom</SPAN></DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> beowulf-bounces@beowulf.org
[mailto:beowulf-bounces@beowulf.org] <B>On Behalf Of </B>Jan
Heichler<BR><B>Sent:</B> Thursday, June 12, 2008 2:28 PM<BR><B>To:</B> Beowulf
Mailing List<BR><B>Subject:</B> [Beowulf] MVAPICH2 and
osu_latency<BR></FONT><BR></DIV>
<DIV></DIV>
<P>Dear all!</P>
<P><BR></P>
<P><BR></P>
<P>I found this <SPAN
class=rvts4>http://mvapich.cse.ohio-state.edu/performance/mvapich2/opteron/MVAPICH2-opteron-gen2-DDR.shtml</SPAN><SPAN
class=rvts6> as reference value for MPI-latency of Infiniband. I try to
reproduce those numbers at the moment but i'm stuck with</SPAN></P>
<P><SPAN class=rvts6><BR></SPAN></P>
<P><SPAN class=rvts6># OSU MPI Latency Test v3.0</SPAN></P>
<P><SPAN class=rvts6>#
Size Latency
(us)</SPAN></P>
<P><SPAN
class=rvts6>0
3.07</SPAN></P>
<P><SPAN
class=rvts6>1
3.17</SPAN></P>
<P><SPAN
class=rvts6>2
3.16</SPAN></P>
<P><SPAN
class=rvts6>4
3.15</SPAN></P>
<P><SPAN
class=rvts6>8
3.19</SPAN></P>
<P><SPAN class=rvts6><BR></SPAN></P>
<P><SPAN class=rvts6>Equipment is two quadsocket Opteron Blades (Supermicro)
with Mellanox Ex DDR cards. Single 24 port switch connects them.</SPAN></P>
<P><SPAN class=rvts6><BR></SPAN></P>
<P><SPAN class=rvts6>Can anybody help with suggestions what i can do to lower
the latency? </SPAN></P>
<P> </P>
<P><BR></P>
<P>Regards, Jan
</P></BLOCKQUOTE></BODY></HTML>