<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><title></title>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Style-Type" content="text/css">
<style type="text/css"><!--
body {
margin: 5px 5px 5px 5px;
background-color: #ffffff;
}
/* ========== Text Styles ========== */
hr { color: #000000}
body, table /* Normal text */
{
font-size: 9pt;
font-family: 'Courier New';
font-style: normal;
font-weight: normal;
color: #000000;
text-decoration: none;
}
span.rvts1 /* Heading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #0000ff;
}
span.rvts2 /* Subheading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #000080;
}
span.rvts3 /* Keywords */
{
font-size: 10pt;
font-family: 'Arial';
font-style: italic;
color: #800000;
}
a.rvts4, span.rvts4 /* Jump 1 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
a.rvts5, span.rvts5 /* Jump 2 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
span.rvts6
{
font-weight: bold;
color: #ffffff;
background-color: #0000ff;
}
span.rvts7
{
}
span.rvts8
{
font-family: 'tahoma';
font-weight: bold;
}
span.rvts9
{
font-family: 'tahoma';
}
span.rvts10
{
font-family: 'arial';
color: #008000;
text-decoration: underline;
}
/* ========== Para Styles ========== */
p,ul,ol /* Paragraph Style */
{
text-align: left;
text-indent: 0px;
padding: 0px 0px 0px 0px;
margin: 0px 0px 0px 0px;
}
.rvps1 /* Centered */
{
text-align: center;
}
--></style>
</head>
<body>
<p>Hallo Tom,</p>
<p><br></p>
<p>Freitag, 13. Juni 2008, meintest Du:</p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2>
<tr valign=top>
<td width=7 style="background-color: #0000ff;">
<p><span class=rvts6>></span></p>
</td>
<td width=696 style="background-color: #ffffff;">
<p><span class=rvts7>So you're concerned with the gap between the 2.63 us that OSU measured and your 3.07 us you measured. I wouldn't be too concerned.</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>1st: i get a value of 2.96 with MVAPICH 1.0.0 - this is exactly the value that i find on the mvapich website ;-)</p>
<p><br></p>
<p>It is not about being concerned not to get "optimal performance" - i know that such micro-benchmarks are of limited use... but i have a customer requirement. And since it seems possible it would be helpfull to get there</p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2>
<tr valign=top>
<td width=7 style="background-color: #0000ff;">
<p><span class=rvts6>></span></p>
</td>
<td width=696 style="background-color: #ffffff;">
<p><span class=rvts7> </span></p>
<p><span class=rvts7>MPI latency can be quite dependent on the systems you use. OSU used dual-processor 2.8 Ghz processors. Such as system has ~60 ns latency to local memory. On your 4-socket Opteron system, your local memory latency is probably in the 90-100 ns range. </span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>Why? And how can i measure this?</p>
<p><br></p>
<p>According to the link i posted they used a 144 Port-Switch. This is 3 HOPs - i have just 1. If that is true the difference should be another 300 ns higher because of the latency of the IB switch silicon...</p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2>
<tr valign=top>
<td width=7 style="background-color: #0000ff;">
<p><span class=rvts6>></span></p>
</td>
<td width=696 style="background-color: #ffffff;">
<p><span class=rvts7> </span></p>
<p><span class=rvts7>Assuming you are also using MVAPICH2, this is probably the main difference for the latency shortfall you are seeing.</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>MVAPICH2 1.03 and 1.02 tested. </p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2>
<tr valign=top>
<td width=7 style="background-color: #0000ff;">
<p><span class=rvts6>></span></p>
</td>
<td width=696 style="background-color: #ffffff;">
<p><span class=rvts7> </span></p>
<p><span class=rvts7>Another possibility is that the CPU you are running the MPI test on is not the closest CPU to the PCIe chipset. Thus, you may be taking some HT hops on the way to the PCIe bus and adapter card.</span></p>
<p><span class=rvts7><br></span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>The value is everytime the same. Shouldn't it be different then every run? And: how can i move the process? numactl or taskset just works on the local process i assume. How can i move the "remote process" on the other host?</p>
<p><br></p>
<p>Regards,</p>
<p>Jan</p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2>
<tr valign=top>
<td width=7 style="background-color: #0000ff;">
<p><span class=rvts6>></span></p>
</td>
<td width=696 style="background-color: #ffffff;">
<p><span class=rvts7>-Tom</span></p>
<p><br></p>
<p><br></p>
<hr noshade size=2>
<p><span class=rvts8>From:</span><span class=rvts9> beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] </span><span class=rvts8>On Behalf Of </span><span class=rvts9>Jan Heichler</span></p>
<p><span class=rvts8>Sent:</span><span class=rvts9> Thursday, June 12, 2008 2:28 PM</span></p>
<p><span class=rvts8>To:</span><span class=rvts9> Beowulf Mailing List</span></p>
<p><span class=rvts8>Subject:</span><span class=rvts9> [Beowulf] MVAPICH2 and osu_latency</span></p>
<p><br></p>
<p><span class=rvts7>Dear all!</span></p>
<p><br></p>
<p><br></p>
<p><span class=rvts7>I found this </span><span class=rvts10>http://mvapich.cse.ohio-state.edu/performance/mvapich2/opteron/MVAPICH2-opteron-gen2-DDR.shtml</span><span class=rvts7> as reference value for MPI-latency of Infiniband. I try to reproduce those numbers at the moment but i'm stuck with</span></p>
<p><br></p>
<p><span class=rvts7># OSU MPI Latency Test v3.0</span></p>
<p><span class=rvts7># Size Latency (us)</span></p>
<p><span class=rvts7>0 3.07</span></p>
<p><span class=rvts7>1 3.17</span></p>
<p><span class=rvts7>2 3.16</span></p>
<p><span class=rvts7>4 3.15</span></p>
<p><span class=rvts7>8 3.19</span></p>
<p><br></p>
<p><span class=rvts7>Equipment is two quadsocket Opteron Blades (Supermicro) with Mellanox Ex DDR cards. Single 24 port switch connects them.</span></p>
<p><br></p>
<p><span class=rvts7>Can anybody help with suggestions what i can do to lower the latency? </span></p>
<p><span class=rvts7> </span></p>
<p><br></p>
<p><span class=rvts7>Regards, Jan </span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p><br></p>
<p><br></p>
<p><br></p>
<p>Bye Jan </p>
</body></html>