<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><title></title>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Style-Type" content="text/css">
<style type="text/css"><!--
body {
margin: 5px 5px 5px 5px;
background-color: #ffffff;
}
/* ========== Text Styles ========== */
hr { color: #000000}
body, table /* Normal text */
{
font-size: 9pt;
font-family: 'Courier New';
font-style: normal;
font-weight: normal;
color: #000000;
text-decoration: none;
}
span.rvts1 /* Heading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #0000ff;
}
span.rvts2 /* Subheading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #000080;
}
span.rvts3 /* Keywords */
{
font-size: 10pt;
font-family: 'Arial';
font-style: italic;
color: #800000;
}
a.rvts4, span.rvts4 /* Jump 1 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
a.rvts5, span.rvts5 /* Jump 2 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
span.rvts6
{
font-weight: bold;
color: #800000;
}
span.rvts7
{
font-weight: bold;
color: #800080;
}
/* ========== Para Styles ========== */
p,ul,ol /* Paragraph Style */
{
text-align: left;
text-indent: 0px;
padding: 0px 0px 0px 0px;
margin: 0px 0px 0px 0px;
}
.rvps1 /* Centered */
{
text-align: center;
}
--></style>
</head>
<body>
<p>Hallo Joe,</p>
<p><br></p>
<p>Dienstag, 26. August 2008, meintest Du:</p>
<p><br></p>
<p><span class=rvts7>>> Questions>> node.</span></p>
<p><span class=rvts7>:</span></p>
<p><br></p>
<p><span class=rvts7>>> 1) What is the cost-effective yet efficient way to connect this cluster </span></p>
<p><span class=rvts7>>> with IB?</span></p>
<p><br></p>
<p><span class=rvts6>JL> Understand that cost-effective is not necessarily the highest</span></p>
<p><span class=rvts6>JL> performance. This is where over-subscription comes in.</span></p>
<p><br></p>
<p><span class=rvts7>>> 2) How many switches are required, and of which size?</span></p>
<p><br></p>
<p><span class=rvts6>JL> With the new 36 port switch chips from Mellanox, you should need 1. For</span></p>
<p><span class=rvts6>JL> a reasonable oversubscription (16 down, 8 up), you would need 3 switches</span></p>
<p><span class=rvts6>JL> ... 1 master and 2 leaf switches.</span></p>
<p><br></p>
<p>2 Switches should be enough - why would you need a master Switch? </p>
<p>
<br>And you could do better:</p>
<p><br></p>
<p>2 Switches with each 13 down and 11 uplinks where the Uplink-Ports are directly connected with each other. So you have exactly the number of ports you need and an oversubscription of 1 : 1.2</p>
<p><br></p>
<p>And if you take into account that a PCIe gen1 IB NIC is capable of handling 1400 MB/s and a DDR-IB Link gives you something like 1800 MB/s you have 13 downlinks with 1400 MB/s each = 18200 MB/s and 11 uplinks with 1800 MB/s = 19800 MB/s - so no oversupscription at all!</p>
<p><br></p>
<p>(The last point is just a quick idea that came to my mind - probably not well thought through... maybe someone could add something ;-) )</p>
<p><br></p>
<p>Regards,</p>
<p>Jan Heichler</p>
</body></html>