<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><title>Re: [Beowulf] Re: scratch File system for small cluster</title>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Style-Type" content="text/css">
<style type="text/css"><!--
body {
margin: 5px 5px 5px 5px;
background-color: #ffffff;
}
/* ========== Text Styles ========== */
hr { color: #000000}
body, table /* Normal text */
{
font-size: 9pt;
font-family: 'Courier New';
font-style: normal;
font-weight: normal;
color: #000000;
text-decoration: none;
}
span.rvts1 /* Heading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #0000ff;
}
span.rvts2 /* Subheading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #000080;
}
span.rvts3 /* Keywords */
{
font-size: 10pt;
font-family: 'Arial';
font-style: italic;
color: #800000;
}
a.rvts4, span.rvts4 /* Jump 1 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
a.rvts5, span.rvts5 /* Jump 2 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
span.rvts6
{
font-size: 8pt;
font-family: 'tahoma';
}
a.rvts7, span.rvts7
{
font-size: 8pt;
font-family: 'tahoma';
color: #0000ff;
text-decoration: underline;
}
span.rvts8
{
font-size: 10pt;
font-family: 'tahoma';
}
a.rvts9, span.rvts9
{
font-size: 10pt;
font-family: 'tahoma';
text-decoration: underline;
}
/* ========== Para Styles ========== */
p,ul,ol /* Paragraph Style */
{
text-align: left;
text-indent: 0px;
padding: 0px 0px 0px 0px;
margin: 0px 0px 0px 0px;
}
.rvps1 /* Centered */
{
text-align: center;
}
--></style>
</head>
<body>
<p>Hallo Greg,</p>
<p><br></p>
<p>Donnerstag, 25. September 2008, meintest Du:</p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2 style="background-color: #ffffff;">
<tr valign=top>
<td width=2 style="background-color: #0000ff;"><br>
</td>
<td width=1467>
<p><span class=rvts6>Glen,</span></p>
<p><br></p>
<p><span class=rvts6>I have had great success with the *right* 10GbE nic and NFS. The important things to consider are:</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>I have to say my experience was different. </p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2 style="background-color: #ffffff;">
<tr valign=top>
<td width=2 style="background-color: #0000ff;"><br>
</td>
<td width=1467>
<p><br></p>
<p><span class=rvts6>How much bandwidth will your backend storage provide? 2 x FC 4 I'm guessing best case is 600Mb but likely less.</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>600 MB/s is already a good value for a SAN-based Storage ;-)</p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2 style="background-color: #ffffff;">
<tr valign=top>
<td width=2 style="background-color: #0000ff;"><br>
</td>
<td width=1467>
<p><span class=rvts6>What access patterns do the "typical apps" have? </span></p>
<p><span class=rvts6>All nodes read from a single file (no prob for NFS, and fscache may help even more) </span></p>
<p><span class=rvts6>All nodes write to a single file (NFS may need some help or may be too slow when tuned for this)</span></p>
<p><span class=rvts6>All nodes read and write to separate files (NFS is fine if the files aren't too big for the OS to cache reasonably).</span></p>
<p><br></p>
<p><span class=rvts6>The number of IO servers really is a function of how much disk throughput you have on the backend, frontend, and through the kernel/filesystem goo. My experience is a 10GbE nic from Myricom can easily sustain 500-700MB/s if the storage behind it can and the access patterns aren't evil. Other nics</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>My experience was this: you get app. half of what you have on blockdevice-level to the network. So i had a setup with 16 x 15k rpm SAS drives. RAID5 on them showed 1.1 GB/s read (limited by PCIe x8 probably) and 550 MB/s write (Controller was LSI 8888ELP). With exporting this to a number of clients i was not able to get more than app. 500 MB/s read and 400 MB/s write with multiple clients. I could show the real measurements if that is of interest. </p>
<p><br></p>
<p>If you look at the hardware that was thrown on the problem the result is a little pathetic. </p>
<p><br></p>
<p>My experience with lustre is that it eats up 10 to 15% of the blockdevice-speed. And the rest you have in the network.</p>
<p><br></p>
<p>So a cheap lustre-setup for scratch would probably include 2 Servers with internal storage and exporting it to the cluster with 10GE or IB. Internal storage is cheap and it is easy to achieve 500+ MB/s on SATA drives. That way you can reach 1 GB/s with just having 2 Servers and 32 to 48 disks involved. </p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2 style="background-color: #ffffff;">
<tr valign=top>
<td width=2 style="background-color: #0000ff;"><br>
</td>
<td width=1467>
<p><span class=rvts6> from large and small vendors can fall apart at 3-4 Gb so be careful and test the network first before assuming your FS is the troublemaker. There are cheap switches with 2 or 4 10GbE CX4 connectors that make this much simpler and safer with or without the Parallel FS options.</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>I never tested anything but Myricom 10GE but you can find cheap Intel-Based cards with CX4 (and i doubt that they are bad) . The Dell PowerConnect 62xx-Series can give you cheap CX4 uplinks - and you get a decent switch that is stackable. </p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2 style="background-color: #ffffff;">
<tr valign=top>
<td width=2 style="background-color: #0000ff;"><br>
</td>
<td width=1467>
<p><br></p>
<p><span class=rvts6>Depending on how big/small and how "scratch" the need is... a big tmpfs/ramdisk can be fun :)</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>I tried once to export tmpfs via NFS - didn't work out of the box.</p>
<p><br></p>
<p>Bye Jan </p>
</body></html>