<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><title>Re[2]: [Beowulf] recommendations for cluster upgrades</title>
<META http-equiv=Content-Type content="text/html; charset=utf-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<style type="text/css"><!--
body {
margin: 5px 5px 5px 5px;
background-color: #ffffff;
}
/* ========== Text Styles ========== */
hr { color: #000000}
body, table /* Normal text */
{
font-size: 9pt;
font-family: 'Courier New';
font-style: normal;
font-weight: normal;
color: #000000;
text-decoration: none;
}
span.rvts1 /* Heading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #0000ff;
}
span.rvts2 /* Subheading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #000080;
}
span.rvts3 /* Keywords */
{
font-size: 10pt;
font-family: 'Arial';
font-style: italic;
color: #800000;
}
a.rvts4, span.rvts4 /* Jump 1 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
a.rvts5, span.rvts5 /* Jump 2 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
span.rvts6
{
font-size: 8pt;
font-family: 'tahoma';
}
a.rvts7, span.rvts7
{
font-size: 8pt;
font-family: 'tahoma';
color: #0000ff;
text-decoration: underline;
}
span.rvts8
{
font-size: 8pt;
font-family: 'tahoma';
color: #888888;
}
/* ========== Para Styles ========== */
p,ul,ol /* Paragraph Style */
{
text-align: left;
text-indent: 0px;
padding: 0px 0px 0px 0px;
margin: 0px 0px 0px 0px;
}
.rvps1 /* Centered */
{
text-align: center;
}
--></style>
</head>
<body>
<p>Hallo Tiago,</p>
<p><br></p>
<p>Sonntag, 17. Mai 2009, meintest Du:</p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2 style="background-color: #ffffff;">
<tr valign=top>
<td width=2 style="background-color: #0000ff;"><br>
</td>
<td width=1683>
<p><span class=rvts6>On Sat, May 16, 2009 at 11:56 PM, Rahul Nabar <</span><a class=rvts7 href="mailto:rpnabar@gmail.com">rpnabar@gmail.com</a><span class=rvts6>> wrote:</span></p>
<p><br></p>
<p><span class=rvts6>On Sat, May 16, 2009 at 2:34 PM, Tiago Marques <</span><a class=rvts7 href="mailto:a28427@ua.pt">a28427@ua.pt</a><span class=rvts6>> wrote:</span></p>
<p><span class=rvts6>> One of the codes, VASP, is very bandwidth limited and loves to run in a</span></p>
<p><span class=rvts6>> number of cores multiple of 3. The 5400s are also very bandwith - memory and</span></p>
<p><span class=rvts6>> FSB - limited which causes that they sometimes don't scale well above 6</span></p>
<p><span class=rvts6>> cores. They are very fast per core, as someone mentioned, when compared to</span></p>
<p><span class=rvts6>> AMD cores.</span></p>
<p><br></p>
<p><span class=rvts6>Thanks Tiago. This is super useful info. VASP is one of our major</span></p>
<p><span class=rvts6>"users" too. Possibly 40% of the cpu-time. Rest is a similar</span></p>
<p><span class=rvts6>computational chemistry code, DACAPO.</span></p>
<p><br></p>
<p><span class=rvts6>It would be interesting to compare my test-run times on our</span></p>
<p><span class=rvts6>AMD-Opterons (Barcelona). Is is possible to share what your benchmark</span></p>
<p><span class=rvts6>job was?</span></p>
<p><br></p>
<p><span class=rvts6>I'll try to talk to the user who crafted it for me before, but it should be no problem to pass it to you after.</span></p>
<p><br></p>
<p><span class=rvts6> </span></p>
<p><br></p>
<p><span class=rvts6>Since you mention VASP is bandwidth limited do you mean memory</span></p>
<p><span class=rvts6>bandwidth or the interconnect? Maybe this question itself is naiive.</span></p>
<p><span class=rvts6>Not sure. What interconnect do you use? We have gigabit ethernet dual</span></p>
<p><span class=rvts6>bonded.</span></p>
<p><br></p>
<p><span class=rvts6>Memory bandwith, as you can see by the performance gain from going to 1600MHz from 1066, even with looser timings IIRC.</span></p>
<p><span class=rvts6>Of course interconnects also play a role, even internal ones, which in the case of Xeons was a very slow FSB.</span></p>
<p><br></p>
<p><span class=rvts6>I use single GbE because for as much as I could benchmark, I hardly found anything that could use more than one node efficiently and no one - not even here - could help me with that. Seems I need infiband. </span></p>
<p><span class=rvts6>I only managed to increase 33% with two nodes when using a really huge job(+100k atoms) on Gromacs.</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>For VASP you should look for ConnectX or InfiniPath. InfiniHost III scales badly for the scenarios i saw. It is probably because of the use of collectives. </p>
<p><br></p>
<div><table border=0 cellpadding=1 cellspacing=2 style="background-color: #ffffff;">
<tr valign=top>
<td width=2 style="background-color: #0000ff;"><br>
</td>
<td width=1683>
<p><span class=rvts6> Which brings to a point that I forgot to mention to you. When considering Intel machines, you can always get a compiler license for $2000, give or take,</span></p>
</td>
</tr>
</table>
</div>
<p><br></p>
<p>2000 USD sounds rather expensive. Node locked licenses are usually cheaper... Look for the package with Compilers, MKL and MPI - the Cluster Toolkit. Is definitely worth it (when buying more than just a single machine).</p>
<p><br></p>
<p><br></p>
<p>Jan </p>
</body></html>