<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><title></title>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Style-Type" content="text/css">
<style type="text/css"><!--
body {
margin: 5px 5px 5px 5px;
background-color: #ffffff;
}
/* ========== Text Styles ========== */
hr { color: #000000}
body, table /* Normal text */
{
font-size: 9pt;
font-family: 'Courier New';
font-style: normal;
font-weight: normal;
color: #000000;
text-decoration: none;
}
span.rvts1 /* Heading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #0000ff;
}
span.rvts2 /* Subheading */
{
font-size: 10pt;
font-family: 'Arial';
font-weight: bold;
color: #000080;
}
span.rvts3 /* Keywords */
{
font-size: 10pt;
font-family: 'Arial';
font-style: italic;
color: #800000;
}
a.rvts4, span.rvts4 /* Jump 1 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
a.rvts5, span.rvts5 /* Jump 2 */
{
font-size: 10pt;
font-family: 'Arial';
color: #008000;
text-decoration: underline;
}
span.rvts6
{
font-weight: bold;
color: #800000;
}
span.rvts7
{
font-weight: bold;
color: #800080;
}
a.rvts8, span.rvts8
{
color: #0000ff;
text-decoration: underline;
}
span.rvts9
{
font-weight: bold;
color: #800080;
}
/* ========== Para Styles ========== */
p,ul,ol /* Paragraph Style */
{
text-align: left;
text-indent: 0px;
padding: 0px 0px 0px 0px;
margin: 0px 0px 0px 0px;
}
.rvps1 /* Centered */
{
text-align: center;
}
--></style>
</head>
<body>
<p>Hallo Daniel,</p>
<p><br></p>
<p>Donnerstag, 24. Juli 2008, meintest Du:</p>
<p><br></p>
<p>[network configurations]</p>
<p><br></p>
<p>I have to say i am not sure that all the configs you sketched really work. I never saw somebody creating loops in an IB fabric. </p>
<p><br></p>
<p><span class=rvts6>DP> Since I am not network expert I would be glad if somebody explains</span></p>
<p><span class=rvts6>DP> why the first solution is the best one.</span></p>
<p><br></p>
<p>Let's say it as follows:</p>
<p><br></p>
<p>1) most applications are latency driven - not bandwidth driven. That means that half bisectional bandwidth is not cutting your application performance down to 50%. For most applications the impact should be less than 5% - for some it is really 0%.</p>
<p><br></p>
<p>2) Static routing in IB networks limits your bandwidth for many of the possible communication patterns anyway. For completely random communication it was like below 50%. So you buy a IB fabric with full bisectional but can't use it anyway - reducing the bisectional bandwidth is not impacting that much anymore (as far as i understood most whitepapers)</p>
<p><br></p>
<p>3) today you have usually 4 or 8 cores in one node. 12 nodes times 4/8 cores makes 48 or 92 cores that are connected with one HOP on the same switch. Many applications don't scale to that number of processes anyway. Before you try to think about optimizing the network to the maximum maybe it is better to think about your application, your ususal job sizes and the scheduling of the jobs. Try to avoid "cross switch communication" if possible. If you run small jobs like let's say of 8 nodes and you have 12 nodes on each switch and half bisectional bandwidth between them then it is 8 nodes on the first switch for job 1. For job 2 it is 4 nodes on switch one and 4 on switch two. Your bisectional bandwidth is big enough to handle this.</p>
<p><br></p>
<p>I vote for the fat tree in picture one because i know it works and with 1) to 3) mentioned above it will give you good performance - especially if you run more than just one application (because optimizing is mostly optimizing for a single use case - if you have more than one it is hard to optimize).</p>
<p><br></p>
<p>Regards,</p>
<p>Jan</p>
<p><br></p>
</body></html>