<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; ">We have a 128 node cluster running redhat/rocks comprised of dell 1850s with dual 3.4 GHz Xeons and 2 GB memory each. The interconnect is a 4x infiniband nonblocking fabric. HPL was built using Intel's mplpk MPP distribution which links the topspin mpich with the intel optimized em64t blas routines. When running hpl, we found that we were able to get decent but not great performance and we seem to be limited by problem size. We can reach about 1.1 Tflops with a full 256 processor run but the problem size where swap space begins to be utilized is very small, around 80K. With N=120K we are using all memory (real+virtual = 4MB) and the program crashes. Theoretically, with 256 GBytes of memory we should be able to use a problem size of around 150K, assuming the OS uses about 1/4 of the RAM. Similar clusters on the top500 list are able to obtain closer to 1.3 TBytes with a NMAX of around 150K. <DIV><BR class="khtml-block-placeholder"></DIV><DIV> Any ideas? <DIV><BR class="khtml-block-placeholder"></DIV><DIV>Thanks</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>-Geoff</DIV><BR><BR><DIV> <SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</P><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">Geoffrey W. Cowles, Research Scientist</P><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">School for Marine Science and Technology phone: (508) 910-6397</P><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">University of Massachusetts, Dartmouth fax: (508) 910-6371</P><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">706 Rodney French Blvd. email: gcowles<A href="mailto:msundermeyer@umassd.edu"><SPAN class="Apple-style-span" style="color: rgb(0, 0, 238); -khtml-text-decorations-in-effect: underline; ">@umassd.edu</SPAN></A></P><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">New Bedford, MA 02744-1221 <A href="http://www.smast.umassd.edu/msundermeyer/"><SPAN class="Apple-style-span" style="color: rgb(0, 0, 238); -khtml-text-decorations-in-effect: underline; ">http://codfish.smast.umassd.edu/</SPAN></A></P><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</P><BR class="Apple-interchange-newline"></SPAN> </DIV><BR></DIV></BODY></HTML>