<html>
<body>
<font size=3>Following Jeff Layton's post to this list [Cheap SDR IB] on
January 28,<br>
we purchased 8 Infinihost LX's and an 8 port switch, and began trying to
get<br>
the OpenFabrics (OFED) release of MVAPICH for Fedora Core 6 to run on our
new<br>
machines. We develop and run a multiphysics code in a relatively
fine grain parallel mode<br>
where latency dominates the performance scaling, so it seemed like a good
thing to try.<br><br>
This is our first exposure to InfiniBand, though we have
considerable<br>
experience with MPI, both in-memory and over GigE, including using
netpipe to<br>
measure latency and bandwidth.<br><br>
Those machines have AMD Athlon X2 6000+'s on Asus M2N-SLI Deluxe
motherboards<br>
with an open PCI Express slot that will handle x4.<br><br>
The main issue is that we are presently running Fedora Core 8 and the
2.6.21<br>
SMP kernel, but there is no OFED release for FC8 yet. Is anyone
else working<br>
on this? Has anyone succeeded at getting it to work?<br><br>
We started with OFED version 1.2.5 from <br>
<a href="http://www.openfabrics.org/downloads/OFED/ofed-1.2.5/OFED-1.2.5-RPMS/" eudora="autourl">
http://www.openfabrics.org/downloads/OFED/ofed-1.2.5/OFED-1.2.5-RPMS/<br>
</a>We downloaded all the rpms from redhat-release-4AS-6.1 version.<br>
In particular the kernel rpms are kernel-ib-devel-1.2-2.6.9_55.ELsmp
and<br>
kernel-ib-1.2-2.6.9_55.ELsmp.<br><br>
We used the 1.2.5 version because there don't seem to be any rpms for the
1.3 version. <br><br>
All the OFED rpm's for FC6 installed on FC8 without difficulty, except
for opensm-3.0.3-0.ppc64.rpm<br>
It didn't say "missing dependencies ..." It just got
stuck. We had to kill the 'rpm -ivh', remove the lock file <br>
and rebuild the rpm database. After that,<br><br>
# lsmod | grep ib<br><br>
shows about 15 IB related kernel mods.<br><br>
Even so, at this point, some of the IB stuff works. We can run
ibnetdiscover and see the HCA's on the<br>
two machines that have the rpm's installed, and the switch, too. We
could use<br>
that to make a topology file, but we don't know where to put it, or even
if we<br>
should put it somewhere. We can run ibchecknet, and though it finds 4
nodes,<br>
it says they are all bad. It also reports "lid 0 address
resolution:<br>
FAILED". We have not succeeded in getting ibping to work, and
aren't really<br>
sure what how to specify the remote address for it.<br><br>
We found <br><br>
/usr/share/doc/ofed-docs-1.2/README.txt<br>
/usr/share/doc/ofed-docs-1.2/OFED_Installation_Guide.txt<br><br>
and, as described there, did<br><br>
# /etc/init.d/openibd
start
<br>
Loading QLogic InfiniPath
driver:
[FAILED]<br>
Loading HCA driver and Access
Layer:
[ OK ]<br>
Setting up InfiniBand network interfaces:<br>
Failed to configure IPoIB connected mode for ib0<br>
Bringing up interface
ib0:
[FAILED]<br>
Setting up service network . .
.
[ done ]<br>
Loading
ib_sdp
[FAILED]<br>
Loading
ib_vnic
[FAILED]<br>
Module ib_vnic not loaded.<br>
Bringing up VNIC
interfaces
[FAILED]<br><br>
That mostly looks bad.<br><br>
Does anyone have any suggestions?<br><br>
We are willing to try a build from source, but we are unsure of what
challenges might lie down that path.<br><br>
We'd rather not fall back to FC6, but we may have to do that.<br><br>
Thanks for your help.<br><br>
<br>
Mike Frese <br>
</font></body>
</html>