[Beowulf] IB troubles - mca_mpool_openib_register

[Lombard, David N]

More memory in your nodes?  Not sure what size of queues and such
openmpi allocates, but you could simply be running out of memory if
openmpi allocates large queue depths.

Have you tried an alternate MPI to see if you have the same problem?
Intel MPI, MVAPICH, MVAPICH2, as well as others support OpenIB.

Can you consider moving to a newer version of OpenIB?

--
David N. Lombard

My statements represent my opinions, not those of Intel Corporation 

> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
On
> Behalf Of Bill Wichser
> Sent: Thursday, June 22, 2006 6:02 AM
> To: beowulf at beowulf.org
> Subject: [Beowulf] IB troubles - mca_mpool_openib_register
> 
> 
> Cluster with dual Xeons and Topsping IB adapters running a RH
> 2.6.9-34.ELsmp kernel (x86_64) with the RH IB stack installed, each
node
> w/8G of memory.
> 
> Updated firmware as per Mellanox in the IB cards.
> 
> Updates /etc/security/limits.conf to have memlock be 8192, both soft
and
> hard limits to overcome the initial trouble of pool allocation.
> 
> Application is cpi.c.
> 
> I can run across the 64 nodes using nodes=64:ppn=1 without trouble,
> except for the
> 
> [btl_openib_endpoint.c:889:mca_btl_openib_endpoint_create_qp]
ibv_create
> _qp: returned 0 byte(s) for max inline data
> 
> error messages, to be fixed I suppose in the next release.  These I
can
> live with, perhaps, for now.
> 
> The problem is that when I run with nodes=64:ppn=2 and only use -np 64
> with my openmpi (v 1.0.2 gcc compiled), it still runs fine, but when I
> run with -np 65 I get megabytes of error messages and the job never
> completes.  The errors all look like this:
> 
> mca_mpool_openib_register: ibv_reg_mr(0x2a96641000,1060864)
> failed with error: Cannot allocate memory
> 
> I've submitted to the openib-general mailing list with no responses.
I'm
> not sure if this is an openmpi problem, an openib problem, or some
> configuration problem with the IB fabric.  Other programs fail with
even
> less processors being allocated with these same errors.  Running over
> TCP, albeit across the GigE network and not over IB, works fine.
> 
> I'm stuck here not knowing how to proceed.  Has anyone found this
issue
> and, more importantly, found a solution?  I don't believe it to be a
> limits.conf issue as I can allocate both processors on a node up to 32
> nodes (-np 64) without problems.
> 
> Thanks,
> Bill
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf