<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>HI Faraz</div><div><br></div><div>My impression is that you're missing the IB libraries, and that Open MPI <br></div><div>was not built with IB support.</div><div>This is very likely to be the case if you're using the Open MPI packages from CentOS (openmpi3.x86_64, openmpi3-devel.x86_64)<br></div><div>which probably only have TCP/IP support built in (the common denominator network of most computers).<br></div><div>Building Open MPI from source is not difficult, and a must if you have IB cards.</div><div><br></div><div>Notwithstanding the MPI expertise of the Beowulf mailing list subscribers, <br></div><div>if you post your message in the Open MPI mailing list, you'll get specific and detailed help in no time,</div><div>and minimize the suffering.</div><div><br></div><div>My two cents,<br></div><div>Gus Correa<br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 30, 2019 at 12:28 PM Faraz Hussain <<a href="mailto:info@feacluster.com" target="_blank">info@feacluster.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks, here is the output below:<br>
<br>
[hussaif1@lustwzb34 ~]$ ompi_info<br>
[lustwzb34:10457] mca_base_component_repository_open: unable to open <br>
mca_btl_usnic: libpsm_infinipath.so.1: cannot open shared object file: <br>
No such file or directory (ignored)<br>
[lustwzb34:10457] mca_base_component_repository_open: unable to open <br>
mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: <br>
No such file or directory (ignored)<br>
[lustwzb34:10457] mca_base_component_repository_open: unable to open <br>
mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: <br>
No such file or directory (ignored)<br>
Package: Open MPI <a href="mailto:mockbuild@x86-041.build.eng.bos.redhat.com" target="_blank">mockbuild@x86-041.build.eng.bos.redhat.com</a><br>
Distribution<br>
Open MPI: 3.0.2<br>
Open MPI repo revision: v3.0.2<br>
Open MPI release date: Jun 01, 2018<br>
Open RTE: 3.0.2<br>
Open RTE repo revision: v3.0.2<br>
Open RTE release date: Jun 01, 2018<br>
OPAL: 3.0.2<br>
OPAL repo revision: v3.0.2<br>
OPAL release date: Jun 01, 2018<br>
MPI API: 3.1.0<br>
Ident string: 3.0.2<br>
Prefix: /usr/lib64/openmpi3<br>
Configured architecture: x86_64-unknown-linux-gnu<br>
Configure host: <a href="http://x86-041.build.eng.bos.redhat.com" rel="noreferrer" target="_blank">x86-041.build.eng.bos.redhat.com</a><br>
Configured by: mockbuild<br>
Configured on: Wed Jun 13 14:18:03 EDT 2018<br>
Configure host: <a href="http://x86-041.build.eng.bos.redhat.com" rel="noreferrer" target="_blank">x86-041.build.eng.bos.redhat.com</a><br>
Configure command line: '--prefix=/usr/lib64/openmpi3'<br>
'--mandir=/usr/share/man/openmpi3-x86_64'<br>
'--includedir=/usr/include/openmpi3-x86_64'<br>
'--sysconfdir=/etc/openmpi3-x86_64'<br>
'--disable-silent-rules' '--enable-builtin-atomics'<br>
'--enable-mpi-cxx' '--with-sge' '--with-valgrind'<br>
'--enable-memchecker' '--with-hwloc=/usr' 'CC=gcc'<br>
'CXX=g++' 'LDFLAGS=-Wl,-z,relro ' 'CFLAGS= -O2 -g<br>
-pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions<br>
-fstack-protector-strong --param=ssp-buffer-size=4<br>
-grecord-gcc-switches -m64 -mtune=generic'<br>
'CXXFLAGS= -O2 -g -pipe -Wall<br>
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions<br>
-fstack-protector-strong --param=ssp-buffer-size=4<br>
-grecord-gcc-switches -m64 -mtune=generic'<br>
'FC=gfortran' 'FCFLAGS= -O2 -g -pipe -Wall<br>
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions<br>
-fstack-protector-strong --param=ssp-buffer-size=4<br>
-grecord-gcc-switches -m64 -mtune=generic'<br>
Built by: mockbuild<br>
Built on: Wed Jun 13 14:25:02 EDT 2018<br>
Built host: <a href="http://x86-041.build.eng.bos.redhat.com" rel="noreferrer" target="_blank">x86-041.build.eng.bos.redhat.com</a><br>
C bindings: yes<br>
C++ bindings: yes<br>
Fort mpif.h: yes (all)<br>
Fort use mpi: yes (limited: overloading)<br>
Fort use mpi size: deprecated-ompi-info-value<br>
Fort use mpi_f08: no<br>
Fort mpi_f08 compliance: The mpi_f08 module was not built<br>
Fort mpi_f08 subarrays: no<br>
Java bindings: no<br>
Wrapper compiler rpath: runpath<br>
C compiler: gcc<br>
C compiler absolute: /usr/bin/gcc<br>
C compiler family name: GNU<br>
C compiler version: 4.8.5<br>
C++ compiler: g++<br>
C++ compiler absolute: /usr/bin/g++<br>
Fort compiler: gfortran<br>
Fort compiler abs: /usr/bin/gfortran<br>
Fort ignore TKR: no<br>
Fort 08 assumed shape: no<br>
Fort optional args: no<br>
Fort INTERFACE: yes<br>
Fort ISO_FORTRAN_ENV: yes<br>
Fort STORAGE_SIZE: no<br>
Fort BIND(C) (all): no<br>
Fort ISO_C_BINDING: yes<br>
Fort SUBROUTINE BIND(C): no<br>
Fort TYPE,BIND(C): no<br>
Fort T,BIND(C,name="a"): no<br>
Fort PRIVATE: no<br>
Fort PROTECTED: no<br>
Fort ABSTRACT: no<br>
Fort ASYNCHRONOUS: no<br>
Fort PROCEDURE: no<br>
Fort USE...ONLY: no<br>
Fort C_FUNLOC: no<br>
Fort f08 using wrappers: no<br>
Fort MPI_SIZEOF: no<br>
C profiling: yes<br>
C++ profiling: yes<br>
Fort mpif.h profiling: yes<br>
Fort use mpi profiling: yes<br>
Fort use mpi_f08 prof: no<br>
C++ exceptions: no<br>
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,<br>
OMPI progress: no, ORTE progress: yes, Event lib:<br>
yes)<br>
Sparse Groups: no<br>
Internal debug support: no<br>
MPI interface warnings: yes<br>
MPI parameter check: runtime<br>
Memory profiling support: no<br>
Memory debugging support: no<br>
dl support: yes<br>
Heterogeneous support: no<br>
mpirun default --prefix: no<br>
MPI I/O support: yes<br>
MPI_WTIME support: native<br>
Symbol vis. support: yes<br>
Host topology support: yes<br>
MPI extensions: affinity, cuda<br>
FT Checkpoint support: no (checkpoint thread: no)<br>
C/R Enabled Debugging: no<br>
MPI_MAX_PROCESSOR_NAME: 256<br>
MPI_MAX_ERROR_STRING: 256<br>
MPI_MAX_OBJECT_NAME: 64<br>
MPI_MAX_INFO_KEY: 36<br>
MPI_MAX_INFO_VAL: 256<br>
MPI_MAX_PORT_NAME: 1024<br>
MPI_MAX_DATAREP_STRING: 128<br>
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA btl: openib (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA btl: self (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA crs: none (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA memchecker: valgrind (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component<br>
v3.0.2)<br>
MCA pmix: pmix2x (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v3.0.2)<br>
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA dfs: app (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA dfs: orted (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA dfs: test (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component<br>
v3.0.2)<br>
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component<br>
v3.0.2)<br>
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component<br>
v3.0.2)<br>
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component<br>
v3.0.2)<br>
MCA errmgr: dvm (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component<br>
v3.0.2)<br>
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA notifier: syslog (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA oob: ud (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA routed: debruijn (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA state: app (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA state: dvm (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v3.0.2)<br>
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA fcoll: static (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v3.0.2)<br>
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v3.0.2)<br>
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component<br>
v3.0.2)<br>
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v3.0.2)<br>
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component<br>
v3.0.2)<br>
<br>
<br>
Quoting John Hearns <<a href="mailto:hearnsj@googlemail.com" target="_blank">hearnsj@googlemail.com</a>>:<br>
<br>
> Hello Faraz. Please start by running this command ompi_info<br>
><br>
> On Tue, 30 Apr 2019 at 15:15, Faraz Hussain <<a href="mailto:info@feacluster.com" target="_blank">info@feacluster.com</a>> wrote:<br>
><br>
>> I installed RedHat 7.5 on two machines with the following Mellanox cards:<br>
>><br>
>> 87:00.0 Network controller: Mellanox Technologies MT27520 Family<br>
>> [ConnectX-3 Pro<br>
>><br>
>> I followed the steps outlined here to verify RDMA is working:<br>
>><br>
>><br>
>> <a href="https://community.mellanox.com/s/article/howto-enable-perftest-package-for-upstream-kernel" rel="noreferrer" target="_blank">https://community.mellanox.com/s/article/howto-enable-perftest-package-for-upstream-kernel</a><br>
>><br>
>> However, I cannot seem to get Open MPI 3.0.2 to work. When I run it, I<br>
>> get this error:<br>
>><br>
>> --------------------------------------------------------------------------<br>
>><br>
>> No OpenFabrics connection schemes reported that they were able to be<br>
>><br>
>> used on a specific port. As such, the openib BTL (OpenFabrics<br>
>><br>
>> support) will be disabled for this port.<br>
>><br>
>><br>
>> Local host: lustwzb34<br>
>><br>
>> Local device: mlx4_0<br>
>><br>
>> Local port: 1<br>
>><br>
>> CPCs attempted: rdmacm, udcm<br>
>><br>
>> --------------------------------------------------------------------------<br>
>><br>
>> Then it just hangs till I press control C.<br>
>><br>
>> I understand this may be an issue with RedHat, Open MPI or Mellanox.<br>
>> Any ideas to debug which place it could be?<br>
>><br>
>> Thanks!<br>
>><br>
>> _______________________________________________<br>
>> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
>> To change your subscription (digest mode or unsubscribe) visit<br>
>> <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
>><br>
<br>
<br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote></div>