[Beowulf] MLNX_OFED vs. rdma-core vs. MLNX_OFED with rdma-core

[Janne Blomqvist]

On 23/09/2019 11.16, Janne Blomqvist wrote:
> Hello,
> 
> scouring the release notes for the latest MLNX_OFED (version
> 4.6-1.0.1.1, and no, still no RHEL 7.7 support), I read a note about an
> upcoming API change at
> https://docs.mellanox.com/display/MLNXOFEDv461000/Changes+and+New+Features
> 
> "As of MLNX_OFED v5.0 release (Q1 of the year 2020), the following
> MLNX_OFED Verbs API will migrate from the legacy version of user space
> verbs libraries (libibervs, libmlx5, etc.) to the Upstream version
> rdma-core.
> For further details on how to install Upstream rdma-core libraries,
> refer to Installing Upstream rdma-core Libraries section in the User
> Manual."
> 
> And in the link how to install with upstream rdma-core there are
> instructions to use either
> 
> /mlnxofedinstall --upstream-libs
> 
> or if using the yum repo, one should use the RPMS_UPSTREAM_LIBS subdir
> rather than RPMS. Looking at the RPMS_UPSTREAM_LIBS subdir, it seems it
> has only a small subset of the components available in the full
> installation, apparently with the expectation that all the rest will be
> installed via the upstream repos.
> 
> Does anybody know more about this? Do I read it correctly that the
> RPMS_UPSTREAM_LIBS will be the only one available as of MLNX_OFED 5.0?
> And thus MLNX_OFED will become a much "thinner" add-on than currently?
> 
> Has anyone tested these different configurations, if there's any
> difference in performance and/or functionality?
> 
> 1. Distro RDMA stack (rdma-core)
> 2. MLNX_OFED full
> 3. MLNX_OFED RPMS_UPSTREAM_LIBS
> 

Not to be outdone, the recently released MLNX_OFED 4.7 has reorganized
this slightly, and now there's yet another option. So,

1. Distro RDMA stack (rdma-core)
2. MLNX_OFED full (directory RPMS/MLNX_LIBS)
3. MLNX_OFED upstream libs 1 (directory RPMS/UPSTREAM_LIBS)
4. MLNX_OFED upstream libs 2 (directory RPMS_UPSTREAM_LIBS)

The difference between options 3 and 4 is that 4 is more bare bones,
e.g. 3 includes the MLNX opensm whereas with 4 you're expected to use
the distro opensm.

-- 
Janne Blomqvist, D.Sc. (Tech.), Scientific Computing Specialist
Aalto University School of Science, PHYS & NBE
+358503841576 || janne.blomqvist at aalto.fi