[Beowulf] MLNX_OFED vs. rdma-core vs. MLNX_OFED with rdma-core

[Janne Blomqvist]

Hello,

scouring the release notes for the latest MLNX_OFED (version 
4.6-1.0.1.1, and no, still no RHEL 7.7 support), I read a note about an 
upcoming API change at 
https://docs.mellanox.com/display/MLNXOFEDv461000/Changes+and+New+Features

"As of MLNX_OFED v5.0 release (Q1 of the year 2020), the following 
MLNX_OFED Verbs API will migrate from the legacy version of user space 
verbs libraries (libibervs, libmlx5, etc.) to the Upstream version 
rdma-core.
For further details on how to install Upstream rdma-core libraries, 
refer to Installing Upstream rdma-core Libraries section in the User 
Manual."

And in the link how to install with upstream rdma-core there are 
instructions to use either

/mlnxofedinstall --upstream-libs

or if using the yum repo, one should use the RPMS_UPSTREAM_LIBS subdir 
rather than RPMS. Looking at the RPMS_UPSTREAM_LIBS subdir, it seems it 
has only a small subset of the components available in the full 
installation, apparently with the expectation that all the rest will be 
installed via the upstream repos.

Does anybody know more about this? Do I read it correctly that the 
RPMS_UPSTREAM_LIBS will be the only one available as of MLNX_OFED 5.0? 
And thus MLNX_OFED will become a much "thinner" add-on than currently?

Has anyone tested these different configurations, if there's any 
difference in performance and/or functionality?

1. Distro RDMA stack (rdma-core)
2. MLNX_OFED full
3. MLNX_OFED RPMS_UPSTREAM_LIBS

-- 
Janne Blomqvist, D.Sc. (Tech.), Scientific Computing Specialist
Aalto University School of Science, PHYS & NBE
+358503841576 || janne.blomqvist at aalto.fi