[Beowulf] log_mtts_per_seg in mlx4 driver
novosirj at rutgers.edu
Tue Jul 9 13:44:42 PDT 2019
Read the source code/answered my own question. Likely bogus since 2012, confirmed at least in 3.10.0-957.21.3 — this was the relevant fix:
* We want to scale the number of MTTs with the size of the
* system memory, since it makes sense to register a lot of
* memory on a system with a lot of memory. As a heuristic,
* make sure we have enough MTTs to cover twice the system
* memory (with PAGE_SIZE entries).
* This number has to be a power of two and fit into 32 bits
* due to device limitations, so cap this at 2^31 as well.
* That limits us to 8TB of memory registration per HCA with
* 4KB pages, which is probably OK for the next few months.
min(1UL << (31 - log_mtts_per_seg),
si.totalram >> (log_mtts_per_seg - 1))));
> On Jul 9, 2019, at 4:12 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> Hi all,
> There seems to be a whole lot of misinformation out there about the appropriate setting for the log_mtts_per_seg parameter in the mlx4 driver. Some folks suggest that it’s different between the RHEL/kernel.org provided driver and Mellanox OFED, some suggest it’s been fixed such that the default can assign 2x the total memory on a node since at least RHEL6.6 (and so one would assume even longer ago in OFED), and some places seem to still be carrying it forward because “maybe it matters.”
> Is anyone here sure of the current state? I’ll probably read the source code if not, but I’d like to spare myself the hassle.
> || \\UTGERS, |---------------------------*O*---------------------------
> ||_// the State | Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
More information about the Beowulf