<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
Yeah,
<div><br>
</div>
<div>I saw that too, which is part of why I was confused. </div>
<div><br>
</div>
<div>It appears to me as if, at the very least, the kernel driver and OFED have two different recommendations. The CentOS kernel driver (from CentOS 7.6 at least, but likely longer) seems to set log_mtts_per_seg to 3 on all of my nodes. You can see from my
earlier reply that this driver already takes total memory into account. </div>
<div><br>
</div>
<div>OFED on the other hand (I was looking at a VM with OFED 4.5 on it — have to confirm the host version, etc.) seems to set that value to 0 by default. </div>
<div><br>
</div>
<div>So I’m not a ton less confused, but reasonably confident the right move on the CentOS kernel driver is not to set that value as the defaults are sane.</div>
<div><br>
<div id="AppleMailSignature" dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="1">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</span></div>
<div dir="ltr"><br>
On Jul 9, 2019, at 21:25, Jonathan Engwall <<a href="mailto:engwalljonathanthereal@gmail.com">engwalljonathanthereal@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Hello,</div>
<div>They make a recommendation: <a href="https://community.mellanox.com/s/article/howto-increase-memory-size-used-by-mellanox-adapters">
https://community.mellanox.com/s/article/howto-increase-memory-size-used-by-mellanox-adapters</a></div>
<div>interesting stuff.</div>
<div>Jonathan Engwall<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jul 9, 2019 at 1:13 PM Ryan Novosielski <<a href="mailto:novosirj@rutgers.edu">novosirj@rutgers.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi all,<br>
<br>
There seems to be a whole lot of misinformation out there about the appropriate setting for the log_mtts_per_seg parameter in the mlx4 driver. Some folks suggest that it’s different between the RHEL/<a href="http://kernel.org" rel="noreferrer" target="_blank">kernel.org</a>
provided driver and Mellanox OFED, some suggest it’s been fixed such that the default can assign 2x the total memory on a node since at least RHEL6.6 (and so one would assume even longer ago in OFED), and some places seem to still be carrying it forward because
“maybe it matters.” <br>
<br>
Is anyone here sure of the current state? I’ll probably read the source code if not, but I’d like to spare myself the hassle.<br>
<br>
--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" target="_blank">
novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</body>
</html>