<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>I use it. It works fine. opensm I believe it is called. UFM has other features that are supposed to be really nice and make monitoring for fabric problems really easy. It's definitely not required to have a working Infiniband setup. <br><br><span style="background-color: rgba(255, 255, 255, 0);">____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*<br>|| \\UTGERS |---------------------*O*---------------------<br>||_// Biomedical | Ryan Novosielski - Senior Technologist<br>|| \\ and Health | <a href="mailto:novosirj@rutgers.edu" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="3">novosirj@rutgers.edu</a>- 973/972.0922 (2x0922)<br>|| \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark<br> `'</span></div><div><br>On Sep 15, 2015, at 13:31, Jörg Saßmannshausen <<a href="mailto:j.sassmannshausen@ucl.ac.uk">j.sassmannshausen@ucl.ac.uk</a>> wrote:<br><br></div><blockquote type="cite"><div><span>Hi Jeff,</span><br><span></span><br><span>no, not yet. </span><br><span>What I want to avoid is: I try the OFED subnet manager and it does not work </span><br><span>and then I have to wait until I get the licence. This project has enough </span><br><span>delays right now and I don't want to add to it. Hence my question.</span><br><span></span><br><span>Having said that: are you happy with the OFED one?</span><br><span></span><br><span>All the best</span><br><span></span><br><span>Jörg</span><br><span></span><br><span>On Dienstag 15 September 2015 Jeff Becker wrote:</span><br><blockquote type="cite"><span>Hi Jörg,</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Have you tried using the subnet manager from Mellanox OFED (which is</span><br></blockquote><blockquote type="cite"><span>free)? That's what we use on our big heterogeneous cluster at NASA.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>HTH</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>-jeff</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>On 09/15/2015 08:55 AM, Jörg Saßmannshausen wrote:</span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span>Dear all,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>I am a bit confused and I was wondering whether somebody on the list</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>could give me a bit of advice here.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>I was previously using QLogic for my QDR InfiniBand network. I got one</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>master switch which got the licence for the InfiniBand installed and</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>things appear to work ok. At least I cannot detect any problems despite</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>adding switches and nodes to the fabric.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Now, we recently purchased a new cluster with 20 cores per node and here</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>I decided to go for FDR to be a bit more future proofed as well. So I</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>got the 'normal' licence from Mellanox for the cluster. I got one</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>licence per node so I assumed that was ok.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Now, we are in the process to set up another cluster with a mixture of</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>older and newer hardware. Again I have decided to opt for the FDR simply</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>to be a bit more future proofed. And this is where the confusion comes</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>in.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Apparently I do need now the UFM (Unified Fibre Manager) from Mellanox to</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>run the InfiniBand. However, the normal licence is only for up to 16</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>cores per node and I would need the more expensive exhanced licence.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> From what I and a colleague of mine can see the UFM is nothing more than</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span> the</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>requires subnet manager plus some diagnostic tools.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>There are two questions here:</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>- do we really need the exhanced UFM licence or is that just a way to</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>make money?</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>- would the open source subnet manage work as well and would the open</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>source diagnostic tools be ok?</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>- why do I need to pay for a licence for each node? Somehow I cannot</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>recall having done that in the past.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Unfortunately, InfiniBand is not my strong side and thus I would</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>appreciate and advice here.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>All the best from a meanwhile sunny London</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Jörg</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>_______________________________________________</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>To change your subscription (digest mode or unsubscribe) visit</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span><a href="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</a></span><br></blockquote></blockquote><span></span><br><span></span><br><span>-- </span><br><span>*************************************************************</span><br><span>Dr. Jörg Saßmannshausen, MRSC</span><br><span>University College London</span><br><span>Department of Chemistry</span><br><span>20 Gordon Street</span><br><span>London</span><br><span>WC1H 0AJ </span><br><span></span><br><span>email: <a href="mailto:j.sassmannshausen@ucl.ac.uk">j.sassmannshausen@ucl.ac.uk</a></span><br><span>web: <a href="http://sassy.formativ.net">http://sassy.formativ.net</a></span><br><span></span><br><span>Please avoid sending me Word or PowerPoint attachments.</span><br><span>See <a href="http://www.gnu.org/philosophy/no-word-attachments.html">http://www.gnu.org/philosophy/no-word-attachments.html</a></span><br></div></blockquote><blockquote type="cite"><div><signature.asc></div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing</span><br><span>To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</a></span><br></div></blockquote></body></html>