<br>Hi Joe,<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I'm beginning to look into configurations for a new cluster and with the AMD 12-core and Intel 8-core chips 'here' (or coming soonish), I'm curious if anyone has any data on the effects of the messaging rate of the IB cards. With a 4-socket node having between 32 and 48 cores, lots of computing can get done fast, possibly stressing the network. <br>
</blockquote>
<br></div>
The big issue will be contention for the resource. As you scale up the number of requesters, if the number of resources don't also scale up (even vitualized non-blocking HCA/NICs are good here), you could hit a problem at some point.</blockquote>
<div class="im"><br> My knowledge of the latest low-level hardware is sadly out of date - does a virtualized non-blocking HCA mean that I can have one HCA which virtualizes into four (one per socket say), and each of those four has its own memory-mapped buffer so that I don't get cache invalidation / contention on multi-socket boxes, or am I totally off-base here? I'm all for scaling up NICs as I scale up cores, but each additional NIC / HCA port means more switch ports, which adds up fast. In fact, if I have a standard 2-socket node now, with 8 cores in it and a DDR IB port, and then get a 2-socket node with 24 cores in it and a QDR IB port,... how's the math work? I've got 3x the cores, 1x the adapters, but that adapter has 2x the speed. Blah. <br>
<br><br>
<blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I know Qlogic has made a big deal about the InfiniPath adapter's extremely good message rate in the past... is this still an important issue? How do the latest Mellanox adapters compare? (Qlogic documents a ~30M messages processsed per second rate on its QLE7342, but I didn't see a number on the Mellanox ConnectX-2... and more to the point, do people see this effecting them?)<br>
</blockquote></blockquote>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
We see this on the storage side. Massive oversubscription of resources leads to contention issues for links, to ib packet requeue failures among other things.</blockquote><div><br> So (ignoring disk latencies and just focusing on link contention), is there any difference between using 2x the storage nodes or the same number of storage nodes, but with 2x the NICs?<br>
<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On a similar note, does a dual-port card provide an increase in on-card processing, or 'just' another link? (The increased bandwidth is certainly nice, even in a flat switched network, I'm sure!)<br>
</blockquote>
<br></div>
Depends. If the card can talk to the PCIe bus at full speed, you might be able to saturate the link with a single QDR port. If your card is throttled for some reason (we have seen this) then adding the extra port might or might not help. If you are at the design stage, I'd suggest "go wide" as you can ... as many IB HCAs as you can get to keep the number of ports/core as high as reasonable.<br>
</blockquote><div><br> Oh dear. I need to go re-learn a lot of things. So if I want multiple full-speed QDR cards in a node, I need that node to have independent PCIe buses, and each card to be placed on a separate bus. <br>
</div><div> <br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Of course I'd have to argue the same thing on the storage side :)</blockquote><div><br> No argument from me there!<br><br> Thanks again, as always, for your input.<br> - Brian<br><br></div></div><br>