[Beowulf] Q: IB message rate & large core counts (per node)?

[Brian Dobbins]

Hi Patrick,

I have been quite vocal in the past against the merit of high packet rate,
> but I have learned to appreciate it. There is a set of applications that can
> benefit from it, especially at scale. Actually, packet rate is much more
> important outside of HPC (where application throughput is what money buys).
>

  The 'especially at scale' bit seems to me to be the critical issue -
weighing the price/performance as the ratio of small-scale to large-scale
runs changes, assuming that an adapter with better large-scale performance
has a significant cost differential.   If only we knew what that ratio would
be ahead of time, this would be easier.  :-)

However, I would pay attention to a different problem with many-core
> machines. Each user-space process uses a dedicated set of NIC resources, and
> this can be a problem with 48 cores per node (it affects all vendors, even
> if they swear otherwise). You may want to consider multiple NICs, unless you
> know that only a subset of the cores are communicating through the network
> (hybrid MPI/Open-MP model for example) or that the multiplexing overhead is
> not a big deal for you.


  Well, clearly we hope to move more towards hybrid methods -all that's old
is new again?- but, again, it's *currently* hard to quantify the variables
involved.  Time to transition, performance differences, user effort, etc.
But getting back to a technical vein, is the multiplexing an issue due to
atomic locks on mapped memory pages?  Or just because each copy reserves its
own independent buffers?  What are the critical issues?


> You need PCIe Gen2 x16 to saturate a 32 Gb/s QDR link. There is no such NIC
> on the market AFAIK (only Gen1 x16 or Gen2 x8). But even then, you won't
> have any PCIe bandwidth left to drive a second port on the same NIC. There
> may be other rationales for a second port, but bandwidth is not one of them.
>

  I thought PCIe Gen2 x 8 @ 500 Mhz gives 8GB/s?  I know there are 250 and
500 Mhz variants in addition to the lane sizes, so while a 250 Mhz x8 link
wouldn't provide enough bandwidth to a dual-port card, the 500 Mhz one
should.  But I'm woefully out of date on my hardware knowledge, it seems.
Of course, EDR (eight data-rate) IB is on the roadmap for 2011, so if we're
in no rush that could help, too.

  Cheers,
  - Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.scyld.com/pipermail/beowulf/attachments/20100223/6911c38f/attachment.html