[Beowulf] interesting article on HPC vs evolution of 'big data' analysis

[Scott Atchley]

On Wed, Apr 8, 2015 at 9:56 PM, Greg Lindahl <lindahl at pbm.com> wrote:

> On Wed, Apr 08, 2015 at 03:57:34PM -0400, Scott Atchley wrote:
>
> > There is concern by some and outright declaration by others (including
> > hardware vendors) that MPI will not scale to exascale due to issues like
> > rank state growing too large for 10-100 million endpoints,
>
> That's weird, given that it's an implementation choice.
>

It is one of the concerns raised, but not the only one. No one is giving up
on MPI; that is not an alternative given the existing code base. There are
efforts to avoid duplication of rank information within a node (no need to
each rank to have duplicates) or use a single MPI rank per node and use
OpenMP/other to manage node-local parallelism at the risk of a large
many-core node's cores all trying to access the NIC at the same time.

I am not advocating for/against MPI or predicting its imminent demise, but
I am aware of the concerns by the vendors.



> Presumably Intel is keeping the PathScale tiny rank state as a
> feature?
>

One would expect, but that is probably necessary but not sufficient for
their many-core future.



> Reliability, now that's a serious issue! And not one that's trivially
> fixed for any problem that must be tightly-coupled.
>

Yes, and we are open to suggestions. ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20150409/77809e48/attachment.html>