[Beowulf] interesting article on HPC vs evolution of 'big data' analysis
e.scott.atchley at gmail.com
Thu Apr 9 06:08:56 PDT 2015
On Wed, Apr 8, 2015 at 9:56 PM, Greg Lindahl <lindahl at pbm.com> wrote:
> On Wed, Apr 08, 2015 at 03:57:34PM -0400, Scott Atchley wrote:
> > There is concern by some and outright declaration by others (including
> > hardware vendors) that MPI will not scale to exascale due to issues like
> > rank state growing too large for 10-100 million endpoints,
> That's weird, given that it's an implementation choice.
It is one of the concerns raised, but not the only one. No one is giving up
on MPI; that is not an alternative given the existing code base. There are
efforts to avoid duplication of rank information within a node (no need to
each rank to have duplicates) or use a single MPI rank per node and use
OpenMP/other to manage node-local parallelism at the risk of a large
many-core node's cores all trying to access the NIC at the same time.
I am not advocating for/against MPI or predicting its imminent demise, but
I am aware of the concerns by the vendors.
> Presumably Intel is keeping the PathScale tiny rank state as a
One would expect, but that is probably necessary but not sufficient for
their many-core future.
> Reliability, now that's a serious issue! And not one that's trivially
> fixed for any problem that must be tightly-coupled.
Yes, and we are open to suggestions. ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf