[Beowulf] interesting article on HPC vs evolution of 'big data' analysis

[Christopher Samuel]

On 09/04/15 03:16, H. Vidal, Jr. wrote:

> Curious as to what the body of thought is here on this article:

Some quick random thoughts on this article from down under:


1) Whilst MPI implies HPC HPC does not necessarily imply MPI (and it
never has in my decade of doing HPC, there have always been perfectly
legitimate embarrassingly parallel applications).


2) Working at a life sciences HPC centre I see roughly zero interest in
Hadoop and even less interest in Spark. Oh, and we've got a computer
scientist who has heard of Chapel.  But he also uses Haskell so you've
got to be careful. ;-)

It's just lots of data in lots of files in (if you're lucky, lots of)
directories with various apps opening them and reading through them and
writing others.  Sometimes a byte at a time.  The applications are
Python, Java and a few in C.


3) I agree with the overarching theme of the article though - HPC
centres like where I am exist for scientists to get results, and as I
ranted^W wrote on Twitter last week:

"I'm sorry, these genes you've discovered relating to breast cancer
aren't that important as your code isn't closely coupled" - says no one

https://twitter.com/chris_bloke/status/583378655180500992

Lets face it, HPL is a very efficient tightly-coupled MPI code that
benefits nobody in practical terms (OK, maybe power companies).


All the best!
Chris
-- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci