[Beowulf] Hadoop's Uncomfortable Fit in HPC

[Andrew Holway]

Important paragraph:

"Some larger players in the HPC arena have begun to provide rich support
for high-performance parallel file systems as a complete alternative to
HDFS.  IBM's GPFS file system has a file placement optimization (FPO)
capability that allows GPFS to act as a drop-in replacement for HDFS, and
Intel was selling native Lustre support before they sold IDH to Cloudera."

And Intel still are but are bundling their Lustre <-> Hadoop connector with
their commercially supported Lustre product "IEEL". It's not OSS however.
They support Cloudera (obviously as they just sunk $740m) and Apache
Hadoop. There is also support for SLURM as a scheduler however I don't have
a nice link. These bits should mean that Hadoop can play nicely as a
"normal" HPC application.





On 17 May 2014 22:46, Kilian Cavalotti <kilian.cavalotti.work at gmail.com>wrote:

> Great write-up by Glenn  Lockwood about the state of Hadoop in HPC. It
> pretty much nails it, and offers an nice overview of the current
> ongoing efforts to make it relevant in that field.
>
>
> http://glennklockwood.blogspot.com/2014/05/hadoops-uncomfortable-fit-in-hpc.html
>
> Most spot on thing I've read in a while. Thanks Glenn.
> Cheers,
> --
> Kilian
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20140519/5f833f09/attachment.html>