[Beowulf] job scheduler and health monitoring system

[Douglas Eadline]

> On Fri, 10 Jan 2014 03:36:58 PM reza azimi wrote:
 snip

>> Our workload are mainly MPI based benchmarks and we want to test some
>> hadoop benchmarks in future.
>
> According to a presentation I saw at the Slurm User Group back in
> September
> Intel are working on Hadoop support in Slurm in such a way that you will
> not
> need a modified Hadoop stack. Not sure when that code will land though.
>

Allow me to provide a little insight into the Hadoop scene. Think of
Hadoop V1 as a monolithic MapReduce (MR) engine, think of Hadoop V2
(released  Oct '13) as a scheduler with application frameworks,
one of which is a Hadoop-1 compatible MR framework. That is, Hadoop is
moving beyond just MR. Graph processing is one example and MPI
applications can "run" under Hadoop-2 (scaling and efficiency
is another issue). There are a slew of other application frameworks
that are being developed to work within the Hadoop big data environment.

The resource manager in Hadoop-2 is called YARN (Yet Another Resource
Negotiator) and is different than the traditional HPC resource
manager. It offers many services that help to create big data
applications. The basic Hadoop Install now has three major components
(all present in the distribution tar ball): HDFS (Hadoop Distributed
File System, YARN, and the MP framework.

As Hadoop-2 begins to gain traction, I think it is important
to remember that Hadoop no longer means MR (H!=MR).

There is a lot more to this developing story, but I believe,
Hadoop-2 will allow some HPC type applications to run on
Hadoop clusters. Efforts such as SLUM integration of MR libraries
will bring MR to HPC clusters, but it all depends where the data live.

--
Doug






> Hope this is useful!
>
> Happy new year all! :-)
>
> Chris
> --
>  Christopher Samuel        Senior Systems Administrator
>  VLSCI - Victorian Life Sciences Computation Initiative
>  Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
>  http://www.vlsci.org.au/      http://twitter.com/vlsci
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> Mailscanner: Clean
>


--
Doug

-- 
Mailscanner: Clean