<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Got it,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks for the clarification from all
</span><span style="font-family:"Segoe UI Emoji",sans-serif;mso-fareast-language:EN-US">😊</span><span style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Have a great evening.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Jonathan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Alex Chekholko <alex@calicolabs.com>
<br>
<b>Sent:</b> 24 November 2020 19:31<br>
<b>To:</b> Jonathan Aquilina <jaquilina@eagleeyet.net><br>
<b>Cc:</b> Douglas Eadline <deadline@eadline.org>; beowulf@beowulf.org<br>
<b>Subject:</b> Re: [Beowulf] Clustering vs Hadoop/spark<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Jonathan,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You need a "cluster" if your task is too large to run on a single system, or maybe if it takes too long to run on a single system.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So the primary motivation is practical, not theoretical.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If you can run your task on just one computer, you should always do that rather than having to build a cluster of some kind and all the associated headaches.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One aspect of your question seems to be about performance, which is ultimately limited by the hardware resources. E.g. a "hadoop cluster" might be 10 servers each with lots of CPUs and RAM and disks, etc. For "hadoop" workloads, typically
the bottleneck is I/O so the primary parameter is the number of disks. So the programming language is not really an issue if you're spending all your time waiting on disk I/O.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Alex<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Nov 24, 2020 at 10:22 AM Jonathan Aquilina via Beowulf <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi Doug,<br>
<br>
So what is the advantage then of a cluster?<br>
<br>
Regards,<br>
Jonathan<br>
<br>
-----Original Message-----<br>
From: Douglas Eadline <<a href="mailto:deadline@eadline.org" target="_blank">deadline@eadline.org</a>>
<br>
Sent: 24 November 2020 18:21<br>
To: Jonathan Aquilina <<a href="mailto:jaquilina@eagleeyet.net" target="_blank">jaquilina@eagleeyet.net</a>><br>
Cc: <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
Subject: RE: [Beowulf] Clustering vs Hadoop/spark<br>
<br>
<br>
First I am not a Java expert (very far from it).<br>
<br>
Second, Java holds up quite well against Julia as compared to Python. (so does Lisp!)<br>
<br>
<a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia.html" target="_blank">
https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia.html</a><br>
<br>
Some other tings to consider is the underlying Hadoop plumbing is written in Java (or Scala for Spark).<br>
However, it is possible with Hadoop to<br>
create mapper and reducing functions in any language (text based std in/out) Similar to Spark, that can use Python, R, Java, or Scala as a front-end.<br>
<br>
So there is a bit of decoupling of parallel compute mechanism and compute code. (I.e. in all these languages the user does think about cores, interconnect, communications, etc) A higher level abstraction.<br>
<br>
Much of the early Hadoop performance was based on running large capability jobs on gobs of data.<br>
Jobs that could not be run otherwise (except for Google) So any performance was good. Spark come along and says, lets put it all in a redundant distributed memory structure.<br>
Speed up is much faster then traditional Hadoop, so Hadoop creates Tez API that does the same thing.<br>
Performance even out.<br>
<br>
Plus analytics jobs are mostly integer. The floating point often comes into play when running the models, which is often not a big data problem (i.e. don't need a big cluster to run)<br>
<br>
--<br>
Doug<br>
<br>
> Hi Doug,<br>
><br>
> Appreciate the clarification where I am not clear is given Hadoop and <br>
> derivatives are java based where all of this performance all of a <br>
> sudden comes from. Is it due to where the data resides?<br>
><br>
> At one of my previous jobs I worked with Hadoop through Amazon AWS EMR <br>
> managed to churn through 5 years' worth of historical data in 1 week. <br>
> Data being calculations on vehicular tracking data.<br>
><br>
> When I learned java as part of my degree I used to see it as clunky <br>
> why go for an interpreted language such as java over something more <br>
> low level like c/c++ on a traditional cluster?<br>
><br>
> Regards,<br>
> Jonathan<br>
><br>
> -----Original Message-----<br>
> From: Douglas Eadline <<a href="mailto:deadline@eadline.org" target="_blank">deadline@eadline.org</a>><br>
> Sent: 24 November 2020 17:38<br>
> To: Jonathan Aquilina <<a href="mailto:jaquilina@eagleeyet.net" target="_blank">jaquilina@eagleeyet.net</a>><br>
> Cc: <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
> Subject: Re: [Beowulf] Clustering vs Hadoop/spark<br>
><br>
><br>
>> Hi Guys,<br>
>><br>
>> I am just wondering what advantages does setting up of a cluster have <br>
>> in relation to big data analytics vs using something like Hadoop/spark?<br>
>><br>
><br>
> Long email and the details are important.<br>
><br>
> It all comes down to filesystems and schedulers. But first remember, <br>
> most Data Analytics projects use many different tools and have various <br>
> stages that often require iteration and development (e.g. ETL->Feature<br>
> Matrix->and running models, repeat, and 80% of the work in in first <br>
> Matrix->two<br>
> steps) And, many end-users do not use Java map-reduce APIs. They use <br>
> higher level tools.<br>
><br>
> Filesystems:<br>
><br>
> 1) Traditional Hadoop filesystem (HDFS) is about slicing large data <br>
> files (or large number of files) across multiple servers, then doing <br>
> the map phase on all servers at the same time (moving computation to <br>
> where the data "live", reduce phase requires a shuffle (data movement) <br>
> and final reduction of data.<br>
><br>
> 2) On-prem HDFS still makes some sense (longer story) however, in the <br>
> Cloud there is move to using native cloud storage using Apache Ozone FS.<br>
> You loose the "data locality," but gain all the cloud Kubernettes stuff.<br>
><br>
> 3) Both Hadoop Map-Reduce (mostly Hive RDB applications now) and Spark <br>
> do "in-memory" map-reduce for performance reasons.<br>
> In this case, data locality for processing is not as important, <br>
> However, loading and storing files on large multi-server memory <br>
> resident jobs still gains from HDFS. Very often Spark writes/reads results into Hive tables.<br>
><br>
> Schedulers:<br>
><br>
> 1) Map Reduce scheduling is different than traditional HPC scheduling.<br>
> The primary Hadoop scheduler is called YARN (Yet Another Resource<br>
> Negotiator) It has two main features not found in most HPC schedulers, <br>
> data locality as a resource and dynamic resource allocation.<br>
><br>
> 2) Data locality is about moving jobs to where the data (slice) lives <br>
> on the storage nodes (hyper-converged storage/compute nodes)<br>
><br>
> 3) Dynamic resource allocation developed because most map-reduce jobs <br>
> need a lot of containers for map phase, but much-much less for reduce <br>
> phase, so Hadoop map-reduce can give back resources and ask for more <br>
> later in other stages of the DAG (multiple map reduce phases are run <br>
> as a Directed Acyclic Graph)<br>
><br>
> Thus, this model is hard to map on to a traditional HPC cluster.<br>
> There are map-reduce libraries for MPI. Another way to think about it <br>
> is Data Analytics is almost always SIMD, all tools language and <br>
> platforms are optimized to take advantage of map-reduce SIMD operations and data flow.<br>
><br>
><br>
> --<br>
> Doug<br>
><br>
><br>
><br>
>><br>
>> Regards,<br>
>> Jonathan<br>
>> _______________________________________________<br>
>> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin
<br>
>> Computing To change your subscription (digest mode or unsubscribe) <br>
>> visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" target="_blank">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
>><br>
><br>
><br>
> --<br>
> Doug<br>
><br>
<br>
<br>
--<br>
Doug<br>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" target="_blank">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>