<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Question becomes then with all the File systems out there how does one choose the best performing as each file system has its own unique advantages etc.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Jonathan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Jim Cownie <jcownie@gmail.com>
<br>
<b>Sent:</b> 25 November 2020 10:05<br>
<b>To:</b> Jonathan Aquilina <jaquilina@eagleeyet.net><br>
<b>Cc:</b> Douglas Eadline <deadline@eadline.org>; beowulf@beowulf.org<br>
<b>Subject:</b> Re: [Beowulf] Clustering vs Hadoop/spark<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">From: Douglas Eadline <<a href="mailto:deadline@eadline.org">deadline@eadline.org</a>> <br>
Sent: 24 November 2020 18:21<o:p></o:p></p>
</blockquote>
<p class="MsoNormal">…<o:p></o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">So there is a bit of decoupling of parallel compute mechanism and compute code. (I.e. in all these languages the user does think about cores, interconnect, communications, etc) A higher level abstraction.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">I think you’re missing a “not”, Doug. "in all these languages the user does
<b><span style="color:black;background:#FFFB00">not</span></b> think about cores, interconnect, communications, etc” <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">More generally “It's always the data movement.” We see that here, but it also applies to our CPUs. Look at the performance of Fugaku and the A64FX which has optimised for that rather than lots of (generally unusable) FLOPS, or consider
why we use roofline models when tuning compute kernels, and, when we do, how often they show that we’re bandwidth bound.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If we assume that most of the time is I/O here, then it's unsurprising that changing the language used for the processing makes little difference. You can think of it as a variant of Amdahl, where you map I/O time to Amdahl’s serial time
(i.e. it’s invariant) and compute to parallel time (I.e. you can change it). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">For example, if the compute is 10% of the time, the best you can hope for even if you could use my new “psychic” language and run the compute in zero time is to get a 1/0.9 = 11% speedup.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">In that context it should be no surprise that the precise performance of the compute doesn't matter much, so you get comparable overall performance independent of the choice of language used for the compute operations.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- Jim<br>
James Cownie <<a href="mailto:jcownie@gmail.com">jcownie@gmail.com</a>><br>
Mob: +44 780 637 7146<br>
<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">-----Original Message-----<br>
From: Douglas Eadline <</span><a href="mailto:deadline@eadline.org"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">deadline@eadline.org</span></a><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">><span class="apple-converted-space"> </span><br>
Sent: 24 November 2020 18:21<br>
To: Jonathan Aquilina <</span><a href="mailto:jaquilina@eagleeyet.net"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">jaquilina@eagleeyet.net</span></a><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">><br>
Cc:<span class="apple-converted-space"> </span></span><a href="mailto:beowulf@beowulf.org"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">beowulf@beowulf.org</span></a><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif"><br>
Subject: RE: [Beowulf] Clustering vs Hadoop/spark<br>
<br>
<br>
First I am not a Java expert (very far from it).<br>
<br>
Second, Java holds up quite well against Julia as compared to Python. (so does Lisp!)<br>
<br>
</span><a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia.html"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia.html</span></a><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
Some other tings to consider is the underlying Hadoop plumbing is written in Java (or Scala for Spark).<br>
However, it is possible with Hadoop to<br>
create mapper and reducing functions in any language (text based std in/out) Similar to Spark, that can use Python, R, Java, or Scala as a front-end.<br>
<br>
So there is a bit of decoupling of parallel compute mechanism and compute code. (I.e. in all these languages the user does think about cores, interconnect, communications, etc) A higher level abstraction.<br>
<br>
Much of the early Hadoop performance was based on running large capability jobs on gobs of data.<br>
Jobs that could not be run otherwise (except for Google) So any performance was good. Spark come along and says, lets put it all in a redundant distributed memory structure.<br>
Speed up is much faster then traditional Hadoop, so Hadoop creates Tez API that does the same thing.<br>
Performance even out.<br>
<br>
Plus analytics jobs are mostly integer. The floating point often comes into play when running the models, which is often not a big data problem (i.e. don't need a big cluster to run)<br>
<br>
--<br>
Doug<br>
<br style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">Hi Doug,<br>
<br>
Appreciate the clarification where I am not clear is given Hadoop and<span class="apple-converted-space"> </span><br>
derivatives are java based where all of this performance all of a<span class="apple-converted-space"> </span><br>
sudden comes from. Is it due to where the data resides?<br>
<br>
At one of my previous jobs I worked with Hadoop through Amazon AWS EMR<span class="apple-converted-space"> </span><br>
managed to churn through 5 years' worth of historical data in 1 week.<span class="apple-converted-space"> </span><br>
Data being calculations on vehicular tracking data.<br>
<br>
When I learned java as part of my degree I used to see it as clunky<span class="apple-converted-space"> </span><br>
why go for an interpreted language such as java over something more<span class="apple-converted-space"> </span><br>
low level like c/c++ on a traditional cluster?<br>
<br>
Regards,<br>
Jonathan<br>
<br>
-----Original Message-----<br>
From: Douglas Eadline <<a href="mailto:deadline@eadline.org">deadline@eadline.org</a>><br>
Sent: 24 November 2020 17:38<br>
To: Jonathan Aquilina <<a href="mailto:jaquilina@eagleeyet.net">jaquilina@eagleeyet.net</a>><br>
Cc: <a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a><br>
Subject: Re: [Beowulf] Clustering vs Hadoop/spark<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">Hi Guys,<br>
<br>
I am just wondering what advantages does setting up of a cluster have<span class="apple-converted-space"> </span><br>
in relation to big data analytics vs using something like Hadoop/spark?<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif"><br>
Long email and the details are important.<br>
<br>
It all comes down to filesystems and schedulers. But first remember,<span class="apple-converted-space"> </span><br>
most Data Analytics projects use many different tools and have various<span class="apple-converted-space"> </span><br>
stages that often require iteration and development (e.g. ETL->Feature<br>
Matrix->and running models, repeat, and 80% of the work in in first<span class="apple-converted-space"> </span><br>
Matrix->two<br>
steps) And, many end-users do not use Java map-reduce APIs. They use<span class="apple-converted-space"> </span><br>
higher level tools.<br>
<br>
Filesystems:<br>
<br>
1) Traditional Hadoop filesystem (HDFS) is about slicing large data<span class="apple-converted-space"> </span><br>
files (or large number of files) across multiple servers, then doing<span class="apple-converted-space"> </span><br>
the map phase on all servers at the same time (moving computation to<span class="apple-converted-space"> </span><br>
where the data "live", reduce phase requires a shuffle (data movement)<span class="apple-converted-space"> </span><br>
and final reduction of data.<br>
<br>
2) On-prem HDFS still makes some sense (longer story) however, in the<span class="apple-converted-space"> </span><br>
Cloud there is move to using native cloud storage using Apache Ozone FS.<br>
You loose the "data locality," but gain all the cloud Kubernettes stuff.<br>
<br>
3) Both Hadoop Map-Reduce (mostly Hive RDB applications now) and Spark<span class="apple-converted-space"> </span><br>
do "in-memory" map-reduce for performance reasons.<br>
In this case, data locality for processing is not as important,<span class="apple-converted-space"> </span><br>
However, loading and storing files on large multi-server memory<span class="apple-converted-space"> </span><br>
resident jobs still gains from HDFS. Very often Spark writes/reads results into Hive tables.<br>
<br>
Schedulers:<br>
<br>
1) Map Reduce scheduling is different than traditional HPC scheduling.<br>
The primary Hadoop scheduler is called YARN (Yet Another Resource<br>
Negotiator) It has two main features not found in most HPC schedulers,<span class="apple-converted-space"> </span><br>
data locality as a resource and dynamic resource allocation.<br>
<br>
2) Data locality is about moving jobs to where the data (slice) lives<span class="apple-converted-space"> </span><br>
on the storage nodes (hyper-converged storage/compute nodes)<br>
<br>
3) Dynamic resource allocation developed because most map-reduce jobs<span class="apple-converted-space"> </span><br>
need a lot of containers for map phase, but much-much less for reduce<span class="apple-converted-space"> </span><br>
phase, so Hadoop map-reduce can give back resources and ask for more<span class="apple-converted-space"> </span><br>
later in other stages of the DAG (multiple map reduce phases are run<span class="apple-converted-space"> </span><br>
as a Directed Acyclic Graph)<br>
<br>
Thus, this model is hard to map on to a traditional HPC cluster.<br>
There are map-reduce libraries for MPI. Another way to think about it<span class="apple-converted-space"> </span><br>
is Data Analytics is almost always SIMD, all tools language and<span class="apple-converted-space"> </span><br>
platforms are optimized to take advantage of map-reduce SIMD operations and data flow.<br>
<br>
<br>
--<br>
Doug<br>
<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif"><br>
Regards,<br>
Jonathan<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin<span class="apple-converted-space"> </span><br>
Computing To change your subscription (digest mode or unsubscribe)<span class="apple-converted-space"> </span><br>
visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
--<br>
Doug<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
--<br>
Doug<br>
<br>
_______________________________________________<br>
Beowulf mailing list,<span class="apple-converted-space"> </span></span><a href="mailto:Beowulf@beowulf.org"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">Beowulf@beowulf.org</span></a><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif"> </span></span><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">sponsored
by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit<span class="apple-converted-space"> </span></span><a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf"><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>