[Beowulf] ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

[Oddo Da]

P.S. I should say that I don't have a horse in this race, I am not selling
anything, I am not pushing anything. I am just asking :-)

On Wed, Oct 14, 2020 at 9:02 AM Oddo Da <oddodaoddo at gmail.com> wrote:

> On Wed, Oct 14, 2020 at 8:42 AM Michael Di Domenico <
> mdidomenico4 at gmail.com> wrote:
>
>> On Wed, Oct 14, 2020 at 8:02 AM Oddo Da <oddodaoddo at gmail.com> wrote:
>
> i think you're implying (perhaps not consciously or i'm reading more
>> into your statements) that MPI/PVM are the only frame works for
>> message passing out there.  this isn't true, charm, upc, shmem, etc
>> have all been developed to do basically what MPI does.  MPI (i
>> believe) is just the only one that's ratified into a standard.  and
>> thus it provides the code stability interconnect vendors need to write
>> the shim code between the hardware and the language
>>
>
> Perhaps but I am seeing these at the same level of abstraction - the low
> level where you have to spell everything out.
>
> i think your "why" spark evolved separately is flawed.  spark/hadoop
>> didn't evolve out of a company's lack of ability to pay for a cluster
>> or a filesystem.  It was designed to solve a very specific problem.
>> bigdata processing.  you could certainly write an mpi program to do
>> the same thing, but the design of spark/hadoop does it more
>> efficiently and at a lower cost.  but the point is each language does
>> something specific really well, spark does data processing, mpi can do
>> complex math (yes no need to circle the flaming wagons, both can do
>> both, but each does one better than the other)
>>
>
> I see what you wrote above as contradicting yourself. First you say that
> Spark/hadoop did not evolve from economic motivations but then you say that
> it did. You say that Spark and MPI can be used interchangeably. OK if
> that's the case, why does the data science/ML industry just not use MPI
> everywhere. Why did they not start with MPI as the underlying paradigm and
> just build all the tooling on top of it? I wish they did... ;-)
>
>
>> i believe your "lack of progress" statement is really just a
>> misunderstanding of what mpi really represents.  to me MPI is like the
>> flathead screw, they've been around a long time and there are
>> certainly a ton of alternate head designs on the market.  however,
>> wooden boat builders still use them because they just work, they're
>> easy to make, and they're easy to fix when it comes time to repair
>>
>
> I see MPI as a low level solution for a problem, at the abstraction level
> where you need to spell everything out. It is like the comparison between C
> and languages like Scala or Haskell or Julia. I am asking why there is no
> progress on the latter in this scenario - we have the message passing
> interface level of abstraction, why are we not interested in using this to
> build tooling that is at the higher level, where we can hide the how and
> focus on the what.
>
> you're equating a 20yr old book with lack of progress and frankly i
>> think that's a flawed statement.
>>
>
> 20 years ago, when I was an undergrad, I took a 200-level course in data
> structures and algorithms and it was taught in a programming language
> called Eiffel. The professor started with saying - "Eiffel is gaining in
> popularity, there are many new books being written about Eiffel but none
> about C. Do you know why?". I raised my hand and said "because C is an old
> language and nothing has changed and everything that was to be said about
> it, was already said about it in many previous books". At least in that
> domain we could discuss these things - new languages and paradigms and
> tooling abound. In the world of HPC, not so much.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201014/994ed1d2/attachment-0001.html>