[Beowulf] ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

[Oddo Da]

On Wed, Oct 14, 2020 at 8:42 AM Michael Di Domenico <mdidomenico4 at gmail.com>
wrote:

> On Wed, Oct 14, 2020 at 8:02 AM Oddo Da <oddodaoddo at gmail.com> wrote:

i think you're implying (perhaps not consciously or i'm reading more
> into your statements) that MPI/PVM are the only frame works for
> message passing out there.  this isn't true, charm, upc, shmem, etc
> have all been developed to do basically what MPI does.  MPI (i
> believe) is just the only one that's ratified into a standard.  and
> thus it provides the code stability interconnect vendors need to write
> the shim code between the hardware and the language
>

Perhaps but I am seeing these at the same level of abstraction - the low
level where you have to spell everything out.

i think your "why" spark evolved separately is flawed.  spark/hadoop
> didn't evolve out of a company's lack of ability to pay for a cluster
> or a filesystem.  It was designed to solve a very specific problem.
> bigdata processing.  you could certainly write an mpi program to do
> the same thing, but the design of spark/hadoop does it more
> efficiently and at a lower cost.  but the point is each language does
> something specific really well, spark does data processing, mpi can do
> complex math (yes no need to circle the flaming wagons, both can do
> both, but each does one better than the other)
>

I see what you wrote above as contradicting yourself. First you say that
Spark/hadoop did not evolve from economic motivations but then you say that
it did. You say that Spark and MPI can be used interchangeably. OK if
that's the case, why does the data science/ML industry just not use MPI
everywhere. Why did they not start with MPI as the underlying paradigm and
just build all the tooling on top of it? I wish they did... ;-)


> i believe your "lack of progress" statement is really just a
> misunderstanding of what mpi really represents.  to me MPI is like the
> flathead screw, they've been around a long time and there are
> certainly a ton of alternate head designs on the market.  however,
> wooden boat builders still use them because they just work, they're
> easy to make, and they're easy to fix when it comes time to repair
>

I see MPI as a low level solution for a problem, at the abstraction level
where you need to spell everything out. It is like the comparison between C
and languages like Scala or Haskell or Julia. I am asking why there is no
progress on the latter in this scenario - we have the message passing
interface level of abstraction, why are we not interested in using this to
build tooling that is at the higher level, where we can hide the how and
focus on the what.

you're equating a 20yr old book with lack of progress and frankly i
> think that's a flawed statement.
>

20 years ago, when I was an undergrad, I took a 200-level course in data
structures and algorithms and it was taught in a programming language
called Eiffel. The professor started with saying - "Eiffel is gaining in
popularity, there are many new books being written about Eiffel but none
about C. Do you know why?". I raised my hand and said "because C is an old
language and nothing has changed and everything that was to be said about
it, was already said about it in many previous books". At least in that
domain we could discuss these things - new languages and paradigms and
tooling abound. In the world of HPC, not so much.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201014/b2e7881f/attachment.html>