[Beowulf] [External] Spark, Julia, OpenMPI etc. - all in one place

[Oddo Da]

On Tue, Oct 13, 2020 at 8:33 AM Michael Di Domenico <mdidomenico4 at gmail.com>
wrote:

> i can't speak from a general industry sense, but i've had everything
> run through my center over the past 11 years.  Hadoop seemed like
> something that was going to take off.  it didn't with my group of
> users.  we aren't counting clicks nor parsing text from huge files, so
> its utility to us faded.  my understanding is the group behind hadoop
> also made several industry missteps when trying to commercialize, i'm
> not sure what happened after that.  i think a lot people realized that
> hadoop made things easier, but the overhead was too high given the
> limited functionality most people wanted to use it for
>

Michael, thank you for the insight. I think Hadoop in general is mostly
dying, Spark is really the derivative that took off. Basically, what you
are saying is that there is no demand on your infra for this kind of work.
Do you have any insights as to why not? Do the AI/DS/ML guys just know that
they cannot use your resources to run standard loads and go straight to the
cloud or local ethernet clusters?

In your estimate, how many of your users write code in Julia vs MPI vs
Python?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201013/1e2caaf4/attachment.html>