[Beowulf] SC13 wrapup, please post your own

[Ellis H. Wilson III]

On 11/23/2013 01:40 PM, Joe Landman wrote:
> I didn't get a chance to see many booths ... I did get free the last
> hour of Thursday to wander, and made sure I got to see a few people and
> companies.

Had a bit of the same experience, but in the reverse.  Was cooped up in 
tech talks the entire time and/or preparing for my own, so only got up 
to the exhibit floor for a few hours on Wednesday.

> 1) not so many chip companies (new processor designs, etc.) there.  I am
> not sure if this is an overall trend, but in general they do not appear
> to be getting VC backing much any more.

Micron did have a really cool new processor called Automata in the 
Emerging Technologies area of the floor.  Quick Google gives me the 
following article about it, which seems pretty representative:

http://www.techworld.com.au/article/532432/micron_challenges_conventional_computer_architecture_new_chip/

> 6) Big data, which I liken to be applied HPC for a number of industries
> was on most customers minds.  How to build/compute with massive data
> sets (sort of the more general case of 5).  Everyone knows this word
> Hadoop.  Very few folks quite understand that its one of many tools to
> handle data at scale.
>
> What's profoundly interesting to me is that all of the issues that the
> "big data" world face are similar issues to what the HPC and more
> specifically the beowulf community have faced.  How to scale
> computation.  How to administer and run large resources.  How to design
> the resources with scaling in mind.

I met a lot of folks in the technical program that weren't yet concerned 
about data problems simply because they were still so compute-limited. 
I see certain niches in HPC as staying that way for years to come.  With 
the exception of checkpointing, which they (sadly) try not to think 
about and defer to the system architects and admins to figure out, their 
problems more or less stay in memory the entire time.

This strikes at the core of a massive problem I witnessed at SC this 
time around: the technical talks and the exhibition seem to have almost 
isolated communities.  There are obvious exceptions -- I consider myself 
one of such, and do my best to keep abreast of where industry is going 
so I can factor those trends into where the most relevant research can 
be done.  This even exemplifies itself in the difference between the 
research being shown in the talks versus the industry presence on the 
exhibition floor:  tons of storage and networking providers on the 
exhibition floor, but only a handful of storage and networking papers, 
mostly isolated to Tuesday (but a few on Wednesday, including my own). 
There were a number of sessions I attended claiming to be data-oriented 
but turned out to be pretty traditional compute-oriented works with a 
data sub-focus (seemingly) just to leverage the necessary buzz-words.

Sad really.  I was under the impression we in academia are supposed to 
be out in front of industry, testing the waters and leading the way 
where industry might not yet have a play to deem such worthwhile, rather 
than behind, optimizing the hell out of old problems as if they were a 
homework problem.  Oh well.  However, to be fair, I think more industry 
folks should try to show up to, and talk with, the academics in the tech 
talks.  This will help drive them towards modern-day problems and 
enlighten them relative to what is on the minds of the "real world" 
today.  This is particularly true for the smaller companies, who are 
more agile and willing to try the wacky things us academics explore (or 
should be exploring).  I recognizing sparing the personnel from a 
smaller company is more difficult though.  I had a few very interesting 
conversations with persons from larger companies after my talk, but 
nowhere near what I thought I would.  Maybe that just reflects on the 
quality of my talk or a lack of alignment with industry for one reason 
or another :/.

Just my (admittedly biased) perspective though, and not the case for all 
the talks, just a greater number than I was comfortable with.

That's all I have time to share for now -- more on the Hadoop bits 
later.  I also got a chance to talk with Doug at the bash, and although 
I missed his presentations (VERY sad about that), I deeply want to learn 
and discuss more about what he's doing over there.  Very cool stuff in 
general.  Seems his Hadoop-foo has leap-frogged mine over the last year :D.

Best,

ellis


-- 
Ph.D. Candidate
Department of Computer Science and Engineering
The Pennsylvania State University
www.ellisv3.com