[Beowulf] SC13 wrapup, please post your own

[Joe Landman]

I didn't get a chance to see many booths ... I did get free the last 
hour of Thursday to wander, and made sure I got to see a few people and 
companies.

What I observed (and please feel free to challenge/contradict/offer 
alternative interpretations/your own views) will definitely be colored 
by the glasses we wear, and the market we are in.

1) not so many chip companies (new processor designs, etc.) there.  I am 
not sure if this is an overall trend, but in general they do not appear 
to be getting VC backing much any more.

2) Compiler companies ... virtually non-existent (I look at NAG more in 
a tools and consulting view though they have a nice Fortran compiler). 
Did I miss any?

Compilers were there, no doubt.  Intel, PGI, ... all there.  Was 
PathScale there?  Others?

I think this is part of a larger trend though ... not that pure compiler 
companies aren't viable, but that tighter integration to get the support 
the hardware vendors need has been in the offing.

3) very few pure cluster plays.  Almost none.  Few have actually 
survived, never mind thrived.

4) many government connected folks around on the show floor.  A number 
of financial folks.

5) The problems people are seeking to solve are quite varied, but almost 
all of them are centered around scaling up computing performance in the 
face of very large, often unstructured data.  This may be biased by our 
own focus.

6) Big data, which I liken to be applied HPC for a number of industries 
was on most customers minds.  How to build/compute with massive data 
sets (sort of the more general case of 5).  Everyone knows this word 
Hadoop.  Very few folks quite understand that its one of many tools to 
handle data at scale.

What's profoundly interesting to me is that all of the issues that the 
"big data" world face are similar issues to what the HPC and more 
specifically the beowulf community have faced.  How to scale 
computation.  How to administer and run large resources.  How to design 
the resources with scaling in mind.

That is, we as a community have much to offer the growing big data 
community.  This is evidenced in part by Doug Eadline's work on the 
limulus box, and its easy transition as a private Hadoop box.  Hadoop is 
all about key-value stores and distributed application of mapping 
functions to extract data in parallel.  This is not a long leap from a 
large distributed Open-MPI application with core algorithms that need to 
communicate while computing.  Though in most of the mapping cases, there 
is very little need for interprocess communication, and the entire 
system is designed to be tolerant of failure.  Indeed, one can start 
using MPI within Hadoop systems to provide that interprocess 
communication for map reduction for non trivial (EP-like) processing.

I find this fascinating.

7) The need for very high performance storage is dramatically 
increasing.  We talked with many people on this.  We showed a 30GB/s 4U 
box (c.f. https://twitter.com/sijoe/status/403313639999815681/photo/1), 
and made a mistake of not leaving the speedometers up (they were coupled 
live to the machine underneath).  This sparked many conversations.

8) The need for very high density storage, multiple PB/rack is 
dramatically increasing.  Many folks we spoke with have a capacity and 
performance bottleneck, and while they don't need the most extreme 
performance, putting PB behind single or dual filer heads makes for a 
very non-scalable solution (their words, not mine).  Most everyone now 
gets that the way to scale out is to add processing and network 
bandwidth as you scale capacity.  Arrays and SANs are definitely rapidly 
on the way out (from the conversations we've had).  Parallel and 
distributed file systems are on the way in.

9) Parallel and distributed file systems:  Lots of folks talked Lustre, 
but many more this year wanted to talk Ceph and Fraunhofer.  We had just 
published a Ceph benchmark in financial services a few weeks earlier, so 
this was serendipitous for us.

10) The tightly-coupled or "converged" message (I liken the latter to 
more marketing than real substance) where one puts massive fire power 
computing and networking right next to the big data pipes ... is rapidly 
emerging in this ultra dense storage and big data/high performance 
computing view.  People are telling us (!!!) data motion is hard, and 
they want to localize computation and data as much as possible.  This is 
inclusive of the big data folks.  This is wonderful IMO (and it reflects 
my companies biases, which I freely admit and embrace).

11) Talent acquisition is hard.  Finding good people is very hard.  I've 
had some interesting conversations on this with a few people privately. 
  This is part of what is driving the aqui-hire trend ... find companies 
which know what they are doing, and make em an offer they can't refuse.

HPC people, with solid computing, architecture, programming skills are 
in high demand.  Maybe not in the broader geographical market, but 
certainly in a number of specific geos.

12) SWAG quantity is down, less bauble-ish, more functional.  The IU 
hats rocked ... it was snowing, they were warm.  LSU had scarves (which 
sadly I did not grab one).  As did NVidia.  We gave out flashlights for 
listening to our partners talks, pens, coffee, biscotti, etc.  Usually 
you hear about some very oddball SWAG, that you need to seek out, but I 
didn't hear that this year.

13) Size:  show floor was smaller.  The impact of fewer US Government 
people and limited booth space was obvious.  Number of attendees on show 
floor appeared to be lower.  This said, we had more traffic at our booth 
than ever.

14) Quality of talks/BOFs: I've heard from many sources that the talks 
and BOFs were great.  I miss having time to attend them, but will push 
for this next year.  The admin BOFs seem to be strongly in demand.

15) Beobash rocked.  I did not get to play pool.  I did spend time 
talking with many folks, and drinking beer.   Lara and team did a bang 
up job putting this together.  As usual, it was the best party at the 
show.

On a personal note, I am grateful to everyone who took the time to talk 
with me, or even wave hi (even if they could not stop to talk).  They 
kept me chained in the booth (no not really, but I couldn't wander far). 
  And they had strict instructions to make sure I circulated and did not 
get locked into the deeper conversations that I like.

I thought it was a good show, nothing revolutionary, quite a bit of 
evolutionary things.  Fewer new faces, many older (yeah, I know, speak 
for myself).

This was a significant show for us as a company, as we had our first 
"larger" booth.  Pics here:

https://twitter.com/sijoe/status/402845383547564032/photo/1/large
https://twitter.com/sijoe/status/403315326000959488/photo/1/large
https://scontent-b-ord.xx.fbcdn.net/hphotos-ash3/q71/s720x720/994043_10151702139320064_2096289564_n.jpg


I'll put up more photos soon.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
twtr : @scalableinfo
phone: +1 734 786 8423 x121
cell : +1 734 612 4615