[Beowulf] SC13 wrapup, please post your own
landman at scalableinformatics.com
Sat Nov 23 10:40:12 PST 2013
I didn't get a chance to see many booths ... I did get free the last
hour of Thursday to wander, and made sure I got to see a few people and
What I observed (and please feel free to challenge/contradict/offer
alternative interpretations/your own views) will definitely be colored
by the glasses we wear, and the market we are in.
1) not so many chip companies (new processor designs, etc.) there. I am
not sure if this is an overall trend, but in general they do not appear
to be getting VC backing much any more.
2) Compiler companies ... virtually non-existent (I look at NAG more in
a tools and consulting view though they have a nice Fortran compiler).
Did I miss any?
Compilers were there, no doubt. Intel, PGI, ... all there. Was
PathScale there? Others?
I think this is part of a larger trend though ... not that pure compiler
companies aren't viable, but that tighter integration to get the support
the hardware vendors need has been in the offing.
3) very few pure cluster plays. Almost none. Few have actually
survived, never mind thrived.
4) many government connected folks around on the show floor. A number
of financial folks.
5) The problems people are seeking to solve are quite varied, but almost
all of them are centered around scaling up computing performance in the
face of very large, often unstructured data. This may be biased by our
6) Big data, which I liken to be applied HPC for a number of industries
was on most customers minds. How to build/compute with massive data
sets (sort of the more general case of 5). Everyone knows this word
Hadoop. Very few folks quite understand that its one of many tools to
handle data at scale.
What's profoundly interesting to me is that all of the issues that the
"big data" world face are similar issues to what the HPC and more
specifically the beowulf community have faced. How to scale
computation. How to administer and run large resources. How to design
the resources with scaling in mind.
That is, we as a community have much to offer the growing big data
community. This is evidenced in part by Doug Eadline's work on the
limulus box, and its easy transition as a private Hadoop box. Hadoop is
all about key-value stores and distributed application of mapping
functions to extract data in parallel. This is not a long leap from a
large distributed Open-MPI application with core algorithms that need to
communicate while computing. Though in most of the mapping cases, there
is very little need for interprocess communication, and the entire
system is designed to be tolerant of failure. Indeed, one can start
using MPI within Hadoop systems to provide that interprocess
communication for map reduction for non trivial (EP-like) processing.
I find this fascinating.
7) The need for very high performance storage is dramatically
increasing. We talked with many people on this. We showed a 30GB/s 4U
box (c.f. https://twitter.com/sijoe/status/403313639999815681/photo/1),
and made a mistake of not leaving the speedometers up (they were coupled
live to the machine underneath). This sparked many conversations.
8) The need for very high density storage, multiple PB/rack is
dramatically increasing. Many folks we spoke with have a capacity and
performance bottleneck, and while they don't need the most extreme
performance, putting PB behind single or dual filer heads makes for a
very non-scalable solution (their words, not mine). Most everyone now
gets that the way to scale out is to add processing and network
bandwidth as you scale capacity. Arrays and SANs are definitely rapidly
on the way out (from the conversations we've had). Parallel and
distributed file systems are on the way in.
9) Parallel and distributed file systems: Lots of folks talked Lustre,
but many more this year wanted to talk Ceph and Fraunhofer. We had just
published a Ceph benchmark in financial services a few weeks earlier, so
this was serendipitous for us.
10) The tightly-coupled or "converged" message (I liken the latter to
more marketing than real substance) where one puts massive fire power
computing and networking right next to the big data pipes ... is rapidly
emerging in this ultra dense storage and big data/high performance
computing view. People are telling us (!!!) data motion is hard, and
they want to localize computation and data as much as possible. This is
inclusive of the big data folks. This is wonderful IMO (and it reflects
my companies biases, which I freely admit and embrace).
11) Talent acquisition is hard. Finding good people is very hard. I've
had some interesting conversations on this with a few people privately.
This is part of what is driving the aqui-hire trend ... find companies
which know what they are doing, and make em an offer they can't refuse.
HPC people, with solid computing, architecture, programming skills are
in high demand. Maybe not in the broader geographical market, but
certainly in a number of specific geos.
12) SWAG quantity is down, less bauble-ish, more functional. The IU
hats rocked ... it was snowing, they were warm. LSU had scarves (which
sadly I did not grab one). As did NVidia. We gave out flashlights for
listening to our partners talks, pens, coffee, biscotti, etc. Usually
you hear about some very oddball SWAG, that you need to seek out, but I
didn't hear that this year.
13) Size: show floor was smaller. The impact of fewer US Government
people and limited booth space was obvious. Number of attendees on show
floor appeared to be lower. This said, we had more traffic at our booth
14) Quality of talks/BOFs: I've heard from many sources that the talks
and BOFs were great. I miss having time to attend them, but will push
for this next year. The admin BOFs seem to be strongly in demand.
15) Beobash rocked. I did not get to play pool. I did spend time
talking with many folks, and drinking beer. Lara and team did a bang
up job putting this together. As usual, it was the best party at the
On a personal note, I am grateful to everyone who took the time to talk
with me, or even wave hi (even if they could not stop to talk). They
kept me chained in the booth (no not really, but I couldn't wander far).
And they had strict instructions to make sure I circulated and did not
get locked into the deeper conversations that I like.
I thought it was a good show, nothing revolutionary, quite a bit of
evolutionary things. Fewer new faces, many older (yeah, I know, speak
This was a significant show for us as a company, as we had our first
"larger" booth. Pics here:
I'll put up more photos soon.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
twtr : @scalableinfo
phone: +1 734 786 8423 x121
cell : +1 734 612 4615
More information about the Beowulf