[Beowulf] Hadoop's Uncomfortable Fit in HPC
andrew.holway at gmail.com
Mon May 19 14:36:35 PDT 2014
"Some larger players in the HPC arena have begun to provide rich support
for high-performance parallel file systems as a complete alternative to
HDFS. IBM's GPFS file system has a file placement optimization (FPO)
capability that allows GPFS to act as a drop-in replacement for HDFS, and
Intel was selling native Lustre support before they sold IDH to Cloudera."
And Intel still are but are bundling their Lustre <-> Hadoop connector with
their commercially supported Lustre product "IEEL". It's not OSS however.
They support Cloudera (obviously as they just sunk $740m) and Apache
Hadoop. There is also support for SLURM as a scheduler however I don't have
a nice link. These bits should mean that Hadoop can play nicely as a
"normal" HPC application.
On 17 May 2014 22:46, Kilian Cavalotti <kilian.cavalotti.work at gmail.com>wrote:
> Great write-up by Glenn Lockwood about the state of Hadoop in HPC. It
> pretty much nails it, and offers an nice overview of the current
> ongoing efforts to make it relevant in that field.
> Most spot on thing I've read in a while. Thanks Glenn.
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf