[Beowulf] Station wagon full of tapes

[Jeff Layton]

Gerry Creager wrote:
> There was an interesting brainstorming session at Rocks-A-Palooza a 
> couple of weeks ago.  Someone wants to offer Amazon resources.  
> Problem remains for me: How can I get sufficient cloud resources for 
> computing (I'll hammer on dataset transport in a moment) that will 
> handle reasonable weather models with their small message MPI chatter, 
> and lots of file I/O?  I've been assured that Amazon's ready to 
> accommodate that.

This is one of the problems - clouds aren't ready for this kind of
usage model yet. They only have GigE and usually it's oversubscribed.
When you say file IO, they hear capacity, not performance (either
throughput or IOPS). And as you point out, the pipe to/from the
cloud is not ready for lots of data.

> However, getting data into S3 for availability, when a daily 
> multi-gigabyte dataset is used for initiation, and another is created 
> as output, is going to be expensive, and likely slow.  I think there 
> are other approaches that have to be evaluated.  I am not sure the 
> cloud is ready for MPI play on a significant basis, just yet.

I haven't seen the cloud ready yet for anything other than embarrassingly
parallel codes (i.e. since node, small IO requirements). Has anyone seen
differently? (as an example of what might work, CloudBurst seems to be
gaining some traction - doing sequencing in the cloud. The only problem
is that sequencing can generate a great deal of data pretty rapidly).

Jeff