[Beowulf] Station wagon full of tapes

Robert G. Brown rgb at phy.duke.edu
Tue May 26 09:26:19 PDT 2009


On Tue, 26 May 2009, Chris Dagdigian wrote:

>
> The flip side to your arguments is that I may not want my tax dollars spent 
> on allowing the NIH to operate peta-scale data repositories. I can't be more 
> specific than this -- my most recent exposure to a large government life 
> science directorate revealed that they were spending $500K/year on EMC 
> maintenance costs for a few tens-of-TBs worth of disk arrays that were going 
> on 6 years old!

Yeah, well, stupidity is a universal problem, even in the
government...;-) But this is why CBAs and smart people (working
together) are so important.

> I think my main interest in utility storage providers is that they can offer 
> geographical redundancy and large capacity at efficiencies that can't be 
> matched locally by individual institutions or even local groups of 
> institutions. When I look at the full costs of hosting, operating and 
> replicating the data in a local facility the numbers from the "utility" 
> providers start to look more attractive.
>
> It will be interesting to see how this all shakes out. The rate at which raw 
> disk cost is shrinking in price is amazing and may choke off the profit for 
> the utility providers who have invested heavily in building out.

I agree.  Moore's Law applies to more than just processors.  If only it
applied to bandwidth....;-)

   rgb

>
> My $.02 of course!
>
>
>
>
>
>
> On May 26, 2009, at 11:16 AM, Robert G. Brown wrote:
>
>> On Tue, 26 May 2009, Chris Dagdigian wrote:
>> 
>>> 
>>> I deal quite often with the "next-gen" DNA sequencing instruments that 
>>> produce 1TB/day in TIFF images that are then distilled down to the DNA 
>>> basecalls before the short reads are subjected to alignment. Then the 
>>> resulting longer sequences are usually aligned again against a reference 
>>> genome.
>>> 
>>> Lots of data, lots of computation.
>>> 
>>> The 1 Terabyte of TIFF images typically reduces down to about 200 GB in 
>>> intermediate data which is further distilled down into a few hundred KB of 
>>> actual sequence data. The entire process is interesting and it is a 
>>> massive Bio/IT challenge as these types of terabyte-scale data producing 
>>> lab instruments are popping up everywhere (the cost of one of these 
>>> instruments is now easily within reach of a single grant-funded researcher 
>>> at a facility of any size...). We are only a few technology revolutions 
>>> away from these boxes showing up in your point of care primary physician's 
>>> office (well not really, probably a backend service lab that your 
>>> physician outsources to ...)
>>> 
>>> Anyway the new data ingestion service that Amazon offers is, I think, 
>>> going to be a big deal in our field.
>> 
>> Sure, but why wouldn't it be cheaper for e.g. NSF or NIH to fund an
>> exact clone of the service Amazon plans to offer and provide it for free
>> to its supported research groups (or rather, do bookkeeping but it is
>> all internal bookkeeping, moving money from one pocket to another).
>> 
>> Amazon has to make a profit.  Granting agencies don't have to pay the
>> profit that Amazon has to make.  Amazon has to take substantial risks to
>> make its profit.  Granting agencies have no risk.
>> 
>> All of the things you assert for DNA sequencing are true for high energy
>> physics.  Enormous datasets, lots of computation.  HEP's INTERNATIONAL
>> solution is ATLAS, not Amazon.
>> 
>> Supporting commercial access into such a DB a la >>google<< but for
>> genomic data, sure, but that's not really cluster computing, that's a
>> large shared DB.  I could see that as a spin off data service of Amazon
>> or Google or a new business altogether, but I'd view it as a niche and
>> not really HPC.
>> 
>> Grant funded research involving large scale shared data resources can
>> ALWAYS be done more cheaply than by buying the data services from
>> profit-making third parties unless there are nonlinear e.g. proprietary
>> IP barriers.  This is trebly true given that research facilities are
>> typically on a very high speed networks e.g. lambda rail that the
>> government is funding anyway, where Amazon or other commercial third
>> parties have to rent time on those networks and then resell the rental
>> back to the government at a profit or use slower commercial networks and
>> with the same sort of throughput markup.
>> 
>> Are there any such barriers here?  I'd have to say that I would be most
>> unhappy seeing my own tax dollars going to make Amazon shareholders rich
>> when they could be spent more efficiently without a middleman raking in
>> a 50 to 100% markup on the service.  Of course I'm easily irked -- when
>> I think of all the money spent on Windows by the US government it makes
>> my blood boil.
>> 
>> I'd want to see a solid CBA proving that this is the cheapest way to
>> proceed before dumping tons of tax money into it, if I were king of the
>> world (or just in charge of a major granting agency).
>>
>>  rgb
>> 
>>> 
>>> For the following reasons:
>>> 
>>> - Bio people are being buried in data
>>> - Once we process the data to get the derived results, the primary data 
>>> just needs to go somewhere cheap
>>> - Amazon and other internet-scale people can do peta-scale or exa-scale 
>>> storage far better & cheaper than any of my customers
>>> - These instruments are popping up in wet labs across campus with 
>>> weak/anemic network links to IT core facilities and data centers
>>> - Scientists in many cases are required to share data that is grant funded
>>> - Amazon has some neat "downloader pays" models that make it easier for 
>>> researchers to affordably offer up peta-scale data sets for sharing
>>> 
>>> I suspect that very large amount of scientific data will be making a 1-way 
>>> trip into the cloud. The data will stay there "forever" as a deep store. 
>>> In the ocasional cases where the data needs to be re-processed or 
>>> re-analyized it would be not unreasonable to fire up some cloud server 
>>> nodes to do the re-work in-situ.
>>> 
>>> The disk ingest service was the final piece. I can see this happening in 
>>> life science environments:
>>> 
>>> - Massive data generated in the wet lab
>>> - Captured to local storage (10 - 40TB) with small HPC component
>>> - Data is processed locally into derived and distilled forms
>>> - Derived data replicated to campus/lab facilities for online primary 
>>> storage
>>> - Derived data (and possibly the full raw data) is compressed, placed onto 
>>> drives and ingested into Amazon for long term storage
>>> - If re-analysis is ever needed, have existing EC2 AMIs preloaded with the 
>>> necessary software
>>> 
>>> Basically it comes down to the fact that Amazon may be able to offer 
>>> big-yet-slow storage in the terabyte to petabyte range at levels of cost 
>>> and geographical redundancy that would be extremely difficult to match 
>>> with local resources at a small non-specialized organization.
>>> 
>>> My $.02 of course
>>> 
>>> -Chris
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On May 26, 2009, at 8:58 AM, Jeff Layton wrote:
>>> 
>>>> Gerry Creager wrote:
>>>>> There was an interesting brainstorming session at Rocks-A-Palooza a 
>>>>> couple of weeks ago.  Someone wants to offer Amazon resources.  Problem 
>>>>> remains for me: How can I get sufficient cloud resources for computing 
>>>>> (I'll hammer on dataset transport in a moment) that will handle 
>>>>> reasonable weather models with their small message MPI chatter, and lots 
>>>>> of file I/O? I've been assured that Amazon's ready to accommodate that.
>>>> This is one of the problems - clouds aren't ready for this kind of
>>>> usage model yet. They only have GigE and usually it's oversubscribed.
>>>> When you say file IO, they hear capacity, not performance (either
>>>> throughput or IOPS). And as you point out, the pipe to/from the
>>>> cloud is not ready for lots of data.
>>>>> However, getting data into S3 for availability, when a daily 
>>>>> multi-gigabyte dataset is used for initiation, and another is created as 
>>>>> output, is going to be expensive, and likely slow.  I think there are 
>>>>> other approaches that have to be evaluated.  I am not sure the cloud is 
>>>>> ready for MPI play on a significant basis, just yet.
>>>> I haven't seen the cloud ready yet for anything other than embarrassingly
>>>> parallel codes (i.e. since node, small IO requirements). Has anyone seen
>>>> differently? (as an example of what might work, CloudBurst seems to be
>>>> gaining some traction - doing sequencing in the cloud. The only problem
>>>> is that sequencing can generate a great deal of data pretty rapidly).
>>>> Jeff
>>>> _______________________________________________
>>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>>> To change your subscription (digest mode or unsubscribe) visit 
>>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>> 
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit 
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>> 
>> Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
>> Duke University Dept. of Physics, Box 90305
>> Durham, N.C. 27708-0305
>> Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
>> 
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu





More information about the Beowulf mailing list