[Beowulf] large MPI adopters
Gerry Creager
gerry.creager at tamu.edu
Tue Oct 7 06:52:54 PDT 2008
I'm top posting, 'cause everyone else is. It's easier.
<soapbox>
Vincent, there are a lot of folks interested in computer and
infrastructure security. They are interested in making it more secure
and interested in exploiting its holes. And these are the "good guys".
Stick to chess. Don't blather about security, an area you appear to
be less aware of than the board games. I don't tell you how to be a
chess grand master, and you don't tell me about security from a purely
speculative view... OK?
</soapbox>
<informative>
Andrea, a lot of scientific computation uses big MPI. Off the top of my
head, and based on examples I'm personally aware of (but don't work in
directly), seismic and reservoir flow models are big MPI operations.
Think "oil company", e.g., Shell, Chevron, Exxon-Mobil, etc., I do know
that, in Houston, all employ big clusters running MPI on a regular
basis. Quantum chemistry is often run as an MPI job. Weather modeling
is a big MPI user, as are ocean and coastal models run at universities,
government facilities (not solely national labs) and corporate entities.
NASA does a fair bit of MPI work... where did Beowulfery start (Don,
that's a cue)? We have a group using MPI applications in genomics
research, and others using it for physical chemistry modeling, LHC data
operations/reduction/interpretation, and nuclear systems prognostication.
The list isn't small: As Dan mentions, the drug companies are big users,
although there's a fair bit of openMPI operation in their codes, too.
CFD in aerospace, and auto manufacturers. The US military almost
certainly does parallel processing involving both message passing and
Monte Carlo simulations. I've written code that solves physical and
satellite geodesy problems using PVM (but I was young and that was a
long time ago; today I'd use MPI); a variant of that was apparently
moved to the production world after I changed positions and couldn't
support it anymore.
I'm not sure to tell you who the biggest corporate users are. It likely
depends on one's exposure to even have an opinion. However, for the
biggest corporate users I'm aware of in my immediate area, Schlumberger,
Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is
obviously a very incomplete list.
</informative>
Gerry
Vincent Diepeveen wrote:
> Hi Dan,
>
> I'd rather guess his question is more: "i want a big list of 10000
> companies that are actively using clusters".
> That list is not there and if it was there it is utmost secretly
> compiled by some NSA type instance.
>
> I'm now speaking more for western Europe, not necessarily for USA:
>
> The research that companies perform is usually applied research: the
> minimum requirement to produce a new product,
> or when forced at gunpoint by government for safety reasons.
>
> For fundamental research usually most computational resources get thrown
> into battle, which is for nearly 100% sponsored
> by government be it directly or indirectly.
>
> In fact there is entire industries, especially clean energy research,
> that is 100% paid by government subsidies.
> Investigations after new products, in 99% of cases get funded by
> government; usually the name of a big company is behind it,
> but it's still government dudes who sit in that company doing the
> research. What matters of course is who pays it.
>
> The exception is persons who have their own company and are some sort of
> big expert at a certain area. Take me.
> I want to do some research after learning algorithms, at a very high
> level (so far they didn't even conclude the right
> value for material for chess, let alone that they were useful at
> anything other than debugging your code, which is a big
> important thing by the way).
>
> Without getting the help of someone working at an university to run such
> a thing overnight at a 160-200 cores,
> i would of course only be able to run it at a single pc. Now i don't get
> any money for all this from government,
> so the big question is whether i'm supported by government then or not.
>
> What i do know is that all public research here at government level is
> major shit. This is explainable by the fact that a field is nonstop
> on the move, only those who have been busy in a field for 10+ years have
> any clue what happens there and are the first to try something;
> the PHD's are simply busy with inferior software meanwhile having
> massive hardware that's idle. A big problem is the nonpublishing of
> accomplishments; First of all i'm not publishing *how* i'm doing the
> learning. When some university offers me a job, i might publish things,
> i'm not getting paid for publications. Why help "the competition"?
> What might get published is the guy who helps me; one tiny thing of all
> this he'll publish probably. That's maybe 1/100 of the conclusions
> drawn.
> Only the NSA type organisations know as they all spy on the internet,
> and you BET that all big
> countries know exactly what i'm doing, they all tap the internet like
> crazy there.
>
> If 1 researcher is real brilliant and CAN need some big computation,
> there is at least a 100 spies who do know something of that field,
> and have to interpret the tapped data. What i do not know, as i don't
> work for any of the N*SA type organisations, is in how far *those*
> verify things using massive hardware.
>
> Most researchers have no clue how big the spying and anti-spying is;
> most might get total scared if they'd realized how much they
> get protected.
>
> It is easier to steal it than to find an Einstein in your own nation
> figuring it out.
> That is the principle that every nation uses; CIA is a very tiny
> organisation compared to what other nations have,
> note that the weird thing is that CIA is notorious for violating
> agreements with other nations (not spying in friendly nations,
> to give an example; and also passing on information to their companies
> that they got from the information stream they
> got from friendly nations spying onto their own companies).
>
> The fact that i write this down already means i have not been employed
> in that field nor am; otherwise i would not even *mention* the word.
>
> Or to quote someone who works in that field when i said that there gotta
> be an awful lot of civil servants in Netherlands busy in that field,
> as there is 3+ million tourists a year in Netherlands:
> "we have tourism in Spain also".
>
> Yet nothing on the internet is safe, that's the basic problem here. It's
> too easy to hack internet lines. 1024 bits or 2048 RSA or something that
> gets used for SSH?
>
> 1024 bits is too easy to crack for 1 organisation of each country by
> simply throwing specialized hardware at it.
> 2048 bits RSA is a tad harder, but also not a problem.
>
> 128 bits AES?
>
> No manner to attack it is public known (would only deliver $20k, which
> is too little anyway, for the risk you take publishing a method).
> Yet even if there would be some sort of randomized GA-algorithm that
> needs some sort of Pollard-Rho order O( 2 ^ 0.25n ),
> then a simple dedicated cpu can already crack it handsdown.
>
> Obviously most companies are not busy spamming the net what they do with
> clusters or do not do.
> Keeping it secret for their competitors is just too important, yet if
> you ask me i feel big companies do real little
> research in areas that do not lead directly to products of them. There
> might be 1 or 2 exceptions, like oil companies,
> but the question is in how far researchers there can be seen as
> employees of that company as they get indirectly
> sponsored by government whose interest in everything that happens with
> oil is real big.
>
> If you ask me however, way too little fundamental research gets
> sponsored by government.
> If you see just in a tiny nation like netherlands; the number of
> 'researchers' that work for government is just a very small part of
> the number of professors. It's like 1 researcher for each 2 professors;
> PHD's not counted.
>
> Note this is also what a few studies recently showed.
>
> Just the comparision to intelligence agencies which can put into action
> each one of them quarter of a million to millions of people,
> and who hire real easy people be it direct or indirect, it is obvious
> that the saying: "Better well stolen than bad invented",
> gets written in just too big capitals.
>
> As we're speaking of a 1000 individuals in a nation having 16.5 million
> inhabitants, and only very few of those have
> time to devote a big part of their time to do research; maybe 5% is?
>
> Most are too busy with meetings and other organisational work and
> posting on the internet.
> Realize that same nation also has over a 1000 companies with
> more than 1000 employees and half a million charity organisations.
>
> That really lets fundamental research look like something that gets
> total neglected.
>
> Vincent
>
> On Oct 7, 2008, at 9:34 AM, <Dan.Kidger at quadrics.com>
> <Dan.Kidger at quadrics.com> wrote:
>
>> Andrea,
>>
>> MPI is of course used by many applications running on commercial
>> clusters.
>> Two obvious examples are computational chemistry by the drug companies
>> and computational fluid dynamics (CFD) for aerospace companies and F1
>> design teams.
>>
>> These are all long-term 'traditional' uses of MPI for
>> scientific/engineering codes.
>>
>> Is this what you are asking? Or are you thinking of non-traditional
>> uses in say computational finance or gaming sites?
>>
>> Daniel
>>
>> -------------------------------------------------------------
>> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
>> One Bridewell St., Mobile: +44 (0)779 209 1851
>> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
>> ----------------------- www.quadrics.com --------------------
>>
>>
>>
>>
>> -----Original Message-----
>> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
>> On Behalf Of Andrea Di Blas
>> Sent: 13 August 2008 00:37
>> To: beowulf at beowulf.org
>> Subject: [Beowulf] large MPI adopters
>>
>> hello,
>>
>>
>> I am curious about what companies, besides the national labs of course,
>> use any implementation of MPI to support large applications of any kind,
>> whether only internally (like mapreduce for google, for example) or not.
>>
>> does anybody know of any cases?
>> thank you and best regards,
>>
>>
>> andrea
>>
>>
>>
>>
>> --
>> Andrea Di Blas, UCSC
>> School of Engineering
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
--
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
More information about the Beowulf
mailing list