[Beowulf] large MPI adopters
Vincent Diepeveen
diep at xs4all.nl
Tue Oct 7 08:33:52 PDT 2008
On Oct 7, 2008, at 3:52 PM, Gerry Creager wrote:
> I'm top posting, 'cause everyone else is. It's easier.
>
> <informative>
> Andrea, a lot of scientific computation uses big MPI. Off the top
> of my head, and based on examples I'm personally aware of (but
> don't work in directly), seismic and reservoir flow models are big
> MPI operations. Think "oil company", e.g., Shell, Chevron, Exxon-
> Mobil, etc., I do know that, in Houston, all employ big clusters
> running MPI on a regular basis. Quantum chemistry is often run as
> an MPI job. Weather modeling is a big MPI user, as are ocean and
> coastal models run at universities, government facilities (not
> solely national labs) and corporate entities. NASA does a fair bit
> of MPI work... where did Beowulfery start (Don, that's a cue)? We
> have a group using MPI applications in genomics research, and
> others using it for physical chemistry modeling, LHC data
> operations/reduction/interpretation, and nuclear systems
> prognostication.
>
> The list isn't small: As Dan mentions, the drug companies are big
> users, although there's a fair bit of openMPI operation in their
> codes, too. CFD in aerospace, and auto manufacturers. The US
> military almost certainly does parallel processing involving both
> message passing and Monte Carlo simulations. I've written code
> that solves physical and satellite geodesy problems using PVM (but
> I was young and that was a long time ago; today I'd use MPI); a
> variant of that was apparently moved to the production world after
> I changed positions and couldn't support it anymore.
>
> I'm not sure to tell you who the biggest corporate users are. It
> likely depends on one's exposure to even have an opinion. However,
> for the biggest
You raise an interesting question.
Military type things not counted, biggest corporate users of generic
cpu's is car industry.
This is quite logical: safety is everything with cars and a car has
far over 50+ processors.
Not seldom 100 nowadays.
Drug type companies eat typical 0.5% system time.
They are far behind, which is not so logical; i would consider it old
fashioned to do experiments
on monkeys (Bar-ilan brain experiments on monkeys rings a bell?), so
doing everything in software is far more logical.
The big question is: should one just look to how many generic cpu's
one uses?
When someone is really in big need for big calculation power, one
makes dedicated cpu's for it.
It may come...
In most embedded devices that badly need high levels of encryption,
typically a coprocessor gets put in to do the job.
Such co processor is really mighty if one would compare it with how
many Tflop such a thing is when putting in generic cpu's.
If you look at it from that viewpoint, ISPs are worlds biggest
corporate users of hardware.
Now go back in your corner and secure the internet :)
> corporate users I'm aware of in my immediate area, Schlumberger,
> Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is
> obviously a very incomplete list.
> </informative>
> Gerry
>
> Vincent Diepeveen wrote:
>> Hi Dan,
>> I'd rather guess his question is more: "i want a big list of 10000
>> companies that are actively using clusters".
>> That list is not there and if it was there it is utmost secretly
>> compiled by some NSA type instance.
>> I'm now speaking more for western Europe, not necessarily for USA:
>> The research that companies perform is usually applied research:
>> the minimum requirement to produce a new product,
>> or when forced at gunpoint by government for safety reasons.
>> For fundamental research usually most computational resources get
>> thrown into battle, which is for nearly 100% sponsored
>> by government be it directly or indirectly.
>> In fact there is entire industries, especially clean energy
>> research, that is 100% paid by government subsidies.
>> Investigations after new products, in 99% of cases get funded by
>> government; usually the name of a big company is behind it,
>> but it's still government dudes who sit in that company doing the
>> research. What matters of course is who pays it.
>> The exception is persons who have their own company and are some
>> sort of big expert at a certain area. Take me.
>> I want to do some research after learning algorithms, at a very
>> high level (so far they didn't even conclude the right
>> value for material for chess, let alone that they were useful at
>> anything other than debugging your code, which is a big
>> important thing by the way).
>> Without getting the help of someone working at an university to
>> run such a thing overnight at a 160-200 cores,
>> i would of course only be able to run it at a single pc. Now i
>> don't get any money for all this from government,
>> so the big question is whether i'm supported by government then or
>> not.
>> What i do know is that all public research here at government
>> level is major shit. This is explainable by the fact that a field
>> is nonstop
>> on the move, only those who have been busy in a field for 10+
>> years have any clue what happens there and are the first to try
>> something;
>> the PHD's are simply busy with inferior software meanwhile having
>> massive hardware that's idle. A big problem is the nonpublishing of
>> accomplishments; First of all i'm not publishing *how* i'm doing
>> the learning. When some university offers me a job, i might
>> publish things,
>> i'm not getting paid for publications. Why help "the competition"?
>> What might get published is the guy who helps me; one tiny thing
>> of all this he'll publish probably. That's maybe 1/100 of the
>> conclusions
>> drawn.
>> Only the NSA type organisations know as they all spy on the
>> internet, and you BET that all big
>> countries know exactly what i'm doing, they all tap the internet
>> like crazy there.
>> If 1 researcher is real brilliant and CAN need some big
>> computation, there is at least a 100 spies who do know something
>> of that field,
>> and have to interpret the tapped data. What i do not know, as i
>> don't work for any of the N*SA type organisations, is in how far
>> *those*
>> verify things using massive hardware.
>> Most researchers have no clue how big the spying and anti-spying
>> is; most might get total scared if they'd realized how much they
>> get protected.
>> It is easier to steal it than to find an Einstein in your own
>> nation figuring it out.
>> That is the principle that every nation uses; CIA is a very tiny
>> organisation compared to what other nations have,
>> note that the weird thing is that CIA is notorious for violating
>> agreements with other nations (not spying in friendly nations,
>> to give an example; and also passing on information to their
>> companies that they got from the information stream they
>> got from friendly nations spying onto their own companies).
>> The fact that i write this down already means i have not been
>> employed in that field nor am; otherwise i would not even
>> *mention* the word.
>> Or to quote someone who works in that field when i said that there
>> gotta be an awful lot of civil servants in Netherlands busy in
>> that field,
>> as there is 3+ million tourists a year in Netherlands:
>> "we have tourism in Spain also".
>> Yet nothing on the internet is safe, that's the basic problem
>> here. It's too easy to hack internet lines. 1024 bits or 2048 RSA
>> or something that
>> gets used for SSH?
>> 1024 bits is too easy to crack for 1 organisation of each country
>> by simply throwing specialized hardware at it.
>> 2048 bits RSA is a tad harder, but also not a problem.
>> 128 bits AES?
>> No manner to attack it is public known (would only deliver $20k,
>> which is too little anyway, for the risk you take publishing a
>> method).
>> Yet even if there would be some sort of randomized GA-algorithm
>> that needs some sort of Pollard-Rho order O( 2 ^ 0.25n ),
>> then a simple dedicated cpu can already crack it handsdown.
>> Obviously most companies are not busy spamming the net what they
>> do with clusters or do not do.
>> Keeping it secret for their competitors is just too important, yet
>> if you ask me i feel big companies do real little
>> research in areas that do not lead directly to products of them.
>> There might be 1 or 2 exceptions, like oil companies,
>> but the question is in how far researchers there can be seen as
>> employees of that company as they get indirectly
>> sponsored by government whose interest in everything that happens
>> with oil is real big.
>> If you ask me however, way too little fundamental research gets
>> sponsored by government.
>> If you see just in a tiny nation like netherlands; the number of
>> 'researchers' that work for government is just a very small part of
>> the number of professors. It's like 1 researcher for each 2
>> professors; PHD's not counted.
>> Note this is also what a few studies recently showed.
>> Just the comparision to intelligence agencies which can put into
>> action each one of them quarter of a million to millions of people,
>> and who hire real easy people be it direct or indirect, it is
>> obvious that the saying: "Better well stolen than bad invented",
>> gets written in just too big capitals.
>> As we're speaking of a 1000 individuals in a nation having 16.5
>> million inhabitants, and only very few of those have
>> time to devote a big part of their time to do research; maybe 5% is?
>> Most are too busy with meetings and other organisational work and
>> posting on the internet.
>> Realize that same nation also has over a 1000 companies with
>> more than 1000 employees and half a million charity organisations.
>> That really lets fundamental research look like something that
>> gets total neglected.
>> Vincent
>> On Oct 7, 2008, at 9:34 AM, <Dan.Kidger at quadrics.com>
>> <Dan.Kidger at quadrics.com> wrote:
>>> Andrea,
>>>
>>> MPI is of course used by many applications running on commercial
>>> clusters.
>>> Two obvious examples are computational chemistry by the drug
>>> companies
>>> and computational fluid dynamics (CFD) for aerospace companies
>>> and F1 design teams.
>>>
>>> These are all long-term 'traditional' uses of MPI for scientific/
>>> engineering codes.
>>>
>>> Is this what you are asking? Or are you thinking of non-
>>> traditional uses in say computational finance or gaming sites?
>>>
>>> Daniel
>>>
>>> -------------------------------------------------------------
>>> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger at quadrics.com
>>> One Bridewell St., Mobile: +44 (0)779 209 1851
>>> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519
>>> ----------------------- www.quadrics.com --------------------
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: beowulf-bounces at beowulf.org [mailto:beowulf-
>>> bounces at beowulf.org] On Behalf Of Andrea Di Blas
>>> Sent: 13 August 2008 00:37
>>> To: beowulf at beowulf.org
>>> Subject: [Beowulf] large MPI adopters
>>>
>>> hello,
>>>
>>>
>>> I am curious about what companies, besides the national labs of
>>> course,
>>> use any implementation of MPI to support large applications of
>>> any kind,
>>> whether only internally (like mapreduce for google, for example)
>>> or not.
>>>
>>> does anybody know of any cases?
>>> thank you and best regards,
>>>
>>>
>>> andrea
>>>
>>>
>>>
>>>
>>> --
>>> Andrea Di Blas, UCSC
>>> School of Engineering
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> Gerry Creager -- gerry.creager at tamu.edu
> Texas Mesonet -- AATLT, Texas A&M University
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
>
More information about the Beowulf
mailing list