[Beowulf] [EXTERNAL] Re: UNCHECKED Re: Spark, Julia, OpenMPI etc. - all in one place

Thu Oct 15 08:56:45 PDT 2020

I don’t know that Quantum computing is something that, say, Bank of America, or the IRS, would be able to effectively leverage.  What they do isn’t computationally complex – the vast majority of the workload is “pick up data from place A, look at it, do something or not, put it back in place A, and potentially generate some new data in place B”, but it’s exceedingly requirements intensive. Consider something simple like handling a monthly mortgage deposit – at the core, it’s “bump current balance with amount of check”, but wait, what if they’re in foreclosure? Or have a forbearance? Or the mortgage is held by an active duty servicemember? And the exception handling is even more complex – what country, state and/or county is the property in, the owner, the lien holder?

There are thousands upon thousands of business rules which have to be continuously maintained and audited for regulatory compliance.  The limiting resource isn’t computational horsepower, it’s organizing the thousands of processes, which is primarily a people problem, not a computer problem.  Their existing work and data flows are already parallelized in some sense, and if they need to do it faster, they just add processors or storage as needed.

From: atul kumthekar <atul.b.kumthekar at gmail.com>
Date: Thursday, October 15, 2020 at 8:08 AM
To: Jim Lux <james.p.lux at jpl.nasa.gov>
Cc: Oddo Da <oddodaoddo at gmail.com>, John Hearns <hearnsj at gmail.com>, "beowulf at beowulf.org" <beowulf at beowulf.org>
Subject: Re: [Beowulf] [EXTERNAL] Re: ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

with Quantum Computing on the horizon, there may be major change in the direction.

On Thu, Oct 15, 2020 at 7:46 PM Lux, Jim (US 7140) via Beowulf <beowulf at beowulf.org<mailto:beowulf at beowulf.org>> wrote:
Not all offensive..
It’s always useful to take a step back and say “well, rather than incremental change X, what about wholesale change Y”.

One interesting phenomenon, too, is that once a large, complex system has been around a while, it becomes the embodiment of the requirements that produced it, yet those requirements are not found anywhere (at least not in a coherent single source). So the risk of new implementation is enormous, since the probability of the new system not properly implementing a requirement is large.  If your system is, say, processing airline reservations or income tax returns, the cost of a problem is enormous.   It doesn’t take many multi-million dollar “oopsies” to make the cost of half a dozen skilled software developers to tinker at the edges negligible.

From: Beowulf <beowulf-bounces at beowulf.org<mailto:beowulf-bounces at beowulf.org>> on behalf of Oddo Da <oddodaoddo at gmail.com<mailto:oddodaoddo at gmail.com>>
Date: Thursday, October 15, 2020 at 4:10 AM
To: John Hearns <hearnsj at gmail.com<mailto:hearnsj at gmail.com>>
Cc: "beowulf at beowulf.org<mailto:beowulf at beowulf.org>" <beowulf at beowulf.org<mailto:beowulf at beowulf.org>>
Subject: Re: [Beowulf] [EXTERNAL] Re: ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

On Thu, Oct 15, 2020 at 1:11 AM John Hearns <hearnsj at gmail.com<mailto:hearnsj at gmail.com>> wrote:
This has been a great discussion. Please keep it going.

I am all out of ammo ;). In all seriousness, it is not easy to ask these questions because it kind of can be interpreted as offensive - in a nutshell, people may perceive what I am asking as "what have y'all been doing for 20 years? Nothing?".

To the points on technical debt, may I also add re-validation?
Let's say you have a weather model which your institute has been running for 20 years.
If you decide to start again from fresh with code in a new language you are going to have to re-run known models
and debate whether or not they fit within error bounds of the old model.
That takes effort - which may of course be justified if you make gains in speed, flexibility or being able to use new hardware like GPUs.

I understand all this but, of course, not everything has to do what has been done. Hopefully, there are plenty of people entering the field or coming back to it, without any technical debt.

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf<https://urldefense.us/v3/__https:/beowulf.org/cgi-bin/mailman/listinfo/beowulf__;!!PvBDto6Hs4WbVuu7!bodty4KlKmA1LZygeemiAxXLmSrV-HOO7syq9oZVVW6Rz9HnK7AunQPqYEB-Q7L1Uy0b7KI$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201015/6bc98a0c/attachment-0001.html>