[Beowulf] [EXTERNAL] Re: ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

[Lux, Jim (US 7140)]

Not all offensive..
It’s always useful to take a step back and say “well, rather than incremental change X, what about wholesale change Y”.

One interesting phenomenon, too, is that once a large, complex system has been around a while, it becomes the embodiment of the requirements that produced it, yet those requirements are not found anywhere (at least not in a coherent single source). So the risk of new implementation is enormous, since the probability of the new system not properly implementing a requirement is large.  If your system is, say, processing airline reservations or income tax returns, the cost of a problem is enormous.   It doesn’t take many multi-million dollar “oopsies” to make the cost of half a dozen skilled software developers to tinker at the edges negligible.




From: Beowulf <beowulf-bounces at beowulf.org> on behalf of Oddo Da <oddodaoddo at gmail.com>
Date: Thursday, October 15, 2020 at 4:10 AM
To: John Hearns <hearnsj at gmail.com>
Cc: "beowulf at beowulf.org" <beowulf at beowulf.org>
Subject: Re: [Beowulf] [EXTERNAL] Re: ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

On Thu, Oct 15, 2020 at 1:11 AM John Hearns <hearnsj at gmail.com<mailto:hearnsj at gmail.com>> wrote:
This has been a great discussion. Please keep it going.

I am all out of ammo ;). In all seriousness, it is not easy to ask these questions because it kind of can be interpreted as offensive - in a nutshell, people may perceive what I am asking as "what have y'all been doing for 20 years? Nothing?".

To the points on technical debt, may I also add re-validation?
Let's say you have a weather model which your institute has been running for 20 years.
If you decide to start again from fresh with code in a new language you are going to have to re-run known models
and debate whether or not they fit within error bounds of the old model.
That takes effort - which may of course be justified if you make gains in speed, flexibility or being able to use new hardware like GPUs.

I understand all this but, of course, not everything has to do what has been done. Hopefully, there are plenty of people entering the field or coming back to it, without any technical debt.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201015/ef9a6b4e/attachment.html>