From james.p.lux at jpl.nasa.gov  Sun Dec  1 04:33:47 2019
From: james.p.lux at jpl.nasa.gov (Lux, Jim (US 337K))
Date: Sun, 1 Dec 2019 12:33:47 +0000
Subject: [Beowulf] [EXTERNAL] Re: Is Crowd Computing the Next Big Thing?
In-Reply-To: <754d862f-ccde-f95f-a7d8-9b8945b08a06@emailplus.org>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <754d862f-ccde-f95f-a7d8-9b8945b08a06@emailplus.org>
Message-ID: <DAEDBDA2-85AE-47D4-ADE2-6F1F37FAE783@jpl.caltech.edu>

Oh, there's no thought that MPI would be even remotely appropriate here, however, if someone is looking for a dissertation or thesis topic <grin>?
Virtually all of my recent HPC has been essentially EP jobs submitted through PBS, with no need for MPI (yet).


On 11/30/19, 10:40 PM, "Beowulf on behalf of Benson Muite" <beowulf-bounces at beowulf.org on behalf of benson_muite at emailplus.org> wrote:

    MPI is not likely the best programming model at present for such a 
    scenario, because fault tolerance is not great. However, there are 
    people running many loosely coupled parallel programs on clusters 
    because the computational capacity and storage mechanisms make it 
    convenient, even if the interconnect is heavily underutilized. Thus it 
    is more a matter of finding appropriate workloads. Many data centers 
    often have idle capacity, appropriate preferential pricing policies can 
    make use of this - not something HPC community typically considers, but 
    something cloud and grid computing people have been working on for a 
    while. Comet computer at SDSC is targetting some of these workloads.
    
    On 12/1/19 4:13 AM, Lux, Jim (US 337K) via Beowulf wrote:
    > But just think of the challenges of implementing MPI on a network of ephemerally connected cellphones. Or any of the cluster filesystems?  What about a scheduler/queue manager?  The nature of the computing nodes and communication would bring new meaning to the Byzantine General problem.
    >
    > This is an order of magnitude more complex than a cluster of RPi or Beagles, which, frankly, is a doddle.
    >
    > (I'm omitting the obligatory "what about a Beowulf cluster of X" comment, since we are, after all, the Beowulf mailing list and it goes without saying.
    >
    >
    > On 11/30/19, 1:38 PM, "Beowulf on behalf of Douglas Eadline" <beowulf-bounces at beowulf.org on behalf of deadline at eadline.org> wrote:
    >
    >      
    >      "Big Thing" as in over-hyped idea: Yes
    >      "Big Thing" as in practical use: No
    >      
    >      --
    >      Doug
    >      
    >      
    >      > Seen the below where a company wants to rent your smartphone as a cloud
    >      > computing resource. From a few years ago there was a company making space
    >      > heaters that contained servers to compute and heat your house.
    >      >
    >      > Are there any classes of problems that would be monitizeable in a grid
    >      > computing environment to make those efforts financially viable?
    >      >
    >      > Is Crowd Computing the Next Big Thing?
    >      > https://www.eejournal.com/article/is-crowd-computing-the-next-big-thing/
    >      >
    >      > Heating houses with 'nerd power'
    >      > https://www.bbc.com/news/magazine-32816775#<https://www.bbc.com/news/magazine-32816775>
    >      >
    >      > Chuck Petras, PE**
    >      > Schweitzer Engineering Laboratories, Inc
    >      > Pullman, WA  99163  USA
    >      > http://www.selinc.com
    >      >
    >      > SEL Synchrophasors - A New View of the Power System
    >      > <http://synchrophasor.selinc.com>
    >      >
    >      > Making Electric Power Safer, More Reliable, and More Economical (R)
    >      >
    >      > ** Registered in Oregon.
    >      >
    >      > _______________________________________________
    >      > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
    >      > To change your subscription (digest mode or unsubscribe) visit
    >      > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
    >      >
    >      
    >      
    >      --
    >      Doug
    >      
    >      _______________________________________________
    >      Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
    >      To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
    >      
    >
    > _______________________________________________
    > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
    > To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
    _______________________________________________
    Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
    To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
    


From deadline at eadline.org  Thu Dec 12 06:35:17 2019
From: deadline at eadline.org (Douglas Eadline)
Date: Thu, 12 Dec 2019 09:35:17 -0500
Subject: [Beowulf] Here we go again
In-Reply-To: <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
Message-ID: <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>


Anyone see anything like this with Epyc, i.e. poor AMD performance
when using Intel compilers or MKL?

https://www.pugetsystems.com/labs/hpc/AMD-Ryzen-3900X-vs-Intel-Xeon-2175W-Python-numpy---MKL-vs-OpenBLAS-1560/



-- 
Doug


From bdobbins at gmail.com  Thu Dec 12 07:06:03 2019
From: bdobbins at gmail.com (Brian Dobbins)
Date: Thu, 12 Dec 2019 08:06:03 -0700
Subject: [Beowulf] Here we go again
In-Reply-To: <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
 <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
Message-ID: <CAFkGP2mum3Kz5325-Cx7qzz4gph6UJDv_T8aYMHzWv8GXgq4_Q@mail.gmail.com>

Hi Doug,

  I've seen pretty decent performance on the AMD processors, and was even
told by AMD people to use the Intel compiler -- but when doing that, we
specify the processor type (eg, AVX capabilities), and it works pretty
well.  However, I don't have any experience using the MKL on them.

  That said, looking at the numbers, it's pretty interesting that there's
roughly a factor of 2 from the AVX2 (OpenBLAS) -> AVX512 (MKL) results on
Intel, and with the two systems being relatively comparable with OpenBLAS
(AVX2).  Then it's *roughly* a factor of 8 going from the MKL on Intel to
the MKL on AMD, and since AVX512 is 8 x 64 floats, it seems it could just
be it's not using any vectorization whatsoever on AMD... presumably because
Intel claims they can't recognize the chip?  That said, I'd love to see the
author try after setting:

MKL_ENABLE_INSTRUCTIONS=AVX2

  That might be an easy fix, if it works[1].

  Anyone got a Zen2 system with NumPy & the MKL to try it with?

  - Brian

[1]
https://software.intel.com/en-us/mkl-linux-developer-guide-instruction-set-specific-dispatching-on-intel-architectures


On Thu, Dec 12, 2019 at 7:35 AM Douglas Eadline <deadline at eadline.org>
wrote:

>
> Anyone see anything like this with Epyc, i.e. poor AMD performance
> when using Intel compilers or MKL?
>
>
> https://www.pugetsystems.com/labs/hpc/AMD-Ryzen-3900X-vs-Intel-Xeon-2175W-Python-numpy---MKL-vs-OpenBLAS-1560/
>
>
>
> --
> Doug
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20191212/94044abc/attachment.html>

From cap at nsc.liu.se  Thu Dec 12 08:29:38 2019
From: cap at nsc.liu.se (Peter =?UTF-8?B?S2plbGxzdHLDtm0=?=)
Date: Thu, 12 Dec 2019 17:29:38 +0100
Subject: [Beowulf] Here we go again
In-Reply-To: <CAFkGP2mum3Kz5325-Cx7qzz4gph6UJDv_T8aYMHzWv8GXgq4_Q@mail.gmail.com>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
 <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
 <CAFkGP2mum3Kz5325-Cx7qzz4gph6UJDv_T8aYMHzWv8GXgq4_Q@mail.gmail.com>
Message-ID: <20191212172938.39c92e1e@yaydoe>

On Thu, 12 Dec 2019 08:06:03 -0700
Brian Dobbins <bdobbins at gmail.com> wrote:

> Hi Doug,
> 
>   I've seen pretty decent performance on the AMD processors, and was
> even told by AMD people to use the Intel compiler -- but when doing
> that, we specify the processor type (eg, AVX capabilities), and it
> works pretty well.  However, I don't have any experience using the
> MKL on them.
> 
>   That said, looking at the numbers, it's pretty interesting that
> there's roughly a factor of 2 from the AVX2 (OpenBLAS) -> AVX512
> (MKL) results on Intel, and with the two systems being relatively
> comparable with OpenBLAS (AVX2).  Then it's *roughly* a factor of 8
> going from the MKL on Intel to the MKL on AMD, and since AVX512 is 8
> x 64 floats, it seems it could just be it's not using any
> vectorization whatsoever on AMD... presumably because Intel claims
> they can't recognize the chip?  That said, I'd love to see the author
> try after setting:
> 
> MKL_ENABLE_INSTRUCTIONS=AVX2

This does not work on MKL+zen (it also indeed defaults to non-simd MKL).

But setting MKL_DEBUG_CPU_TYPE=5 has an effect (essentially turns off
cpu detect and sets it to type 5, which is AVX2).

As far as the compilers go they seem to run ok on AMD with some caveats:

 * a binary built with -xAVX2 will fail due intel cpu req.
 * a binary built with -march=core-avx2 works ok

YMMV,
 Peter K

From i.n.kozin at googlemail.com  Thu Dec 12 11:50:08 2019
From: i.n.kozin at googlemail.com (INKozin)
Date: Thu, 12 Dec 2019 19:50:08 +0000
Subject: [Beowulf] Here we go again
In-Reply-To: <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
 <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
Message-ID: <CAPOouzAN9_ksFTWQfT-+myKkq81n0kv=SdPW=ycqp_MiCS-yHQ@mail.gmail.com>

All i can say is that AMD has finally released its own build of HPL and
that wasn't the case when Naples came out.

On Thu, 12 Dec 2019, 14:35 Douglas Eadline, <deadline at eadline.org> wrote:

>
> Anyone see anything like this with Epyc, i.e. poor AMD performance
> when using Intel compilers or MKL?
>
>
> https://www.pugetsystems.com/labs/hpc/AMD-Ryzen-3900X-vs-Intel-Xeon-2175W-Python-numpy---MKL-vs-OpenBLAS-1560/
>
>
>
> --
> Doug
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20191212/262ab42a/attachment.html>

From migueldiascosta at gmail.com  Thu Dec 12 17:54:02 2019
From: migueldiascosta at gmail.com (Miguel Dias Costa)
Date: Fri, 13 Dec 2019 09:54:02 +0800
Subject: [Beowulf] Here we go again
In-Reply-To: <20191212172938.39c92e1e@yaydoe>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
 <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
 <CAFkGP2mum3Kz5325-Cx7qzz4gph6UJDv_T8aYMHzWv8GXgq4_Q@mail.gmail.com>
 <20191212172938.39c92e1e@yaydoe>
Message-ID: <7905de37-3c9f-4a34-21db-8ff2afeed2a4@gmail.com>


On 13/12/2019 00:29, Peter Kjellström wrote:
> But setting MKL_DEBUG_CPU_TYPE=5 has an effect (essentially turns off
> cpu detect and sets it to type 5, which is AVX2).

I have been using MKL_DEBUG_CPU_TYPE=5 on Naples with the 2018 versions 
of MKL with good results, but when switching to 2019 versions some 
application tests started failing (dramatically wrong results) in FFT 
dependent stuff (and no failure without MKL_DEBUG_CPU_TYPE=5).

I could also reproduce this behaviour in some of MKL's own FFT examples, 
so it shouldn't be hard for other people to check.

Setting both MKL_DEBUG_CPU_TYPE=5 and MKL_CBWR=COMPATIBLE solved the 
issue, and MKL is still using AVX2, but it does mean one needs to be 
careful with MKL_DEBUG_CPU_TYPE...

My two (sing)cents,

Miguel



From bill at cse.ucdavis.edu  Fri Dec 13 06:29:37 2019
From: bill at cse.ucdavis.edu (Bill Broadley)
Date: Fri, 13 Dec 2019 06:29:37 -0800
Subject: [Beowulf] Here we go again
In-Reply-To: <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
 <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
Message-ID: <a785fa04-25ef-03c3-faec-18de951572cd@cse.ucdavis.edu>

On 12/12/19 6:35 AM, Douglas Eadline wrote:
> 
> Anyone see anything like this with Epyc, i.e. poor AMD performance
> when using Intel compilers or MKL?
> 
> https://www.pugetsystems.com/labs/hpc/AMD-Ryzen-3900X-vs-Intel-Xeon-2175W-Python-numpy---MKL-vs-OpenBLAS-1560/

I as getting anomalously slow performance with Matlab on AMD Rome.  Turns out 
Matlab uses the MKL and Intel ignores the AMD vector unit, despite the CPU 
capability flag showing that AMD support AVX2-256.

Turns out "export MKL_DEBUG_CPU_TYPE=5" fixed it.  The improvement is highly 
variable, but in one case was 6x faster.

Awhile back I tinkered with AOCC, AMD's tweak of clang/flang for Zen.  Had 
issues compiling OpenMPI (and dependencies), fftw3, OpenBLAS and friends.  I 
tried again with AOCC-2.1 and had no problems.  Generally things are looking 
good and we are considering supporting an AOCC based platform for our users to 
make the most of the AMD Rome chips.  Kudos to AMD for working on AOCC and 
working on upstreaming the changes to the numerous projects involved.

Related projects of possible interest:

  AMD FFTW3 = https://github.com/amd/amd-fftw

  BLIS/BLAS = https://github.com/amd/blis

  AMD Math library/Libm = 
http://developer.amd.com/amd-cpu-libraries/amd-math-library-libm/

  AMD LAPACK/Libflame = https://github.com/amd/libflame

  AMD Clang/Flang/LLVM = https://developer.amd.com/amd-aocc/



From cap at nsc.liu.se  Mon Dec 16 01:12:27 2019
From: cap at nsc.liu.se (Peter =?UTF-8?B?S2plbGxzdHLDtm0=?=)
Date: Mon, 16 Dec 2019 10:12:27 +0100
Subject: [Beowulf] Here we go again
In-Reply-To: <7905de37-3c9f-4a34-21db-8ff2afeed2a4@gmail.com>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
 <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
 <CAFkGP2mum3Kz5325-Cx7qzz4gph6UJDv_T8aYMHzWv8GXgq4_Q@mail.gmail.com>
 <20191212172938.39c92e1e@yaydoe>
 <7905de37-3c9f-4a34-21db-8ff2afeed2a4@gmail.com>
Message-ID: <20191213114654.23acb8b6@yaydoe>

On Fri, 13 Dec 2019 09:54:02 +0800
Miguel Dias Costa <migueldiascosta at gmail.com> wrote:
...
> I have been using MKL_DEBUG_CPU_TYPE=5 on Naples with the 2018
> versions of MKL with good results, but when switching to 2019
> versions some application tests started failing (dramatically wrong
> results) in FFT dependent stuff (and no failure without
> MKL_DEBUG_CPU_TYPE=5).
> 
> I could also reproduce this behaviour in some of MKL's own FFT
> examples, so it shouldn't be hard for other people to check.
> 
> Setting both MKL_DEBUG_CPU_TYPE=5 and MKL_CBWR=COMPATIBLE solved the 
> issue, and MKL is still using AVX2, but it does mean one needs to be 
> careful with MKL_DEBUG_CPU_TYPE...

We've had similar (but maybe not as serious) issues even on Intel
Skylake using MKL and AVX2 or AVX512. That is, using MKL_CBWR needed
for sanity.

Could you dig up which of the simple Intel tests that fail? I'd love to
try that on my Rome box (and skylake setup for reference).

/Peter
 

From migueldiascosta at gmail.com  Mon Dec 16 02:46:35 2019
From: migueldiascosta at gmail.com (Miguel Costa)
Date: Mon, 16 Dec 2019 18:46:35 +0800
Subject: [Beowulf] Here we go again
In-Reply-To: <20191213114654.23acb8b6@yaydoe>
References: <MWHPR2201MB16456DC807516E4F1D987FE083450@MWHPR2201MB1645.namprd22.prod.outlook.com>
 <47f157dd1c7a9d580f7c44bd412315a3.squirrel@mail.eadline.org>
 <C6AC8F9F-5D91-4B33-8433-8452F40208D8@jpl.caltech.edu>
 <e5e9eca54b876c69f9856c36b41d2e23.squirrel@mail.eadline.org>
 <dafbc4932d56fde7ce9a5743b6e7c669.squirrel@mail.eadline.org>
 <CAFkGP2mum3Kz5325-Cx7qzz4gph6UJDv_T8aYMHzWv8GXgq4_Q@mail.gmail.com>
 <20191212172938.39c92e1e@yaydoe>
 <7905de37-3c9f-4a34-21db-8ff2afeed2a4@gmail.com>
 <20191213114654.23acb8b6@yaydoe>
Message-ID: <CAPGRr+QnCs=_+urXuNeLGFppPnGtY2PJo0ejnftic1h9VAEt3w@mail.gmail.com>

On Mon, Dec 16, 2019 at 5:12 PM Peter Kjellström <cap at nsc.liu.se> wrote:

> ...
> We've had similar (but maybe not as serious) issues even on Intel
> Skylake using MKL and AVX2 or AVX512. That is, using MKL_CBWR needed
> for sanity.
>

I suppose it could be a more general regression (but on Haswell without
MKL_CBWR I didn't have any problem...)


> Could you dig up which of the simple Intel tests that fail? I'd love to
> try that on my Rome box (and skylake setup for reference).
>

For
instance, compilers_and_libraries_2019.1.144/linux/mkl/examples/fftw3xc,
example dp_plan_dft_3d

Cheers,
Miguel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20191216/85d132b3/attachment.html>