[Beowulf] OpenMP on AMD dual core processors

[Bill Broadley]

Greg Lindahl wrote:
> On Fri, Nov 21, 2008 at 09:38:29AM -0600, Nathan Moore wrote:
> 
>> I had a similar problem with gfortran, and it only appeared with large array
>> sizes (bigger than 4000x4000 as I recall).  "ulimit" was no help, I assume
>> there's a memory constraint built in somewhere.
> 
> With OpenMP the compiler has to set up multiple stacks, and some are
> more clever than others. If you were using the PathScale compiler, for
> example, and overran one of the thread stacks, it would print out an
> error message saying what happened and how to raise that limit.

I see warnings from Pathscale-3.2 when I run 2 8k x 8k arrays of doubles using
1 or 2 threads (but not 4 or 8 threads):
** OpenMP warning: requested pthread stack too large, using 4294967296 bytes
instead

But it still seems to work.

On my machine (with 8GB ram) I can run 2 8k x 8k arrays of doubles without
problem.  For runs with 8kx8k arrays, convergence_v = 1, 24 iterations and
gcc-4.3.2 -O3 -fopenmp:
real    3m39.818s
real    2m21.298s
real    2m19.850s
real    1m39.412s

Pathscale-3.2 -O3 -mp:
real    3m10.803s
real    2m24.492s
real    1m43.183s
real    1m20.669s

I believe that arrays in C/Fortran have to be contiguous by default and that
depending on kernel (32 vs 64bit), PAE, and BIOS settings sometimes all
physical memory isn't contiguous.