[Beowulf] Intel Phi musings

[Richard Walsh]

Hey Stuart,

Thanks much for the detail.

So, if I am reading you correctly your test was on a single
physical PHI (you will later expand to multiple PHIs).  This
was a highly parallel single precision application which showed
the expected linear speed up to 60 cores ... then a kink as you
cross into hyper-threaded operation with a 1/2 as steep slope
up to factor of two to 120 core-equivalents with a 4 to 1 over
subscription of hyper-threads.  This was all done with the Intel
compilers on an unmodified pthreaded code that is well-vectored.

A good result ... on an application that is a perfect candidate
for PHI.  To run elsewhere with CUDA, OpenMP, or OpenACC
directives would require quite a bit of recoding which you were
happy to avoid.  My guess is if you had a CUDA implementation
you would see better performance on a FERMI or KEPLER,
but that is a programming path you do not wish to take.

This is an interesting case to hear about.  The flack (technical
marketing) from NVIDIA is to focus on the difficulty of using
the 'offload' model and Intel extensions to OpenMP, Cylk, etc.,
articulate their hardware's performance advantages, and talk
about OpenACC. These arguments are not unreasonable, but
clearly not universallydeciding.

Thanks much ... and good luck getting all your other codes
to scale just as well.

rbw

On Thu, Feb 14, 2013 at 10:18 AM, Dr Stuart Midgley <sdm900 at gmail.com>wrote:

> Evening
>
> Sorry for the slow response.
>
> Most of our codes are pthreads, we have avoided MPI and OpenMP as much as
> possible.  Our current cluster consists of Nehalem, Westmere, Sandy Bridge
> and Interlagos of various flavours.  Our Phi cards are in Sandy Bridge
> systems (host machine has 16 cores with 128GB ram).  We run the intel
> compilers.
>
> Our fastest systems are the 64core Interlagos systems (256GB ram) running
> at 2.6GHz.  For a few of our most important kernels, a single phi had
> greater throughput than a whole node.  Which, if you count the flops, is
> expected.  The Phi's have a massive amount of single precision floating
> point performance (our codes are single precision).
>
> Our kernels vectorise very well (lots of hand coded SSE3) and are expected
> for run very well on the phi (we haven't tested these codes yet).  The
> codes we have tested are trivially parallel and very FP heavy - they ported
> easily to the phi and run very well.
>
> The codes I tested (in like 2hrs) saw linear speedup to 60cores and then a
> "kink" in performance and then continued performance gains right up to 240
> threads.  Essentially these codes are single cpu with a trivial wrapper
> around them to hand out work.  This is exactly what hyper threading was
> designed to help.  So at 240 threads, we were about 120 times faster than a
> single thread of this code.  At 60 threads, we were 60 times faster :)
>
> Again, since the codes I tested were small data in, small data out and
> heavy compute and trivially parallel, running over multiple phi's is
> trivial and provide linear performance gains.  As we start porting more of
> our complex codes, I expect to see similar gains.  Our codes already run
> very very well on 64 cores…
>
> The phi's are separate cards, in separate pcie slots.  I have not delved
> into the programming api's fully, but I suspect you can utilise the one phi
> card for your threaded codes.  The way I've been running is with a native
> phi application (basically using the Phi as a separate linux cluster node)…
> using it in offload mode is very different and you may well be able to get
> your kernel running across both with the right pragmas.
>
> To be 100% honest, we took the boots and all approach.  If we only
> purchased 1 phi to test on, we would never expend the energy to port all
> our codes.  Purchasing hundreds of them gives you a lot of impetus to port
> your codes quickly :)
>
>
> --
> Dr Stuart Midgley
> sdm900 at sdm900.com
>
>
>
>
> On 13/02/2013, at 12:38 AM, Richard Walsh <rbwcnslt at gmail.com> wrote:
>
> >
> > Hey Stuart,
> >
> > Thanks for your answer ...
> >
> > That sounds compelling.  May I ask a few more questions?
> >
> > So should I assume that this was a threaded SMP type application
> > (OpenMP, pthreads) or it is MPI based? Is the supporting CPU of the
> > multi-core Sandy Bridge vintage? Have you been able to compare
> > the hyper-threaded, multi-core scaling on that Sandy Bridge side of the
> > system with that on the Phi (fewer cores to compare of course).  Using
> the
> > Intel compilers I assume ... how well do your kernels vectorize?  Curious
> > about the observed benefits of hyper-threading, which generally offers
> > little to floating-point intensive HPC computations where functional unit
> > collision is an issue.  You said you have 2 Phis per node.  Were you
> > running a single job across both?  Were the Phis in separate PCIE
> > slots or on the same card (sorry I should know this, but I have just
> > started looking at Phi).  If they are on separate cards in separate
> > slots can I assume that I am limited to MPI parallel implementations
> > when using both.
> >
> > Maybe that is more than a few questions ... ;-) ...
> >
> > Regards,
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20130214/00f5ec23/attachment.html>