[Beowulf] HPC with CUDA

[Adam DeConinck]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Fri, Jun 13, 2014 at 10:34:05PM +0700, "C. Bergström" wrote:
> On 06/13/14 09:16 PM, Greg Keller wrote:
> >Supermicro has a board that provides Eight x16 slots, but I understand
> >it's wired so that 2 slots are effectively sharing 16 lanes.  Let me know
> >if you find anything more awesome...
> Newer Intel CPU can support 1x 32 lanes or 2x 32 lanes? At some point you'll
> hit a bottleneck where you can't actually use all the lanes. In some of the
> supercomputer designs they are doing a 1GPU to 1CPU ratio. I think getting
> the workload balanced across this design would be challenging and that's
> hoping all the driver issues have been ironed out.

An IVB Xeon CPU like the E5-2690v2 has 40 PCIe lanes, which I've typically seen
divided up into two x16 links and one x8 link. So if you want full
bandwidth between the CPU and each GPU, you're limited to two GPUs per
socket. Most motherboards which provide more than two x16 links per
socket use a PCIe switch to "split" a single x16 link. These switches
can generally provide full bandwidth between any pair of devices on the
switch, but they all share a single "uplink" to the CPU.

Choosing the right GPU:CPU ration depends a lot on what applications you
expect to be running. If your app performance depends heavily on
transfers between GPU and host memory, you probably don't want more than
2 GPUs per CPU socket. But if your app makes heavy use of GPU-to-GPU
peer-to-peer transfers, you might care more about getting as many GPUs
on a socket as you can.

(Very small self-plug: I gave a talk on managing GPU-enabled HPC
clusters at the GPU Technology Conference in March, where I talked a bit
about managing compute node topology as well as other tools. Slides are
at
http://on-demand.gputechconf.com/gtc/2014/presentations/S4253-tools-tips-for-managing-a-gpu-cluster.pdf)

Cheers,
Adam


- -- 
Adam DeConinck
Email: ajdecon at ajdecon.org | Twitter: @ajdecon
Web: https://www.ajdecon.org/
PGP key: https://www.ajdecon.org/ajdecon-key.txt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Darwin)

iQEcBAEBCgAGBQJTm0a9AAoJEGqxms5cZfz0GQUIAKjVMaW+OP8h3EnNBNDHNJjN
EqiruOfbMX27js0WF5s83qSDEHJvkX7ADP9kguo4vQ4IsOZ49e+fH5831rMrRdT7
1keYYPgr4LSOvUF6c2+lc5OrV7dzlOy6hCrQNwu1Yg20BgcAoAeUCL0kjDRHOzJL
0KLGI8egxVTh/tGZJ+SQnalHG4qeHXZgvnEjpoMivMJl5r6DwuE+rG6KFq4bCi5f
T428EkUWitguTeOTZVVrmModKXpqv7dTscD99bvbgqHJhuTUui25G1LImqdOMgwy
IU0AQhAzMSJifD/Nmkw05WnqAV9FvtMR6kzv6IBgkg37fPfQe3m4wa/6w2AXd+Y=
=u19A
-----END PGP SIGNATURE-----