[Beowulf] inspur gpu box?

[Michael Di Domenico]

On Thu, Apr 27, 2017 at 1:32 PM, Kilian Cavalotti
<kilian.cavalotti.work at gmail.com> wrote:
> Yup, you can have as many PCIe devices in a single root-complex with
> the appropriate number of PCIe switches.

Yes, understood.  It used to be that there was some odd issue with
most BIOS chips though, in that you couldn't actually have >8 video
cards in a system.  i don't recall the exact issue.

i know supermicro has 10, but that might just be a "hey it worked"
rather then the bios was specifically designed to handle a large
(meaning 16+) number of GPU's

> Will it create contentions? Certainly, because your CPU only have so
> many (40 typically) PCIe lanes. Single PCI root-complex is good for
> internal device-to-device communication (provided a sane PCIe
> switching architecture), but transferring data from the host memory to
> the PCIe devices (and vice-versa) will go through those 40-ish lanes,
> and that could quickly become a serious bottleneck.

agreed.  fortunately, some of the apps my users run, make almost zero
data transfers between the host memory and the gpu memory.  the just
want to spin the cores.  so even a heavily oversubscribed PCI-e
network wouldn't really harm us too much.  that being said it has to
at least work...  oversubb'ed is one thing, congestion death is
another