[Beowulf] traverse @ princeton

[Scott Atchley]

Okay, that is the same slot Summit/Sierra use for the EDR HCA. You may want
to check out our paper at SC19 where we look at several new features in EDR
as well as how to best stripe data over the four virtual ports.

On Thu, Oct 10, 2019 at 1:49 PM Bill Wichser <bill at princeton.edu> wrote:

> Actually 12 per rack.  The reasoning was that there were 2 connections
> per host to top of rack switch leaving 12 uplinkks to two tier0 switches
> at 6 each.
>
> For the IB cards they are some special flavored Mellanox which attach to
> the PCIv4 sockets, 8 lanes each.  And since 8 lanes of v4 == 16 lanes of
> v3, we get full EDR to both CPU sockets.
>
> Bill
>
> On 10/10/19 12:57 PM, Scott Atchley wrote:
> > That is better than 80% peak, nice.
> >
> > Is it three racks of 15 nodes? Or two racks of 18 and 9 in the third
> rack?
> >
> > You went with a single-port HCA per socket and not the shared, dual-port
> > HCA in the shared PCIe slot?
> >
> > On Thu, Oct 10, 2019 at 8:48 AM Bill Wichser <bill at princeton.edu
> > <mailto:bill at princeton.edu>> wrote:
> >
> >     Thanks for the kind words.  Yes, we installed more like a mini-Sierra
> >     machine which is air cooled.  There are 46 nodes of the IBM AC922,
> two
> >     socket, 4 V100 where each socket uses the SMT threading x4.  So two
> 16
> >     core chips, 32/node, 128 threads per node.  The GPUs all use NVLink.
> >
> >     There are two EDR connections per host, each tied to a CPU, 1:1 per
> >     rack
> >     of 12 and 2:1 between racks.  We have a 2P scratch filesystem running
> >     GPFS.  Each node also has a 3T NVMe card as well for local scratch.
> >
> >     And we're running Slurm as our scheduler.
> >
> >     We'll see if it makes the top500 in November.  It fits there today
> but
> >     who knows what else got on there since June.  With the help of
> >     nVidia we
> >     managed to get 1.09PF across 45 nodes.
> >
> >     Bill
> >
> >     On 10/10/19 7:45 AM, Michael Di Domenico wrote:
> >      > for those that may not have seen
> >      >
> >      >
> >
> https://insidehpc.com/2019/10/traverse-supercomputer-to-accelerate-fusion-research-at-princeton/
> >      >
> >      > Bill Wischer and Prentice Bisbal are frequent contributors to the
> >      > list, Congrats on the acquisition.  Its nice to see more HPC
> >     expansion
> >      > in our otherwise barren hometown... :)
> >      >
> >      > Maybe one of them will pass along some detail on the machine...
> >      > _______________________________________________
> >      > Beowulf mailing list, Beowulf at beowulf.org
> >     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
> >      > To change your subscription (digest mode or unsubscribe) visit
> >     https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
> >      >
> >     _______________________________________________
> >     Beowulf mailing list, Beowulf at beowulf.org
> >     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
> >     To change your subscription (digest mode or unsubscribe) visit
> >     https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20191011/9d0f1489/attachment.html>