[Beowulf] traverse @ princeton

[Scott Atchley]

That is better than 80% peak, nice.

Is it three racks of 15 nodes? Or two racks of 18 and 9 in the third rack?

You went with a single-port HCA per socket and not the shared, dual-port
HCA in the shared PCIe slot?

On Thu, Oct 10, 2019 at 8:48 AM Bill Wichser <bill at princeton.edu> wrote:

> Thanks for the kind words.  Yes, we installed more like a mini-Sierra
> machine which is air cooled.  There are 46 nodes of the IBM AC922, two
> socket, 4 V100 where each socket uses the SMT threading x4.  So two 16
> core chips, 32/node, 128 threads per node.  The GPUs all use NVLink.
>
> There are two EDR connections per host, each tied to a CPU, 1:1 per rack
> of 12 and 2:1 between racks.  We have a 2P scratch filesystem running
> GPFS.  Each node also has a 3T NVMe card as well for local scratch.
>
> And we're running Slurm as our scheduler.
>
> We'll see if it makes the top500 in November.  It fits there today but
> who knows what else got on there since June.  With the help of nVidia we
> managed to get 1.09PF across 45 nodes.
>
> Bill
>
> On 10/10/19 7:45 AM, Michael Di Domenico wrote:
> > for those that may not have seen
> >
> >
> https://insidehpc.com/2019/10/traverse-supercomputer-to-accelerate-fusion-research-at-princeton/
> >
> > Bill Wischer and Prentice Bisbal are frequent contributors to the
> > list, Congrats on the acquisition.  Its nice to see more HPC expansion
> > in our otherwise barren hometown... :)
> >
> > Maybe one of them will pass along some detail on the machine...
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
> >
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20191010/c6a877bc/attachment.html>