[Beowulf] [External] Re: traverse @ princeton

[Prentice Bisbal]

It's 4 racks of 10, and one rack of 6. For a total of 5 racks, not 
counting the storage system.

I believe  this is because of power/cooling limitations of the 
air-cooled systems. We have water-cooled rear-door heat exchangers, but 
they're only good up to about 35 KW/rack. Since we have 4 GPUs per 
server these things are consuming more power and putting out more heat 
than your average 1U pizza-box or blade server. Bill can answer more 
authoritatively, since he was involved in those discussions.

--
Prentice

On 10/10/19 12:57 PM, Scott Atchley wrote:
> That is better than 80% peak, nice.
>
> Is it three racks of 15 nodes? Or two racks of 18 and 9 in the third rack?
>
> You went with a single-port HCA per socket and not the shared, 
> dual-port HCA in the shared PCIe slot?
>
> On Thu, Oct 10, 2019 at 8:48 AM Bill Wichser <bill at princeton.edu 
> <mailto:bill at princeton.edu>> wrote:
>
>     Thanks for the kind words.  Yes, we installed more like a mini-Sierra
>     machine which is air cooled.  There are 46 nodes of the IBM AC922,
>     two
>     socket, 4 V100 where each socket uses the SMT threading x4. So two 16
>     core chips, 32/node, 128 threads per node.  The GPUs all use NVLink.
>
>     There are two EDR connections per host, each tied to a CPU, 1:1
>     per rack
>     of 12 and 2:1 between racks.  We have a 2P scratch filesystem running
>     GPFS.  Each node also has a 3T NVMe card as well for local scratch.
>
>     And we're running Slurm as our scheduler.
>
>     We'll see if it makes the top500 in November.  It fits there today
>     but
>     who knows what else got on there since June.  With the help of
>     nVidia we
>     managed to get 1.09PF across 45 nodes.
>
>     Bill
>
>     On 10/10/19 7:45 AM, Michael Di Domenico wrote:
>     > for those that may not have seen
>     >
>     >
>     https://insidehpc.com/2019/10/traverse-supercomputer-to-accelerate-fusion-research-at-princeton/
>     >
>     > Bill Wischer and Prentice Bisbal are frequent contributors to the
>     > list, Congrats on the acquisition.  Its nice to see more HPC
>     expansion
>     > in our otherwise barren hometown... :)
>     >
>     > Maybe one of them will pass along some detail on the machine...
>     > _______________________________________________
>     > Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     > To change your subscription (digest mode or unsubscribe) visit
>     https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>     >
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit
>     https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20191010/e443b9fe/attachment-0001.html>