[Beowulf] Dunking for Density: New Projects Pursue 3M’s Take on Immersion Cooling

[Eugen Leitl]

(go visit the site for pretty pikchers)

http://www.datacenterknowledge.com/archives/2013/11/13/3m-immersion-cooling/

Home Cooling » Low Latency Dunking for Density: New Projects Pursue 3M’s Take
on Immersion Cooling

Dunking for Density: New Projects Pursue 3M’s Take on Immersion Cooling

November 13th, 2013By: Rich Miller

This immersion cooling project in Hong Kong was created by Allied Control
using a two-phase cooling technique called open bath immersion (OBI), using
3M’s Novec fluid.

We’re continuing to see new examples of immersion cooling at meaningful
scale. In July we brought you an update on an immersion cooling system at CGG
using technology from Green Revolution Cooling. We’ve also been tracking
early projects using “open bath immersion” cooling based on technology
developed by 3M.

Open bath immersion (OBI) is an example of passive two-phase cooling, which
uses a boiling liquid to remove heat from a surface and then condenses the
liquid for reuse, all without a pump. The servers are immersed in 3M’s Novec,
a non-conductive chemical with a very low boiling point, which easily
condenses from gas back to liquid. The OBI technique, which we first saw at
last year’s Data Center World show, is now in use in a handful of sites.
Here’s an overview of some of these projects.

Allied Control

Hong Kong-based Allied Control is specializing in developing high-density
cooling solution using 3M’s Novec and OBI. The company has recently deployed
a 500kW high performance computing (HPC) production installation known as
Immersion-2 for a client in Hong Kong. This design uses OBI in standard
19-inch racks, and was deployed in less than six months. Allied Control says
the system operates at a Power Usage Effectiveness (PUE) of 1.02, which would
make it one of the most efficient designs in the world, even though Hong Kong
has a hot and humid climate. The facility is located in a high rise building
and fits in the size of a standard shipping container.

One issue with a rack-mounted approach to immersion cooling is weight. “The
weight is indeed a small challenge for standard sized racks, but actually
more due to the increased system density, not really the fluid,” said Alex
Kampl, VP of Engineering for Allied Control. “You also remove a lot of weight
by not using air cooling. We’ve been working with a rack manufacturer who has
been very helpful.” Here’s a look at the facility:

These racks are filled with high-density servers immersed in cooling fluid.
The installation in Hong Kong was created by Applied Control using 3M's Novec
fluid. (Photo: Applied Control)  These racks are filled with tanks containing
high-density servers immersed in cooling fluid. The installation in Hong Kong
was created by Allied Control using 3M’s Novec fluid. (Photo: Allied Control)

Inside the rack-mounted tanks, the heat from dozens of servers causes the
Novec fluid to boil. The vapor cools when it reaches the condenser at the top
of the tank and is then reused. (Photo: Applied Control) Inside the
rack-mounted tanks, the heat from dozens of servers causes the Novec fluid to
boil. The vapor cools when it reaches the condenser at the top of the tank
and is then reused. (Photo: Allied Control)

The company previously built a dedicated immersion-cooled facility called
Immersion-1 to cool a unique supercomputer comprised of FPGAs (field
programmable gate arrays), which are semiconductor devices that can be
programmed after manufacturing.

Allied Control created Immersion-1, a system using 6,048 FPGA chips combining
890 million logic cells, which will encompass up to 24 tanks. The company
says that a similar installation using traditional air cooling would require
more than 8,500 Dual Xeon 1U servers in more than 200 racks.

“Immersion-1 has become a massive prototype and proof of concept for a whole
new generation of computing,” Applied Control says on its web site. “Since
the special application tweaks the maximum performance out of each FPGA, they
generate much more heat than in traditional FPGA applications. Often, FPGAs
have to be throttled down or are not running at maximum performance due to
cooling issues. In case of Immersion-1, the cluster would not be able to run
on passive cooling and the FPGA chip temperature rises  above its maximum
specifications within seconds. Only by using immersion cooling it was
possible to build and run Immersion-1 with its very demanding cooling
requirements.”

The company has also developed a design concept to adapt Intel’s Xeon Phi
coprocessor for HPC workloads in immersion cooling, and is  interested in
developing high-density designs using Intel’s Dense Form Factor (DFF) cards
in open bath immersion.

“OBI is in its early stages, but I am sure we’ll see exciting progress very
soon,” said Kampl. “Unfortunately we are wasting a lot of time right now to
literally remove unnecessary parts from hardware built for air-cooling, so my
hope is that system designers start offering similar hardware like the DFF
cards.”

The Allied Control technology will be on display at the 3M booth at next
week’s SC13 conference in Denver.

3M

The 3M team that developed open bath immersion is working on a supercomputer
as a proof of concept. ”We are building a small supercomputer based on state
of the art hardware that is normally water cooled,” the 3M team reports on
its Facebook page.

The half-rack (21U) system can accommodate up to 144 Intel Sandy Bridge CPUs.
3M has built a custom tank to hold the servers and Novec fluid, as well as
custom cooling tower to support the system:

3m-cooling-unit Hand-Built Dry Tower that will Cool the 3M Supercomputer
System. (Photo: 3m)

3M commenced building its own supercomputer to provide a larger showcase for
the potential of Novec and the OBI concept, hoping to jump-start interest
among both users and equipment vendors.

“It has been challenging for us as the fluid provider to move this forward,”
said Phil Tuma, an Application Specialist at 3M Electronic Markets who has
championed OBI cooling. “When we did find an end user interested in the
technology, we couldn’t point them to a system vendor, only a pail of fluid.
With one exception, efforts with the big OEMs were largely a waste of time.
They all move slowly. We decided early this year to approach smaller, more
agile HPC companies interested in differentiating themselves.”

Tuma said 3M has been working with HPC server vendor Cirrascale and modular
data enclosure specialist Elliptical Mobile Solutions on developing open bath
immersion deployments.

Other Installations

Lawrence Berkeley National Laboratory is also developing an OBI/Novec
project, working with technical computing hardware vendor SGI to create an
immersion cooling system for the Naval Research Laboratory (NRL).  The
initiative is being funded by the Environmental Security Technology
Certification Program (ESTCP), the Department of Defense’s environmental
technology demonstration and validation program. ESTCP works to help
innovative technologies to overcome barriers to deployment. The LBNL/NRL
project is expected to house a half-rack of SGI IceX hardware with 144
Sandybridge sockets. Under Turbo 2.0 conditions, it will draw over 40kW. 

The Mayo Clinic is also test-driving a Novec-based OBI solution. The clinic’s
Special Purpose Processor Development Group (SPPDG) has built an 80kW tank to
serve as a test platform for developing OBI technology.