[Beowulf] Three-Mile-High Supercomputer Poses Unique Challenges

[Eugen Leitl]

http://slashdot.org/topic/datacenter/three-mile-high-supercomputer-poses-unique-challenges/


Three-Mile-High Supercomputer Poses Unique Challenges

by Mark Hachman | January 4, 2013

How do you install, build and operate a supercomputer at 16,000 feet? Slowly.

The correlator in the ALMA Array Operations Site Technical Building.

Building and operating a supercomputer at more than three miles above sea
level poses some unique problems, the designers of the recently installed
Atacama Large Millimeter/submillimeter Array (ALMA) Correlator discovered.

Why build a supercomputer at 16,000 feet? Because the ALMA computer serves as
the brains behind the ALMA astronomical telescope, a partnership between
Europe, North American, and South American agencies. It’s the largest such
project in existence. Based high in the Andes mountains in northern Chile,
the telescope includes an array of 66 dish-shaped antennas in two groups. The
telescope correlator’s 134 million processors continually combine and compare
faint celestial signals received by the antennas in the ALMA array, which are
separated by up to 16 kilometers, enabling the antennas to work together as a
single, enormous telescope, according to Space Daily.

Funded by the US National Science Foundation (NSF), and designed,
constructed, and installed primarily by the National Radio Astronomy
Observatory (NRAO), the back-end correlator is tuned for signal processing.
The four “quadrants” of the correlator can each process data coming from 504
antenna pairs. ALMA’s Website reports that signals are processed via tunable
filter bank cards, four of which are needed per antenna. Each tuner card can
“slice and dice” the perceived spectrum that the antennas “see,” allowing
them to make specialized observations.

(Fifty radio antennas make up the main ALMA array; an additional array of 16
antennas, called the Atacama Compact Array (ACA), is provided by the National
Astronomical Observatory of Japan (NAOJ) and has its own Fujitsu-designed
correlator; there’s a 2010 paper describing that design.)

All told, the supercomputer includes 134 million processors and performs up
to 17 quadrillion operations per second, according to Space Daily. That would
put it close to the top of the semiannual TOP500 supercomputer list, which
ranks the world’s most powerful supercomputers; since ALMA Correlator is a
special-purpose machine, however, it won’t qualify.  High-Altitude Challenges

The extreme high altitude makes it nearly impossible to maintain on-site
support staff for significant lengths of time, with ALMA reporting that human
intervention will be kept to an absolute minimum. Data acquired via the array
is archived at a lower-altitude support site.

But the altitude also poses other challenges. For one thing, the actuators
that glide above the surface of a hard disk no longer operate properly,
requiring the use of solid-state disks. The thin air also poses a cooling
problem, requiring twice the normal airflow to sufficiently cool the machines
(which draw some 140 kilowatts of power). Seismic activity is also common in
the area, so the correlator had to be designed to withstand the vibrations
associated with earthquakes.

The altitude also limited the construction crew’s ability to actually build
the thing, requiring 20 weeks of human effort just to unpack and install it.
“There are thousands upon thousands of cable connections that we had to make,
and every one of our cables is the same color blue, so I’m just glad we
devised a good labeling system while at sea level,” NRAO’s Rich Lacasse,
leader of the ALMA Correlator Team, wrote in a statement released by the
agency.

What lessons can data-center designers learn from ALMA? Probably not too
many, given the specialized nature of the machine. But if nothing else,
building a high-altitude facility to take advantage of lower ambient
temperatures may be more trouble than it’s worth. The facility will be
formally inaugurated in March.

http://www.spacedaily.com/reports/All_Systems_Go_for_Highest_Altitude_Supercomputer_999.html

All Systems Go for Highest Altitude Supercomputer

by Staff Writers

Munich, Germany (SPX) Dec 27, 2012

The ALMA correlator, one of the most powerful supercomputers in the world,
has now been fully installed and tested at its remote, high altitude site in
the Andes of northern Chile. This wide-angle view shows some of the racks of
the correlator in the ALMA Array Operations Site Technical Building. This
photograph shows one of four quadrants of the correlator. The full system has
four identical quadrants, with over 134 million processors, performing up to
17 quadrillion operations per second. Credit: ESO.

One of the most powerful supercomputers in the world has now been fully
installed and tested at its remote, high altitude site in the Andes of
northern Chile. This marks one of the major remaining milestones toward
completion of the Atacama Large Millimeter/submillimeter Array (ALMA), the
most elaborate ground-based telescope in history.

The special-purpose ALMA correlator has over 134 million processors and
performs up to 17 quadrillion operations per second, a speed comparable to
the fastest general-purpose supercomputer in operation today.

The correlator is a critical component of ALMA, an astronomical telescope
which is composed of an array of 66 dish-shaped antennas.

The correlator's 134 million processors continually combine and compare faint
celestial signals received by the antennas in the ALMA array, which are
separated by up to 16 kilometres, enabling the antennas to work together as a
single, enormous telescope. The information collected by each antenna must be
combined with that from every other antenna. At the correlator's maximum
capacity of 64 antennas [1] as many as 17 quadrillion calculations every
second must be performed [2].

The correlator was built specifically for this task, but the number of
calculations per second is comparable to the performance of the fastest
general-purpose supercomputers in the world [3].

"This unique computing challenge needed innovative design, both for the
individual components and the overall architecture of the correlator," says
Wolfgang Wild, the European ALMA Project Manager, from ESO.

The initial design of the correlator, as well as its construction and
installation, was led by the US National Radio Astronomy Observatory (NRAO),
the lead North American partner in ALMA. The correlator project was funded by
the US National Science Foundation, with contributions from ESO.

"The completion and installation of the correlator is a huge milestone
towards the fulfillment of North America's share of the international ALMA
construction project," said Mark McKinnon, North American ALMA Project
Director at NRAO. "The technical challenges were enormous, and our team
pulled it off," he added.

As the European partner in ALMA, ESO also provided a key part of the
correlator: an entirely new and versatile digital filtering system conceived
in Europe was incorporated into the initial NRAO design.

A set of 550 state-of-the-art digital filter circuit boards was designed and
built for ESO by the University of Bordeaux in France [4]. With these
filters, the wavelengths of light which ALMA sees can be split up 32 times
more finely than in the initial design, into ranges that can be finely tuned.

"This vastly improved flexibility is fantastic; it lets us 'slice and dice'
the spectrum of light that ALMA sees, so we can concentrate on the precise
wavelengths needed for a given observation, whether it's mapping the gas
molecules in a star-forming cloud, or searching for some of the most distant
galaxies in the Universe," said Alain Baudry, from the University of
Bordeaux, the European ALMA correlator team leader.

Another challenge was the extreme location. The correlator is housed in the
ALMA Array Operations Site (AOS) Technical Building, the highest altitude
high-tech building in the world. At 5000 metres, the air is thin, so twice
the normal airflow is necessary to cool the machine, which draws some 140
kilowatts of power.

In this thin air, spinning computer disk drives cannot be used, as their
read/write heads rely on a cushion of air to stop them crashing into their
platters. Seismic activity is common, so the correlator had to be designed to
withstand the vibrations associated with earthquakes.

ALMA began science observations in 2011 with a partial array of antennas. A
section of the correlator was already being used to combine the signals from
the partial array, but now the full system is complete. The correlator is
ready for ALMA to begin operating with a larger number of antennas, which
will increase the sensitivity and image quality of the observations. ALMA is
nearing completion and will be inaugurated in March 2013.

Notes

[1] The ALMA correlator is one of two such systems in the ALMA complex.
ALMA's total of 66 antennas comprise a main array of 50 antennas (half
provided by ESO, and half by NRAO) and an additional, complementary array of
16 antennas called the Atacama Compact Array (ACA), which is provided by the
National Astronomical Observatory of Japan (NAOJ).

A second correlator, built by the Fujitsu company and delivered by NAOJ,
provides independent correlation of the 16 antennas in the ACA, except for
times when select ACA antennas are combined with the 50 more widely dispersed
main array antennas.

[2] 17 quadrillion = 17 000 000 000 000 000.

[3] The current record holder in the TOP500 list of general-purpose
supercomputers is the Titan, from Cray Inc., which has been measured at 17.59
quadrillion floating point operations per second. Note that the ALMA
correlator is a special-purpose supercomputer and is not eligible for this
ranking.

[4] This work followed work on new concepts for the correlator, done by the
University of Bordeaux in a consortium also involving ASTRON in the
Netherlands, and the INAF-Osservatorio di Arcetri in Italy.