a tale of two supercomputers

[Eugen Leitl]

http://www.nytimes.com/2002/06/25/science/physical/25COMP.html?pagewanted=print&position=bottom

June 25, 2002
At Los Alamos, Two Visions of Supercomputing
By GEORGE JOHNSON

Moore's Law holds that the number of transistors on a microprocessor — the 
brain of a modern computer — doubles about every 18 months, causing the 
speed of its calculations to soar. But there is a downside to this 
oft-repeated tale of technological progress: the heat produced by the chip 
also increases exponentially, threatening a self-inflicted meltdown.

A computer owner in Britain recently dramatized the effect by propping a 
makeshift dish of aluminum foil above the chip inside his PC and frying an 
egg for breakfast. (The feat — cooking time 11 minutes — was reported in 
The Register, a British computer industry publication.) By 2010, 
scientists predict, a single chip may hold more than a billion 
transistors, shedding 1,000 watts of thermal energy — far more heat per 
square inch than a nuclear reactor.

The comparison seems particularly apt at Los Alamos National Laboratory in 
northern New Mexico, which has two powerful new computers, Q and Green 
Destiny. Both achieve high calculating speeds by yoking together webs of 
commercially available processors. But while the energy-voracious Q was 
designed to be as fast as possible, Green Destiny was built for 
efficiency. Side by side, they exemplify two very different visions of the 
future of supercomputing.

Los Alamos showed off the machines last month at a ceremony introducing 
the laboratory's Nicholas C. Metropolis Center for Modeling and 
Simulation. Named for a pioneering mathematician in the Manhattan Project, 
the three-story, 303,000-square-foot structure was built to house Q, which 
will be one of the world's two largest computers (the other is in Japan). 
Visitors approaching the imposing structure might mistake it for a power 
generating plant, its row of cooling towers spewing the heat of 
computation into the sky.

Supercomputing is an energy-intensive process, and Q (the name is meant to 
evoke both the dimension-hopping Star Trek alien and the gadget-making 
wizard in the James Bond thrillers) is rated at 30 teraops, meaning that 
it can perform as many as 30 trillion calculations a second. (The measure 
of choice used to be the teraflop, for "trillion floating-point 
operations," but no one wants to think of a supercomputer as flopping 
trillions of times a second.)

Armed with all this computing power, Q's keepers plan to take on what for 
the Energy Department, anyway, is the Holy Grail of supercomputing: a 
full-scale, three-dimensional simulation of the physics involved in a 
nuclear explosion.

"Obviously with the various treaties and rules and regulations, we can't 
set one of these off anymore," said Chris Kemper, deputy leader of the 
laboratory's computing, communications and networking division. "In the 
past we could test in Nevada and see if theory matched reality. Now we 
have do to it with simulations."

While decidedly more benign than a real explosion, Q's artificial blasts — 
described as testing "in silico" — have their own environmental impact. 
When fully up and running later this year, the computer, which will occupy 
half an acre of floor space, will draw three megawatts of electricity. Two 
more megawatts will be consumed by its cooling system. Together, that is 
enough to provide energy for 5,000 homes.

And that is just the beginning. Next in line for Los Alamos is a 
100-teraops machine. To satisfy its needs, the Metropolis center can be 
upgraded to provide as much as 30 megawatts — enough to power a small 
city.

That is where Green Destiny comes in. While Q was attracting most of the 
attention, researchers from a project called Supercomputing in Small 
Spaces gathered nearby in a cramped, stuffy warehouse to show off their 
own machine — a compact, energy-efficient computer whose processors do not 
even require a cooling fan.

With a name that sounds like an air freshener or an environmental group 
(actually it's taken from the mighty sword in "Crouching Tiger, Hidden 
Dragon"), Green Destiny measures about two by three feet and stands six 
and a half feet high, the size of a refrigerator.

Capable of a mere 160 gigaops (billions of operations a second), the 
machine is no match for Q. But in computational bang for the buck, Green 
Destiny wins hands down. Though Q will be almost 200 times as fast, it 
will cost 640 times as much — $215 million, compared with $335,000 for 
Green Destiny. And that does not count housing expenses — the $93 million 
Metropolis center that provides the temperature-controlled, dust-free 
environment Q demands.

Green Destiny is not so picky. It hums away contentedly next to piles of 
cardboard boxes and computer parts. More important, while Q and its 
cooling system will consume five megawatts of electrical power, Green 
Destiny draws just a thousandth of that — five kilowatts. Even if it were 
expanded, as it theoretically could be, to make a 30-teraops machine 
(picture a hotel meeting room crammed full of refrigerators), it would 
still draw only about a megawatt.

"Bigger and faster machines simply aren't good enough anymore," said Dr. 
Wu-Chung Feng, the leader of the project. The time has come, he said, to 
question the doctrine of "performance at any cost."

The issue is not just ecological. The more power a computer consumes, the 
hotter it gets. Raise the operating temperature 18 degrees Fahrenheit, Dr. 
Feng said, and the reliability is cut in half. Pushing the extremes of 
calculational speed, Q is expected to run in sprints for just a few hours 
before it requires rebooting. A smaller version of Green Destiny, called 
Metablade, has been operating in the warehouse since last fall, requiring 
no special attention.

"There are two paths now for supercomputing," Dr. Feng said. "While 
technically feasible, following Moore's Law may be the wrong way to go 
with respect to reliability, efficiency of power use and efficiency of 
space. We're not saying this is a replacement for a machine like Q but 
that we need to look in this direction."

The heat problem is nothing new. In taking computation to the limit, 
scientists constantly consider the trade-off between speed and efficiency. 
I.B.M.'s Blue Gene project, for example, is working on energy-efficient 
supercomputers to run simulations in molecular biology and other sciences.

"All of us who are in this game are busy learning how to run these big 
machines," said Dr. Mike Levine, a scientific director at the Pittsburgh 
Supercomputing Center and a physics professor at Carnegie Mellon 
University. A project like Green Destiny is "a good way to get people's 
attention," he said, "but it is only the first step in solving the 
problem."

Green Destiny belongs to a class of makeshift supercomputers called 
Beowulf clusters. Named for the monster-slaying hero in the eighth-century 
Old English epic, the machines are made by stringing together 
off-the-shelf PC's into networks, generally communicating via Ethernet — 
the same technology used in home and office networking. What results is 
supercomputing for the masses — or, in any case, for those whose operating 
budgets are in the range of tens or hundreds of thousands of dollars 
rather than the hundreds of millions required for Q.

Dr. Feng's team, which also includes Dr. Michael S. Warren and Eric H. 
Weigle, began with a similar approach. But while traditional Beowulfs are 
built from Pentium chips and other ordinary processors, Green Destiny uses 
a special low-power variety intended for laptop computers.

A chip's computing power is ordinarily derived from complex circuits 
packed with millions of invisibly tiny transistors. The simpler Transmeta 
chips eliminate much of this energy-demanding hardware by performing 
important functions using software instead — instructions coded in the 
chip's memory. Each chip is mounted along with other components on a small 
chassis, called a blade. Stack the blades into a tower and you have a 
Bladed Beowulf, in which the focus is on efficiency rather than raw 
unadulterated power.

The method has its limitations. A computer's power depends not just on the 
speed of its processors but on how fast they can cooperate with one 
another. Linked by high-speed fiber-optical cable, Q's many subsections, 
or nodes, exchange data at a rate as high as 6.3 gigabits a second. Green 
Destiny's nodes are limited to 100-megabit Ethernet.

The tightly knit communication used by Q is crucial for the intense 
computations involved in modeling nuclear tests. A weapons simulation 
recently run on the Accelerated Strategic Computing Initiative's ASCI 
White supercomputer at Lawrence Livermore National Laboratory in 
California took four months of continuous calculating time — the 
equivalent of operating a high-end personal computer 24 hours a day for 
more than 750 years.

Dr. Feng has looked into upgrading Green Destiny to gigabit Ethernet, 
which seems destined to become the marketplace standard. But with current 
technology that would require more energy consumption, erasing the 
machine's primary advantage.

For now, a more direct competitor may be the traditional Beowulfs with 
their clusters of higher-powered chips. Though they are cheaper and 
faster, they consume more energy, take up more space, and are more prone 
to failure. In the long run, Dr. Feng suggests, an efficient machine like 
Green Destiny might actually perform longer chains of sustained 
calculations.

At some point, in any case, the current style of supercomputing is bound 
to falter, succumbing to its own heat. Then, Dr. Feng hopes, something 
like the Bladed Beowulfs may serve as "the foundation for the 
supercomputer of 2010."

Meanwhile, the computational arms race shows no signs of slowing down. 
Half of the computing floor at the Metropolis Center has been left empty 
for expansion. And ground was broken this spring at Lawrence Livermore for 
a new Terascale Simulation Facility. It is designed to hold two 
100-teraops machines.