[Beowulf] What It'll Take to Go Exascale

[Eugen Leitl]

http://www.sciencemag.org/content/335/6067/394.full 

Science 27 January 2012:

Vol. 335 no. 6067 pp. 394-396

DOI: 10.1126/science.335.6067.394

Computer Science

What It'll Take to Go Exascale

Robert F. Service

Scientists hope the next generation of supercomputers will carry out a
million trillion operations per second. But first they must change the way
the machines are built and run.

On fire.

More powerful supercomputers now in the design stage should make modeling
turbulent gas flames more accurate and revolutionize engine designs.

"CREDIT: J. CHEN/CENTER FOR EXASCALE SIMULATION OF COMBUSTION IN TURBULENCE,
SANDIA NATIONAL LABORATORIES"

Using real climate data, scientists at Lawrence Berkeley National Laboratory
(LBNL) in California recently ran a simulation on one of the world's most
powerful supercomputers that replicated the number of tropical storms and
hurricanes that had occurred over the past 30 years. Its accuracy was a
landmark for computer modeling of global climate. But Michael Wehner and his
LBNL colleagues have their eyes on a much bigger prize: understanding whether
an increase in cloud cover from rising temperatures would retard climate
change by reflecting more light back into space, or accelerate it by trapping
additional heat close to Earth.

To succeed, Wehner must be able to model individual cloud systems on a global
scale. To do that, he will need supercomputers more powerful than any yet
designed. These so-called exascale computers would be capable of carrying out
1018 floating point operations per second, or an exaflop. That's nearly 100
times more powerful than today's biggest supercomputer, Japan's “K Computer,”
which achieves 11.3 petaflops (1015 flops) (see graph), and 1000 times faster
than the Hopper supercomputer used by Wehner and his colleagues. The United
States now appears poised to reach for the exascale, as do China, Japan,
Russia, India, and the European Union.

It won't be easy. Advances in supercomputers have come at a steady pace over
the past 20 years, enabled by the continual improvement in computer chip
manufacturing. But this evolutionary approach won't cut it in getting to the
exascale. Instead, computer scientists must first figure out ways to make
future machines far more energy efficient and tolerant of errors, and find
novel ways to program them.

“The step we are about to take to exascale computing will be very, very
difficult,” says Robert Rosner, a physicist at the University of Chicago in
Illinois, who chaired a recent Department of Energy (DOE) committee charged
with exploring whether exascale computers would be achievable. Charles Shank,
a former director of LBNL who recently headed a separate panel collecting
widespread views on what it would take to build an exascale machine, agrees.
“Nobody said it would be impossible,” Shank says. “But there are significant
unknowns.”

Gaining support

The next generation of powerful supercomputers will be used to design
high-efficiency engines tailored to burn biofuels, reveal the causes of
supernova explosions, track the atomic workings of catalysts in real time,
and study how persistent radiation damage might affect the metal casing
surrounding nuclear weapons. “It's a technology that has become critically
important for many scientific disciplines,” says Horst Simon, LBNL's deputy
director.

That versatility has made supercomputing an easy sell to politicians. The
massive 2012 spending bill approved last month by Congress contained $1.06
billion for DOE's program in advanced computing, which includes a down
payment to bring online the world's first exascale computer. Congress didn't
specify exactly how much money should be spent on the exascale initiative,
for which DOE had requested $126 million. But it asked for a detailed plan,
due next month, with multiyear budget breakdowns listing who is expected to
do what, when. Those familiar with the ways of Washington say that the
request reflects an unusual bipartisan consensus on the importance of the
initiative.

“In today's political atmosphere, this is very unusual,” says Jack Dongarra,
a computer scientist at the University of Tennessee, Knoxville, who closely
follows national and international high-performance computing trends. “It
shows how critical it really is and the threat perceived of the U.S. losing
its dominance in the field.” The threat is real: Japan and China have built
and operate the three most powerful supercomputers in the world.

The rest of the world also hopes that their efforts will make them less
dependent on U.S. technology. Of today's top 500 supercomputers, the vast
majority were built using processors from Intel, Advanced Micro Devices
(AMD), and NVIDIA, all U.S.-based companies. But that's beginning to change,
at least at the top. Japan's K machine is built using specially designed
processors from Fujitsu, a Japanese company. China, which had no
supercomputers in the Top500 List in 2000, now has five petascale machines
and is building another with processors made by a Chinese company. And an
E.U. research effort plans to use ARM processing chips made by a U.K.
company.

Getting over the bumps

Although bigger and faster, supercomputers aren't fundamentally different
from our desktops and laptops, all of which rely on the same sorts of
specialized components. Computer processors serve as the brains that carry
out logical functions, such as adding two numbers together or sending a bit
of data to a location where it is needed. Memory chips, by contrast, hold
data for safekeeping for later use. A network of wires connects processors
and memory and allows data to flow where and when they are needed.

For decades, the primary way of improving computers was creating chips with
ever smaller and faster circuitry. This increased the processor's frequency,
allowing it to churn through tasks at a faster clip. Through the 1990s,
chipmakers steadily boosted the frequency of chips. But the improvements came
at a price: The power demanded by a processor is proportional to its
frequency cubed. So doubling a processor's frequency requires an eightfold
increase in power.

New king.

Japan has the fastest machine (bar), although the United States still has the
most petascale computers (number in parentheses).

"CREDIT: ADAPTED FROM JACK DONGARRA/TOP 500 LIST/UNIVERSITY OF TENNESSEE"

On the rise.

The gap in available supercomputing capacity between the United States and
the rest of the world has narrowed, with China gaining the most ground.

"CREDIT: ADAPTED FROM JACK DONGARRA/TOP 500 LIST/UNIVERSITY OF TENNESSEE"

With the rise of mobile computing, chipmakers couldn't raise power demands
beyond what batteries could store. So about 10 years ago, chip manufacturers
began placing multiple processing “cores” side by side on single chips. This
arrangement meant that only twice the power was needed to double a chip's
performance.

This trend swept through the world of supercomputers. Those with single
souped-up processors gave way to today's “parallel” machines that couple vast
numbers of off-the-shelf commercial processors together. This move to
parallel computing “was a huge, disruptive change,” says Robert Lucas, an
electrical engineer at the University of Southern California's Information
Sciences Institute in Los Angeles.

Hardware makers and software designers had to learn how to split problems
apart, send individual pieces to different processors, synchronize the
results, and synthesize the final ensemble. Today's top machine—Japan's “K
Computer”—has 705,000 cores. If the trend continues, an exascale computer
would have between 100 million and 1 billion processors.

But simply scaling up today's models won't work. “Business as usual will not
get us to the exascale,” Simon says. “These computers are becoming so
complicated that a number of issues have come up that were not there before,”
Rosner agrees.

The biggest issue relates to a supercomputer's overall power use. The largest
supercomputers today use about 10 megawatts (MW) of power, enough to power
10,000 homes. If the current trend of power use continues, an exascale
supercomputer would require 200 MW. “It would take a nuclear power reactor to
run it,” Shank says.

Even if that much power were available, the cost would be prohibitive. At $1
million per megawatt per year, the electricity to run an exascale machine
would cost $200 million annually. “That's a non-starter,” Shank says. So the
current target is a machine that draws 20 MW at most. Even that goal will
require a 300-fold improvement in flops per watt over today's technology.

Ideas for getting to these low-power chips are already circulating. One would
make use of different types of specialized cores. Today's top-of-the-line
supercomputers already combine conventional processor chips, known as CPUs,
with an alternative version called graphical processing units (GPUs), which
are very fast at certain types of calculations. Chip manufacturers are now
looking at going from “multicore” chips with four or eight cores to
“many-core” chips, each containing potentially hundreds of CPU and GPU cores,
allowing them to assign different calculations to specialized processors.
That change is expected to make the overall chips more energy efficient.
Intel, AMD, and other chip manufacturers have already announced plans to make
hybrid many-core chips.

Another stumbling block is memory. As the number of processors in a
supercomputer skyrockets, so, too, does the need to add memory to feed bits
of data to the processors. Yet, over the next few years, memory manufacturers
are not projected to increase the storage density of their chips fast enough
to keep up with the performance gains of processors. Supercomputer makers can
get around this by adding additional memory modules. But that's threatening
to drive costs too high, Simon says.

Even if researchers could afford to add more memory modules, that still won't
solve matters. Moving ever-growing streams of data back and forth to
processors is already creating a backup for processors that can dramatically
slow a computer's performance. Today's supercomputers use 70% of their power
to move bits of data around from one place to another.

One potential solution would stack memory chips on top of one another and run
communication and power lines vertically through the stack. This more-compact
architecture would require fewer steps to route data. Another approach would
stack memory chips atop processors to minimize the distance bits need to
travel.

A third issue is errors. Modern processors compute with stunning accuracy,
but they aren't perfect. The average processor will produce one error per
year, as a thermal fluctuation or a random electrical spike flips a bit of
data from one value to another.

Such errors are relatively easy to ferret out when the number of processors
is low. But it gets much harder when 100 million to 1 billion processors are
involved. And increasing complexity produces additional software errors as
well. One possible solution is to have the supercomputer crunch different
problems multiple times and “vote” for the most common solution. But that
creates a new problem. “How can I do this without wasting double or triple
the resources?” Lucas asks. “Solving this problem will probably require new
circuit designs and algorithms.”

Finally, there is the challenge of redesigning the software applications
themselves, such as a novel climate model or a simulation of a chemical
reaction. “Even if we can produce a machine with 1 billion processors, it's
not clear that we can write software to use it efficiently,” Lucas says.
Current parallel computing machines use a strategy, known as message passing
interface, that divides computational problems and parses out the pieces to
individual processors, then collects the results. But coordinating all this
traffic for millions of processors is becoming a programming nightmare.
“There's a huge concern that the programming paradigm will have to change,”
Rosner says.

DOE has already begun laying the groundwork to tackle these and other
challenges. Last year it began funding three “co-design” centers,
multi-institution cooperatives led by researchers at Los Alamos, Argonne, and
Sandia national laboratories. The centers bring together scientific users who
write the software code and hardware makers to design complex software and
computer architectures that work in the fastest and most energy-efficient
manner. It poses a potential clash between scientists who favor openness and
hardware companies that normally keep their activities secret for proprietary
reasons. “But it's a worthy goal,” agrees Wilfred Pinfold, Intel's director
of extreme-scale programming in Hillsboro, Oregon.

Not so fast.

Researchers have some ideas on how to overcome barriers to building exascale
machines.

Coming up with the cash

Solving these challenges will take money, and lots of it. Two years ago,
Simon says, DOE officials estimated that creating an exascale computer would
cost $3 billion to $4 billion over 10 years. That amount would pay for one
exascale computer for classified defense work, one for nonclassified work,
and two 100-petaflops machines to work out some of the technology along the
way.

Those projections assumed that Congress would deliver a promised 10-year
doubling of the budget of DOE's Office of Science. But those assumptions are
“out of the window,” Simon says, replaced by the more likely scenario of
budget cuts as Congress tries to reduce overall federal spending.

Given that bleak fiscal picture, DOE officials must decide how aggressively
they want to pursue an exascale computer. “What's the right balance of being
aggressive to maintain a leadership position and having the plan sent back to
the drawing board by [the Office of Management and Budget]?” Simon asks. “I'm
curious to see.” DOE's strategic plan, due out next month, should provide
some answers.

The rest of the world faces a similar juggling act. China, Japan, the
European Union, Russia, and India all have given indications that they hope
to build an exascale computer within the next decade. Although none has
released detailed plans, each will need to find the necessary resources
despite these tight fiscal times.

The victor will reap more than scientific glory. Companies use 57% of the
computing time on the machines on the Top500 List, looking to speed product
design and gain other competitive advantages, Dongarra says. So government
officials see exascale computing as giving their industries a leg up. That's
particularly true for chip companies that plan to use exascale designs to
improve future commodity electronics. “It will have dividends all the way
down to the laptop,” says Peter Beckman, who directs the Exascale Technology
and Computing Initiative at Argonne National Laboratory in Illinois.

The race to provide the hardware needed for exascale computing “will be
extremely competitive,” Beckman predicts, and developing software and
networking technology will be equally important, according to Dongarra. Even
so, many observers think that the U.S. track record and the current alignment
of its political and scientific forces makes it America's race to lose.

Whatever happens, U.S. scientists are unlikely to be blindsided. The task of
building the world's first exascale computer is so complex, Simon says, that
it will be nearly impossible for a potential winner to hide in the shadows
and come out of nowhere to claim the prize.