[Beowulf] Google Throws Open Doors to Its Top-Secret Data Center

[Eugen Leitl]

(PUE 1.2 doesn't sound so otherworldly to me)

http://www.wired.com/wiredenterprise/2012/10/ff-inside-google-data-center/all/

Google Throws Open Doors to Its Top-Secret Data Center

By Steven Levy 10.17.12 7:30 AM

Follow @StevenLevy

Photo: Google/Connie Zhou

If you’re looking for the beating heart of the digital age — a physical
location where the scope, grandeur, and geekiness of the kingdom of bits
become manifest—you could do a lot worse than Lenoir, North Carolina. This
rural city of 18,000 was once rife with furniture factories. Now it’s the
home of a Google data center.

Engineering prowess famously catapulted the 14-year-old search giant into its
place as one of the world’s most successful, influential, and frighteningly
powerful companies. Its constantly refined search algorithm changed the way
we all access and even think about information. Its equally complex
ad-auction platform is a perpetual money-minting machine. But other, less
well-known engineering and strategic breakthroughs are arguably just as
crucial to Google’s success: its ability to build, organize, and operate a
huge network of servers and fiber-optic cables with an efficiency and speed
that rocks physics on its heels. Google has spread its infrastructure across
a global archipelago of massive buildings—a dozen or so information palaces
in locales as diverse as Council Bluffs, Iowa; St. Ghislain, Belgium; and
soon Hong Kong and Singapore—where an unspecified but huge number of machines
process and deliver the continuing chronicle of human experience.

This is what makes Google Google: its physical network, its thousands of
fiber miles, and those many thousands of servers that, in aggregate, add up
to the mother of all clouds. This multibillion-dollar infrastructure allows
the company to index 20 billion web pages a day. To handle more than 3
billion daily search queries. To conduct millions of ad auctions in real
time. To offer free email storage to 425 million Gmail users. To zip millions
of YouTube videos to users every day. To deliver search results before the
user has finished typing the query. In the near future, when Google releases
the wearable computing platform called Glass, this infrastructure will power
its visual search results.

The problem for would-be bards attempting to sing of these data centers has
been that, because Google sees its network as the ultimate competitive
advantage, only critical employees have been permitted even a peek inside, a
prohibition that has most certainly included bards. Until now.

A server room in Council Bluffs, Iowa. Previous spread: A central cooling
plant in Google’s Douglas County, Georgia, data center.  Photo: Google/Connie
Zhou

Here I am, in a huge white building in Lenoir, standing near a reinforced
door with a party of Googlers, ready to become that rarest of species: an
outsider who has been inside one of the company’s data centers and seen the
legendary server floor, referred to simply as “the floor.” My visit is the
latest evidence that Google is relaxing its black-box policy. My hosts
include Joe Kava, who’s in charge of building and maintaining Google’s data
centers, and his colleague Vitaly Gudanets, who populates the facilities with
computers and makes sure they run smoothly.

A sign outside the floor dictates that no one can enter without hearing
protection, either salmon-colored earplugs that dispensers spit out like
trail mix or panda-bear earmuffs like the ones worn by airline ground crews.
(The noise is a high-pitched thrum from fans that control airflow.) We grab
the plugs. Kava holds his hand up to a security scanner and opens the heavy
door. Then we slip into a thunderdome of data …

Urs Hölzle had never stepped into a data center before he was hired by Sergey
Brin and Larry Page. A hirsute, soft-spoken Swiss, Hölzle was on leave as a
computer science professor at UC Santa Barbara in February 1999 when his new
employers took him to the Exodus server facility in Santa Clara. Exodus was a
colocation site, or colo, where multiple companies rent floor space. Google’s
“cage” sat next to servers from eBay and other blue-chip Internet companies.
But the search company’s array was the most densely packed and chaotic. Brin
and Page were looking to upgrade the system, which often took a full 3.5
seconds to deliver search results and tended to crash on Mondays. They
brought Hözle on to help drive the effort.

It wouldn’t be easy. Exodus was “a huge mess,” Hölzle later recalled. And the
cramped hodgepodge would soon be strained even more. Google was not only
processing millions of queries every week but also stepping up the frequency
with which it indexed the web, gathering every bit of online information and
putting it into a searchable format. AdWords—the service that invited
advertisers to bid for placement alongside search results relevant to their
wares—involved computation-heavy processes that were just as demanding as
search. Page had also become obsessed with speed, with delivering search
results so quickly that it gave the illusion of mind reading, a trick that
required even more servers and connections. And the faster Google delivered
results, the more popular it became, creating an even greater burden.
Meanwhile, the company was adding other applications, including a mail
service that would require instant access to many petabytes of storage. Worse
yet, the tech downturn that left many data centers underpopulated in the late
’90s was ending, and Google’s future leasing deals would become much more
costly.

For Google to succeed, it would have to build and operate its own data
centers—and figure out how to do it more cheaply and efficiently than anyone
had before. The mission was codenamed Willpower. Its first built-from-scratch
data center was in the Dalles, a city in Oregon near the Columbia River.

Hözle and his team designed the $600 million facility in light of a radical
insight: Server rooms did not have to be kept so cold. The machines throw off
prodigious amounts of heat. Traditionally, data centers cool them off with
giant computer room air conditioners, or CRACs, typically jammed under raised
floors and cranked up to arctic levels. That requires massive amounts of
energy; data centers consume up to 1.5 percent of all the electricity in the
world.  Data centers consume up to 1.5 percent of all the world’s
electricity.

Google realized that the so-called cold aisle in front of the machines could
be kept at a relatively balmy 80 degrees or so—workers could wear shorts and
T-shirts instead of the standard sweaters. And the “hot aisle,” a tightly
enclosed space where the heat pours from the rear of the servers, could be
allowed to hit around 120 degrees. That heat could be absorbed by coils
filled with water, which would then be pumped out of the building and cooled
before being circulated back inside. Add that to the long list of Google’s
accomplishments: The company broke its CRAC habit.

Google also figured out money-saving ways to cool that water. Many data
centers relied on energy-gobbling chillers, but Google’s big data centers
usually employ giant towers where the hot water trickles down through the
equivalent of vast radiators, some of it evaporating and the remainder
attaining room temperature or lower by the time it reaches the bottom. In its
Belgium facility, Google uses recycled industrial canal water for the
cooling; in Finland it uses seawater.

The company’s analysis of electrical flow unearthed another source of waste:
the bulky uninterrupted-power-supply systems that protected servers from
power disruptions in most data centers. Not only did they leak electricity,
they also required their own cooling systems. But because Google designed the
racks on which it placed its machines, it could make space for backup
batteries next to each server, doing away with the big UPS units altogether.
According to Joe Kava, that scheme reduced electricity loss by about 15
percent.

All of these innovations helped Google achieve unprecedented energy savings.
The standard measurement of data center efficiency is called power usage
effectiveness, or PUE. A perfect number is 1.0, meaning all the power drawn
by the facility is put to use. Experts considered 2.0—indicating half the
power is wasted—to be a reasonable number for a data center. Google was
getting an unprecedented 1.2.

For years Google didn’t share what it was up to. “Our core advantage really
was a massive computer network, more massive than probably anyone else’s in
the world,” says Jim Reese, who helped set up the company’s servers. “We
realized that it might not be in our best interest to let our competitors
know.”

But stealth had its drawbacks. Google was on record as being an exemplar of
green practices. In 2007 the company committed formally to carbon neutrality,
meaning that every molecule of carbon produced by its activities—from
operating its cooling units to running its diesel generators—had to be
canceled by offsets. Maintaining secrecy about energy savings undercut that
ideal: If competitors knew how much energy Google was saving, they’d try to
match those results, and that could make a real environmental impact. Also,
the stonewalling, particularly regarding the Dalles facility, was becoming
almost comical. Google’s ownership had become a matter of public record, but
the company still refused to acknowledge it.

In 2009, at an event dubbed the Efficient Data Center Summit, Google
announced its latest PUE results and hinted at some of its techniques. It
marked a turning point for the industry, and now companies like Facebook and
Yahoo report similar PUEs.

Make no mistake, though: The green that motivates Google involves
presidential portraiture. “Of course we love to save energy,” Hölzle says.
“But take something like Gmail. We would lose a fair amount of money on Gmail
if we did our data centers and servers the conventional way. Because of our
efficiency, we can make the cost small enough that we can give it away for
free.”

Google’s breakthroughs extend well beyond energy. Indeed, while Google is
still thought of as an Internet company, it has also grown into one of the
world’s largest hardware manufacturers, thanks to the fact that it builds
much of its own equipment. In 1999, Hözle bought parts for 2,000
stripped-down “breadboards” from “three guys who had an electronics shop.” By
going homebrew and eliminating unneeded components, Google built a batch of
servers for about $1,500 apiece, instead of the then-standard $5,000. Hölzle,
Page, and a third engineer designed the rigs themselves. “It wasn’t really
‘designed,’” Hölzle says, gesturing with air quotes.

More than a dozen generations of Google servers later, the company now takes
a much more sophisticated approach. Google knows exactly what it needs inside
its rigorously controlled data centers—speed, power, and good connections—and
saves money by not buying unnecessary extras. (No graphics cards, for
instance, since these machines never power a screen. And no enclosures,
because the motherboards go straight into the racks.) The same principle
applies to its networking equipment, some of which Google began building a
few years ago.

Outside the Council Bluffs data center, radiator-like cooling towers chill
water from the server floor down to room temperature.  Photo: Google/Connie
Zhou

So far, though, there’s one area where Google hasn’t ventured: designing its
own chips. But the company’s VP of platforms, Bart Sano, implies that even
that could change. “I’d never say never,” he says. “In fact, I get that
question every year. From Larry.”

Even if you reimagine the data center, the advantage won’t mean much if you
can’t get all those bits out to customers speedily and reliably. And so
Google has launched an attempt to wrap the world in fiber. In the early
2000s, taking advantage of the failure of some telecom operations, it began
buying up abandoned fiber-optic networks, paying pennies on the dollar. Now,
through acquisition, swaps, and actually laying down thousands of strands,
the company has built a mighty empire of glass.

But when you’ve got a property like YouTube, you’ve got to do even more. It
would be slow and burdensome to have millions of people grabbing videos from
Google’s few data centers. So Google installs its own server racks in various
outposts of its network—mini data centers, sometimes connected directly to
ISPs like Comcast or AT&T—and stuffs them with popular videos. That means
that if you stream, say, a Carly Rae Jepsen video, you probably aren’t
getting it from Lenoir or the Dalles but from some colo just a few miles from
where you are.

Over the years, Google has also built a software system that allows it to
manage its countless servers as if they were one giant entity. Its in-house
developers can act like puppet masters, dispatching thousands of computers to
perform tasks as easily as running a single machine. In 2002 its scientists
created Google File System, which smoothly distributes files across many
machines. MapReduce, a Google system for writing cloud-based applications,
was so successful that an open source version called Hadoop has become an
industry standard. Google also created software to tackle a knotty issue
facing all huge data operations: When tasks come pouring into the center, how
do you determine instantly and most efficiently which machines can best
afford to take on the work? Google has solved this “load-balancing” issue
with an automated system called Borg.

These innovations allow Google to fulfill an idea embodied in a 2009 paper
written by Hözle and one of his top lieutenants, computer scientist Luiz
Barroso: “The computing platform of interest no longer resembles a pizza box
or a refrigerator but a warehouse full of computers … We must treat the data
center itself as one massive warehouse-scale computer.”

This is tremendously empowering for the people who write Google code. Just as
your computer is a single device that runs different programs
simultaneously—and you don’t have to worry about which part is running which
application—Google engineers can treat seas of servers like a single unit.
They just write their production code, and the system distributes it across a
server floor they will likely never be authorized to visit. “If you’re an
average engineer here, you can be completely oblivious,” Hözle says. “You can
order x petabytes of storage or whatever, and you have no idea what actually
happens.”

But of course, none of this infrastructure is any good if it isn’t reliable.
Google has innovated its own answer for that problem as well—one that
involves a surprising ingredient for a company built on algorithms and
automation: people.

At 3 am on a chilly winter morning, a small cadre of engineers begin to
attack Google. First they take down the internal corporate network that
serves the company’s Mountain View, California, campus. Later the team
attempts to disrupt various Google data centers by causing leaks in the water
pipes and staging protests outside the gates—in hopes of distracting
attention from intruders who try to steal data-packed disks from the servers.
They mess with various services, including the company’s ad network. They
take a data center in the Netherlands offline. Then comes the coup de
grâce—cutting most of Google’s fiber connection to Asia.

Turns out this is an inside job. The attackers, working from a conference
room on the fringes of the campus, are actually Googlers, part of the
company’s Site Reliability Engineering team, the people with ultimate
responsibility for keeping Google and its services running. SREs are not
merely troubleshooters but engineers who are also in charge of getting
production code onto the “bare metal” of the servers; many are embedded in
product groups for services like Gmail or search. Upon becoming an SRE,
members of this geek SEAL team are presented with leather jackets bearing a
military-style insignia patch. Every year, the SREs run this simulated
war—called DiRT (disaster recovery testing)—on Google’s infrastructure. The
attack may be fake, but it’s almost indistinguishable from reality: Incident
managers must go through response procedures as if they were really
happening. In some cases, actual functioning services are messed with. If the
teams in charge can’t figure out fixes and patches to keep things running,
the attacks must be aborted so real users won’t be affected. In classic
Google fashion, the DiRT team always adds a goofy element to its dead-serious
test—a loony narrative written by a member of the attack team. This year it
involves a Twin Peaks-style supernatural phenomenon that supposedly caused
the disturbances. Previous DiRTs were attributed to zombies or aliens.

Some halls in Google’s Hamina, Finland, data center remain vacant—for now.
Photo: Google/Connie Zhou

As the first attack begins, Kripa Krishnan, an upbeat engineer who heads the
annual exercise, explains the rules to about 20 SREs in a conference room
already littered with junk food. “Do not attempt to fix anything,” she says.
“As far as the people on the job are concerned, we do not exist. If we’re
really lucky, we won’t break anything.” Then she pulls the plug—for real—on
the campus network. The team monitors the phone lines and IRC channels to see
when the Google incident managers on call around the world notice that
something is wrong. It takes only five minutes for someone in Europe to
discover the problem, and he immediately begins contacting others.

“My role is to come up with big tests that really expose weaknesses,”
Krishnan says. “Over the years, we’ve also become braver in how much we’re
willing to disrupt in order to make sure everything works.” How did Google do
this time? Pretty well. Despite the outages in the corporate network,
executive chair Eric Schmidt was able to run a scheduled global all-hands
meeting. The imaginary demonstrators were placated by imaginary pizza. Even
shutting down three-fourths of Google’s Asia traffic capacity didn’t shut out
the continent, thanks to extensive caching. “This is the best DiRT ever!”
Krishnan exclaimed at one point.

The SRE program began when Hözle charged an engineer named Ben Treynor with
making Google’s network fail-safe. This was especially tricky for a massive
company like Google that is constantly tweaking its systems and
services—after all, the easiest way to stabilize it would be to freeze all
change. Treynor ended up rethinking the very concept of reliability. Instead
of trying to build a system that never failed, he gave each service a
budget—an amount of downtime it was permitted to have. Then he made sure that
Google’s engineers used that time productively. “Let’s say we wanted Google+
to run 99.95 percent of the time,” Hözle says. “We want to make sure we don’t
get that downtime for stupid reasons, like we weren’t paying attention. We
want that downtime because we push something new.”

Nevertheless, accidents do happen—as Sabrina Farmer learned on the morning of
April 17, 2012. Farmer, who had been the lead SRE on the Gmail team for a
little over a year, was attending a routine design review session. Suddenly
an engineer burst into the room, blurting out, “Something big is happening!”
Indeed: For 1.4 percent of users (a large number of people), Gmail was down.
Soon reports of the outage were all over Twitter and tech sites. They were
even bleeding into mainstream news.

The conference room transformed into a war room. Collaborating with a peer
group in Zurich, Farmer launched a forensic investigation. A breakthrough
came when one of her Gmail SREs sheepishly admitted, “I pushed a change on
Friday that might have affected this.” Those responsible for vetting the
change hadn’t been meticulous, and when some Gmail users tried to access
their mail, various replicas of their data across the system were no longer
in sync. To keep the data safe, the system froze them out.

The diagnosis had taken 20 minutes, designing the fix 25 minutes more—pretty
good. But the event went down as a Google blunder. “It’s pretty painful when
SREs trigger a response,” Farmer says. “But I’m happy no one lost data.”
Nonetheless, she’ll be happier if her future crises are limited to DiRT-borne
zombie attacks.

One scenario that dirt never envisioned was the presence of a reporter on a
server floor. But here I am in Lenoir, earplugs in place, with Joe Kava
motioning me inside.

We have passed through the heavy gate outside the facility, with
remote-control barriers evoking the Korean DMZ. We have walked through the
business offices, decked out in Nascar regalia. (Every Google data center has
a decorative theme.) We have toured the control room, where LCD dashboards
monitor every conceivable metric. Later we will climb up to catwalks to
examine the giant cooling towers and backup electric generators, which look
like Beatle-esque submarines, only green. We will don hard hats and tour the
construction site of a second data center just up the hill. And we will stare
at a rugged chunk of land that one day will hold a third mammoth
computational facility.

But now we enter the floor. Big doesn’t begin to describe it. Row after row
of server racks seem to stretch to eternity. Joe Montana in his prime could
not throw a football the length of it.

During my interviews with Googlers, the idea of hot aisles and cold aisles
has been an abstraction, but on the floor everything becomes clear. The cold
aisle refers to the general room temperature—which Kava confirms is 77
degrees. The hot aisle is the narrow space between the backsides of two rows
of servers, tightly enclosed by sheet metal on the ends. A nest of copper
coils absorbs the heat. Above are huge fans, which sound like jet engines
jacked through Marshall amps.  The huge fans sound like jet engines jacked
through Marshall amps.

We walk between the server rows. All the cables and plugs are in front, so no
one has to crack open the sheet metal and venture into the hot aisle, thereby
becoming barbecue meat. (When someone does have to head back there, the
servers are shut down.) Every server has a sticker with a code that
identifies its exact address, useful if something goes wrong. The servers
have thick black batteries alongside. Everything is uniform and in
place—nothing like the spaghetti tangles of Google’s long-ago Exodus era.

Blue lights twinkle, indicating … what? A web search? Someone’s Gmail
message? A Glass calendar event floating in front of Sergey’s eyeball? It
could be anything.

Every so often a worker appears—a long-haired dude in shorts propelling
himself by scooter, or a woman in a T-shirt who’s pushing a cart with a
laptop on top and dispensing repair parts to servers like a psychiatric nurse
handing out meds. (In fact, the area on the floor that holds the replacement
gear is called the pharmacy.)

How many servers does Google employ? It’s a question that has dogged
observers since the company built its first data center. It has long stuck to
“hundreds of thousands.” (There are 49,923 operating in the Lenoir facility
on the day of my visit.) I will later come across a clue when I get a peek
inside Google’s data center R&D facility in Mountain View. In a secure area,
there’s a row of motherboards fixed to the wall, an honor roll of generations
of Google’s homebrewed servers. One sits atop a tiny embossed plaque that
reads july 9, 2008. google’s millionth server. But executives explain that
this is a cumulative number, not necessarily an indication that Google has a
million servers in operation at once.

Wandering the cold aisles of Lenoir, I realize that the magic number, if it
is even obtainable, is basically meaningless. Today’s machines, with
multicore processors and other advances, have many times the power and
utility of earlier versions. A single Google server circa 2012 may be the
equivalent of 20 servers from a previous generation. In any case, Google
thinks in terms of clusters—huge numbers of machines that act together to
provide a service or run an application. “An individual server means
nothing,” H\0xF6lzle says. “We track computer power as an abstract metric.”
It’s the realization of a concept H\0xF6lzle and Barroso spelled out three
years ago: the data center as a computer.

As we leave the floor, I feel almost levitated by my peek inside Google’s
inner sanctum. But a few weeks later, back at the Googleplex in Mountain
View, I realize that my epiphanies have limited shelf life. Google’s
intention is to render the data center I visited obsolete. “Once our people
get used to our 2013 buildings and clusters,” Hözle says, “they’re going to
complain about the current ones.”

Asked in what areas one might expect change, Hözle mentions data center and
cluster design, speed of deployment, and flexibility. Then he stops short.
“This is one thing I can’t talk about,” he says, a smile cracking his bearded
visage, “because we’ve spent our own blood, sweat, and tears. I want others
to spend their own blood, sweat, and tears making the same discoveries.”
Google may be dedicated to providing access to all the world’s data, but some
information it’s still keeping to itself.

Senior writer Steven Levy (steven_levy at wired.com) interviewed Mary Meeker in
issue 20.10.