[Beowulf] Facebook: Yes, We Need 100-GigE

[Eugen Leitl]

http://www.lightreading.com/document.asp?doc_id=181899

Facebook: Yes, We Need 100-GigE

September 16, 2009 | Craig Matsumoto | Comments (13)

It's become cliché to say that companies like Facebook would use 100-Gbit/s
Ethernet right now if they had it. But it helps when someone from Facebook
actually shows up and hammers on that point.

Facebook network engineer Donn Lee did that yesterday, pleading his case at a
technology seminar on 40- and 100-Gbit/s Ethernet, hosted in Santa Clara,
Calif., by The Ethernet Alliance .

Representatives from Google (Nasdaq: GOOG) and the Amsterdam Internet
Exchange B.V. (AMS-IX) gave similar pleas, but Lee's presentation included
some particularly sobering numbers. He said it's reasonable to think Facebook
will need its data center backbone fabric to grow to 64 Tbit/s total capacity
by the end of next year.

How to build such a thing? Lee said his ideal Ethernet box would have
16-Tbit/s switching capacity and 80 100-Gbit/s Ethernet ports or 800
10-Gbit/s Ethernet ports.

No such box exists commercially, and Lee is reluctant to go build his own.

That leaves him with an unpleasant alternative. Lee drew up a diagram of what
Facebook's future data center fabric -- that is, the interconnection of its
switch/routers -- would look like if he had to use today's equipment and
10-Gbit/s Ethernet. Instead of the familiar criss-crossing mesh diagram, he
got a solid wall of black, signifying just how many connections he'd need.

"I would say anybody in the top 25 Websites easily has this problem," he said
later.

(Lee didn't say anything about how long it would take just to plug in all
those fibers. Maybe that job could be created by funds from the U.S. Recovery
Act.)

Lee also showed charts showing the disconnect between Facebook's wish list
and the market. Facebook needed 512 10-Gbit/s Ethernet ports per chassis in
2007 and is likely to need 1,500 in 2010. No chassis offers more than 200
ports, he said.

Even though Lee is a veteran of Google and Cisco Systems Inc. (Nasdaq: CSCO),
you might wonder if he's just one renegade engineer who doesn't represent the
Facebook norm. Not really. It turns out Facebook has only five network
engineers -- although Lee said that's a 20 percent increase from the spring
of 2008 [math note: which means they had approximately 4.165 engineers at
that time].

Even though 100-Gbit/s development started four years ago, Lee thinks it came
too late, and that's got him worried about the next generation. He's pulling
for 400-Gbit/s Ethernet discussions to start right away.

"Let's start the work that doesn't require money, now," he said. "If we have
the standard, we can build the product later. I don't mind using an old
standard."

He might get his wish. The Optoelectronics Industry Development Association
(OIDA) is already organizing meetings with an aim toward getting federal
money for terabit Ethernet research, said John D'Ambrosia, a Force10 Networks
Inc. scientist who helped organize yesterday's event.

Of course, money is a major obstacle to the next wave of Ethernet.

During an open commenting and Q&A session, multiple audience members pointed
out that optical components margins are too thin to support advanced research
at many companies, and that carriers are seeing their big, expensive networks
getting used to make money for over-the-top services. "There's no revenue in
all that bandwidth increase," one audience member commented, citing the
carrier case in particular.