[Beowulf] 100 Gigabit Ethernet: Bridge to Terabit Ethernet

[Eugen Leitl]

http://www.networkworld.com/news/2009/042009-terabit-ethernet.html?ts0hb&story=ts_spmc 

100 Gigabit Ethernet: Bridge to Terabit Ethernet

40 Gigabit and 100 Gigabit Ethernet products to ship by year-end; terabit
speeds anticipated by 2015 By Jim Duffy , Network World , 04/20/2009 

IT managers who are getting started with – or even pushing the limits of --
10 Gigabit Ethernet in their LANs and data centers won't have to wait long
for higher speed connectivity.

Pre-standard 40 Gigabit and 100 Gigabit Ethernet products – server network
interface cards, switch uplinks and switches -- are expected to hit the
market later this year. And standards-compliant products are expected to ship
in the second half of next year, not long after the expected June 2010
ratification of the 802.3ba standard.

The IEEE, which began work on the standard in late 2006, is expected to
define two different speeds of Ethernet for two different applications: 40G
for server connectivity and 100G for core switching.

Despite the global economic slowdown, global revenue for 10G fixed Ethernet
switches doubled in 2008, according to Infonetics. And there is pent-up
demand for 40 Gigabit and 100 Gigabit Ethernet, says John D'Ambrosia, chair
of the 802.3ba task force in the IEEE and a senior research scientist at
Force10 Networks.

"There are a number of people already who are using link aggregation to try
and create pipes of that capacity," he says. "It's not the cleanest way to do
things…(but) people already need that capacity."

D'Ambrosia says even though 40/100G Ethernet products haven't arrived yet,
he's already thinking ahead to Terabit Ethernet standards and products by
2015. "We are going to see a call for a higher speed much sooner than we saw
the call for this generation" of 10/40/100G Ethernet, he says.

According to the 802.3ba task force, bandwidth requirements for computing and
core networking applications are growing at different rates, which
necessitates the definition of two distinct data rates for the next
generation of Ethernet. Servers, high performance computing clusters, blade
servers, storage-area networks and network-attached storage all currently
make use of 1G and 10G Ethernet, with 10G growing significantly in 2007 and
2008.

I/O bandwidth projections for server and computing applications, including
server traffic aggregation, indicate that there will be a significant market
potential for a 40G Ethernet interface, according to the task force. Ethernet
at 40G will provide approximately the same cost balance between the LAN and
the attached stations as 10G Ethernet, the task force believes.

Core networking applications have demonstrated the need for bandwidth beyond
existing capabilities and beyond the projected bandwidth requirements for
computing applications. Switching, routing, and aggregation in data centers,
internet exchanges and service provider peering points, and high bandwidth
applications such as video on demand and high performance computing, need a
100 Gigabit Ethernet interface, according to the task force.

"Initial applications (of 40/100G Ethernet) are already showing up, in
stacking and highly aggregated LAN links, but the port counts are low," says
George Zimmerman, CTO of SolarFlare, a maker of Ethernet physical layer
devices.

Zimmerman says 10G is just now taking off in the access layer of large
networks and will eventually move to the client side, creating the need for
40/100G in the distribution layer and the network core.

He says the application of 100 Gigabit Ethernet in the core is imminent, and
is about two years away in the distribution layer. "Both will be driven by
and drive 10G adoption in the access and client end of the network, where
today the numbers are still much smaller than the potential," he says

Spec designed for seamless upgrades

The 802.3ba specification will conform to the full-duplex operating mode of
the IEEE 802.3 Media Access Control (MAC) layer, according to the task force.
As was the case in previous 802.3 amendments, new physical layers specific to
either 40Gbps or 100Gbps operation will be defined.

By employing the existing 802.3 MAC protocol, 802.3ba is intended to maintain
full compatibility with the installed base of Ethernet nodes, the task force
says. The spec is also expected to use "proven and familiar media," including
optical fiber, backplanes and copper cabling, and preserve existing network
architecture, management and software, in an effort to keep design,
installation and maintenance costs at a minimum.

Even though the 802.3ba standard is not expected to be ratified until June
2010, initial interoperability testing will commence later this year, says
Brad Booth, chair of the Ethernet Alliance. Public demonstrations will emerge
in 2010, and certification testing will start once the standard is ratified,
Booth says

The specification and formation of the 40/100G task force did not come
without some controversy, however. Participants in the Higher Speed Study
Group (HSSG) within the IEEE were divided on whether to include 40G Ethernet
as part of their charter or stay the course with 100 Gigabit Ethernet.  

After about a month though, the HSSG agreed to work on a single standard that
encompassed both 40G and 100G.

"In a sense, we were a little bit late with this," D'Ambrosia says. "By our
own projections, the need for 100G was in the 2010 timeframe. We should have
been done with the 100G (spec) probably in the 2007-08 timeframe, at the
latest. We actually started it late, which is going to make the push for
terabit seem early by comparison. But when we look at the data forecasts that
we're seeing, it looks to be on cue."

Driving demand for 40/100G Ethernet are the same drivers currently stoking
10G: data center virtualization and storage, and high-definition
videoconferencing and medical imaging. Some vendors are building 40/100G
Ethernet capabilities into their products now.  Vendors prepare for 100
Gigabit Ethernet

Cisco's Nexus 7000 data center switch, which debuted early last year, is
designed for future delivery of 40/100G Ethernet.

"We have a little more headroom, which isn't bad to have when you look at
future Ethernet speed transitions coming in the market," says Doug Gourlay,
senior director of data center marketing and product management at Cisco.
"We're pretty early advocates of the 100G effort in the IEEE.

"[But] the earliest you'll see products from any company that are credible
deliveries and reasonably priced: second half of 2010 onward for 40/100G," he
adds.

Verizon Business last fall began offering 10G Ethernet LAN and Ethernet
Virtual Private Line services to customers in 100 U.S. metro markets. Verizon
Business also offers "10G-capable" Ethernet Private Line services.

The carrier has 40G Ethernet services on its five-year road map but no
specific deployment dates, says Jeff Schwartz, Group Manager, Global Ethernet
Product Marketing. Instead, Verizon Business has more 10G Ethernet access
services on tap for later this year.

"We want to get to 100G," Schwartz says. "40G may be an intermediary step."

Once Verizon Business moves its backbone architecture towards 40/100G,
products and services will be following, he says.

Spirent Communications, a maker of Ethernet testing gear, plans to release a
40G Ethernet testing module in the second half of this year, and 100 Gigabit
Ethernet modules in early 2010, says Tim Jefferson, general manager of the
converged core solutions group at Spirent. Jefferson says one of the caveats
that users should be aware of as they migrate from 10G to 40/100G Ethernet is
the need to ensure precise clocking synchronization between systems –
especially between equipment from different vendors. 

Imprecise clocking between systems at 40/100G – even at 10G – can increase
latency and packet loss, Jefferson says.

"This latency issue is a bigger issue than most people anticipate," he says.
"At 10G, especially at high densities, the specs allow for a little variance
for clocks. As you aggregate traffic into 10G ports, just the smallest
difference in the clocks between ports can cause high latency and packet
loss. At 40G, it's an order of magnitude more important than it is for 10G
and Gig.

"This is a critical requirement in data centers today because a lot of the
innovations going on with Ethernet and a lot of the demand for all these
changes in data centers are meant to address lower latencies," Jefferson
adds.  Cabling challenges

Another challenge is readying the cabling infrastructure for 40/100G, experts
say. Ensuring the appropriate grade and length of fiber is essential to
smooth, seamless operation, they say.

"The big consideration is, what's a customer's cabling installation going to
look like and what they're looking for to be able to handle that," Booth
says. "They are probably going to need to have a parallel fiber capability."

"The recommendations we're making to customers on their physical plant today
are designed to take them from 1G to 10G; 10G to a unified fabric; and then
address future 40G," Cisco's Gourlay says.

The proposed physical interfaces (PHY) for 40G Ethernet include a range to
cover distances inside the data center up to 100 meters, to accommodate a
range of server form factors, including blade, rack and pedestal, according
to the Ethernet Alliance. The 100 Gigabit Ethernet rate will include
distances and media appropriate for data center, as well as service provider
interconnection for intra-office and inter-office applications, according to
the organization.

The proposed PHYs for 40G Ethernet are 1 meter backplane, 10 meter copper and
100 meter multimode fiber; and 10 meter copper, 100 meter multimode, and 10
kilometer and 40 kilometer single-mode fiber for 100 Gigabit Ethernet.