Fiber Mountain's goal is to make data center networks more scalable with fiber and photonic switches, creating "the programmable light path."

Charles Babcock, Editor at Large, Cloud

January 16, 2015

6 Min Read
M.H. Raza

Cloud Storage: 8 Ways You're Wasting Money

Cloud Storage: 8 Ways You're Wasting Money


Cloud Storage: 8 Ways You're Wasting Money (Click image for larger view and slideshow.)

There's a traffic-aggregating energy hog hiding in the enterprise data center, but cloud service providers and other hyperscale data center builders have found a way to hunt it down and root it out.

It's the high-traffic, copper-based, top-of-rack (ToR) switch and the centralized data center switches to which it connects. Both are standard features of the typical enterprise data center, but they are a threatened species inside the likes of Google, Facebook, and other hyperscale data centers.  

Both the ToR and nearby "core" switches are running constantly, feeding server interactions to the storage network, other racks in the data center, the corporate network, and the public Internet. In some cases, copper has been replaced by fiber optic cables, but Facebook and an increasing number of data center advocates are saying more drastic design changes are needed.

At its Altoona, Iowa, data center, which opened last November, Facebook built a facility that looks much like its Prineville, Ore., data center.  But on the inside, it added a new feature: a data center network fabric that relies heavily on fiber optic cable and reduces the number of big, energy-hungry switches. Facebook's chief goal was to make the network more scalable, based on Open Compute white-box switches, without adding complexity.

[Want to learn more about the scale of Facebook's data centers? See Facebook's Data Center: Where Likes Live.]

Other advocates are saying Facebook's re-crafted network points in the right direction, but they can carry horizontal scaling out even further. With their photonic, or light-pulse, switch designs, their customers will get more cost savings and reduced energy use. The advocates include Calient Technologies and Fiber Mountain, young companies with no investment in the copper world. Furthermore, innovations in fiber optics and photonic switches are making more radical design ideas possible.

"We love Facebook. They're proving you don't need big core switches," said M.H. Raza, founder and CEO of two-year-old Fiber Mountain, a maker of a cross-connect photonic switch and optical ToR switches. What he proposes instead is his company's cross-bar photonic switch and an innovation in data center network cabling.

Intel and Corning Glass collaborated in early 2014 to produce intelligent fiber optic cable, which has MXC connectors with an embedded chip.

The chip can tell surrounding devices that it's there and that it has a unique TCP/IP or MAC address. The intelligent cable comes in an insulating case that looks like any other data center cable, but in fact it sheaths four ribbons. Each ribbon is composed of 16 fiber optic channels. Instead of one communications line, the cable houses 64.

"With this many lanes, you can make each server its own switch," said Raza in an interview at the InformationWeek offices.

"We want to flood the data center with fiber optic capacity," he said. Each line of fiber will be mapped by a central network management system. Fiber Mountain calls its system the Alpine Orchestration System, and it knows which lines are in use and which ones represent surplus capacity. Any time a line is disconnected, that's logged as an event by Alpine; likewise, any new connection is added to the data center map. The map is up to date, even when it's changing, Raza told us.

Getting data center architects to think about "flooding the data center with fiber" is hard, he conceded, because high-speed fiber optic cable is considered an expensive alternative that's reserved for high-level uses. But the economics are changing with the innovations. And a data center network with lots of surplus bandwidth is easier to manage and move things around in than a constricted one.

Fiber Mountain makes a ToR switch, the AP 3620, but Raza believes that most of the ToR switch function will soon disappear into the server itself. Fiber Mountain's ToR switch is priced at $30,000. Facebook uses white-box switches that Raza estimates cost only $10,000. But each white box needs two transceivers to convert electrical impulses to light impulses for the new Facebook network fabric.

In the future, if there's plentiful fiber available, servers can be tied directly into a core switch. In the Fiber Mountain scheme, that would be a "Glass Core" cross-bar switch, which has the capacity to connect any server in a row of 10 or 12 server racks to any other server. The cross-bar switch contains an endpoint for each device on the network and connects one directly to another without the need to go through a router.

Either a photonic ToR switch or, someday, a direct connection from the server to a Glass Core switch has the effect of simplifying the network and reducing its energy consumption. Fewer switches serve a cluster of racks, and one core switch can both connect everything in the cluster and connect it to other clusters. Existing data center networks rely on hub-and-spoke designs with very large switches close to the core and smaller switches as you progress toward the edge.

A flatter and simpler data center network will better lend itself to software-defined networking and network virtualization. Virtualizing enterprise servers has led to big savings in IT hardware expenses; over the next 10 years virtualizing the network and equipping it with fiber and photonics will produce similar savings in capital and energy expenses. The new fiber cables and switches produce twice the capacity at half the capital expense cost, Raza claimed.

Still, he runs into the problem that he's asking large enterprises to take out equipment made by Cisco, with 72,247 employees, and replace it with equipment from Fiber Mountain, with 25 employees. Raza says he's gotten a few in the New York area to do so. That is, he has purchase orders from them to start working with the kind of data center network that he advocates. But he can't name them, and he doesn't have any reference customers yet.

Nevertheless, the data center of the future, whether in the cloud, like Facebook, or on-premises in the enterprise, will be a compute-intensive place. It will be running thousands or hundreds of thousands of virtual machines and Linux containers. It will dynamically shuffle them around and need network resources to match the high degree of compute virtualization.

If idle network resources can be quickly assigned, or networks reconfigured to create a path with sufficient bandwidth, such moves could be executed programmatically by management systems that know when there's sufficient network capacity to complete the move in the required time.

Raza wants Fiber Mountain to supply the switching for the future software-driven data center, one composed of fiber optics and photonics, not copper. When one application wants to talk to another on the other side of the data center, Fiber Mountain's Alpine will respond, "I'll create a programmable light path."

Apply now for the 2015 InformationWeek Elite 100, which recognizes the most innovative users of technology to advance a company's business goals. Winners will be recognized at the InformationWeek Conference, April 27-28, 2015, at the Mandalay Bay in Las Vegas. Application period ends Jan. 16, 2015.

About the Author(s)

Charles Babcock

Editor at Large, Cloud

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive Week. He is a graduate of Syracuse University where he obtained a bachelor's degree in journalism. He joined the publication in 2003.

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like


More Insights