Big Data's Surprising Uses: From Lady Gaga To CIA

Facebook and the Open Compute Project (OCP) announced Wednesday that they've made huge strides toward the goal of setting standards for the most efficient server, storage and data center hardware available for scalable computing.

Facebook launched OCP 18 months ago hoping to crowdsource the problem of creating better hardware for high-scale computing. From its start with one member, Facebook, and 200 participants, the group now has more than 50 member companies and saw more than 2,000 participants attend this week's Open Compute Platform Summit in Santa Clara, Calif.

OCP says its standards promise to deliver hardware that is 24% more energy efficient and 38% more cost efficient, on average, than so-called commodity hardware. The group is working on specs for storage, motherboard and server design, racks, interoperability, hardware management, and data center design.

"We know what we need and we see the current and future challenges that the deluge of data, analytics, compute power and these massively scaled data centers are bringing forward," said Frank Frankovsky, VP of hardware design and supply chain at Facebook and chairman of OCP's Open Compute Foundation, which is modeled after the Apache Software Foundation.

[ Want more on Facebook big data projects? Read Meet Facebook's Graph Search Tool. ]

Hardware vendors have also joined OCP and are designing hardware to its specs. AMD and Intel announced in May collaborative work with OCP members Fidelity and Goldman Sachs to develop new boards to suit their financial processing workloads, and both chip makers presented finished products at this week's event.

Intel has contributed specs to OCP for Silicon Photonic interconnect technologies that already surpass 100 gigabits per second -- nearly twice the speed of the fastest interconnect technologies currently available. Presenting on the technology, Intel CTO Justin Rattner said the products will soon be commercially available from Intel and would solve data I/O constraints both inside servers and in connecting servers within racks and racks within data centers.

"It has been a decade-long effort to create the technology, and it has enormous potential due to its combination of speed, low power consumption and low cost," Rattner said.

Dell also presented at the event, showing off Intel- and ARM-based OCP-compliant servers that will soon be commercially available. Other manufacturers working on or delivering OCP-compliant hardware and components include Applied Micro, Calxeda, Delta, Emerson and Hyve Solutions.

Many of the cost savings in OCP-standard products come down to two design themes: commonality and disaggregation. In an example of commonality, OCP unveiled a "common slot architecture" that will enable ARM and X86 chips to coexist on the same motherboard. The design will give end-user organizations flexibility to test and configure servers for best possible performance and then reconfigure as workloads change without facing proprietary design constraints. The spec is being supported by AMD, Applied Micro, Calxeda and Intel.

Disaggregation is about separating and modularizing storage, compute, interconnects, power, cooling and other components so companies can custom configure to their workload requirements.

This approach also supports smarter technology refreshes, so companies can swap out and replace quickly evolving components, such as CPUs, while keeping in service slowly evolving components, such as memory and network interface cards.

"You might be five generations behind on the processor while 80% of the other components of the server are still good," Frankovsky said, noting that monolithic designs don't let you replace dated components such as CPUs. "Smarter technology refreshes not only make sense from a financial perspective; think about how much less IT equipment is going to go into the waste stream."

While many OCP members are hyper-scale Internet companies, the point was made Wednesday that the group's efforts will benefit companies with mainstream computing requirements within a few short years.

"Hadoop, Hive, Hbase and other new platforms are accelerating the growth of big data adoption, so our big data challenges of today are everyone's big data challenges of tomorrow," said Jay Parikh, Facebook's VP of infrastructure engineering.