University Launches Data Center Lab, Consolidation Effort

Despite years of Lip Service, most businesses have done a poor job of using technology to manage their technology. They've tried standardization, consolidation, automation, best practices, and outsourcing. Yet they're still spending about 80% of their IT budgets to maintain systems and keep the lights on, while only 20% goes toward new technology and innovation.

Changing that ratio is difficult, despite falling hardware prices and the availability of inexpensive or free open source software. As companies consolidate data centers, it can take more costly electricity to power and cool closely packed racks of servers and storage systems. And those centers require more experienced--and more expensive--staffers to manage the systems, applications, and databases.

Tech companies, which are trying to help their customers deal with these issues, face the same challenges. Hewlett-Packard last week said it will consolidate 85 data centers worldwide into six centers in three U.S. cities. The effort is expected to reduce the company's IT spending by about $1 billion in the coming years and dramatically shift where the money goes. HP spends about half its IT budget on operations and half on new initiatives and innovation, says Randy Mott, HP's executive VP and CIO.

Once the consolidation is completed, HP expects spending on IT operations to drop to around 20% of the IT budget. "There are huge costs to be saved by making the right technology investments so that you are utilizing the power of technology," says Mott, a former CIO at Wal-Mart and Dell.

The latest challenge is the cost of energy, which has spiked with the jump in oil prices. In many cases, it now costs more to power and cool a data center than to buy the servers and storage systems to fill it. Tech vendors are trying to help. Advanced Micro Devices and Intel are producing dual-core server processors with built-in virtualization capabilities that require less power and throw off less heat, reducing cooling needs. And IBM last week introduced the IBM PowerExecutive, a system to automate the management of power consumption in data centers to cut energy costs.

Define The Problem

The best first step is to clearly understand where the IT dollars go--but that isn't always easy. A survey of IT executives by Forrester Research shows that 38% of IT budgets is spent on salaries and benefits for full-time staff, more than twice what's spent on hardware. A Carnegie Mellon University survey of dozens of IT managers shows that IT personnel costs range from four to seven times the cost of hardware.

"It's amazing how poorly as an industry we understand these costs," says Greg Ganger, director of Carnegie Mellon's Parallel Data Laboratory. "If we can't define the problem, you're going to have a hard time coming up with answers."

The university started to work more intensively on the problem last week when it opened its Data Center Observatory, which will serve as a consolidated data center for the school and as a lab for researchers to study data center operations and costs. The main focus: Develop strategies for improving energy efficiency in data centers and reduce management and administrative workloads and costs. Carnegie Mellon also is moving a dozen or more small server clusters used by various research groups at the university into the center.

Carnegie Mellon's efforts are being supported and funded by HP, IBM, Intel, Microsoft, Sun Microsystems, Symantec, and other vendors as well as government sponsors, including the Air Force Office for Scientific Research, the Army Research Office, the Defense Advanced Research Projects Agency, and the National Science Foundation.

The university plans to fill the 2,000-square-foot facility with 40 racks of servers. Just that level of density presents problems: It will require about 775 kilowatts of energy to power the center--enough to run about 750 2,000-square-foot homes. To power and cool a center of that size also will require a raised floor 3 to 4 feet deep to run power cables and cooling conduits and systems, rather than the usual 31-inch raised floor.

Carnegie Mellon installed American Power Conversion's power and cooling InfraStruXure technology, including the Hot-Aisle Containment System. A conventional approach to data center cooling uses hot and cool aisles of equipment to keep cool air flowing throughout. The APC system builds modular units of heat-generating computing, storage, and networking equipment and places them in a walled-in structure that prevents hot air from moving into the data center. That structure is cooled with chilled water.

"It's kind of like an oven in a house," says Ronald Seftick, VP of construction and facilities engineering at APC. "When you're trying to cook something, you don't keep the oven door open."

Waste Not, Want Not

But Carnegie Mellon has more ambitious goals: It wants to fully understand where its IT dollars go and then change those spending patterns. It has launched a project that requires data center managers and staff to create detailed logs of all their time spent working in the data center. Within six months, Ganger expects the university to be able to analyze the collected data and begin creating new management tools to attack areas identified as most critical and wasteful.

The university may be trying to reinvent the wheel. The IT Infrastructure Library offers best practices to reduce the costs of dealing with common IT problems. There are a host of systems management platforms and applications available from a variety of vendors, and companies such as HP, IBM, and Symantec have devoted years to building systems that automate many aspects of systems and network management. IBM last week said it plans to ship its much-anticipated Tivoli Change and Configuration Management Database in June. It's designed to make it easier to make and manage changes to servers, applications, and other systems in complex IT environments.

"I think we know" where the money goes, HP's Mott says. "We make a business of it as far as helping our customers understand those implications and what the costs are. You can get tools and products to better manage those resources, you can get consulting and integration services, and we can come in from a managed services standpoint and take control of the data center. What I do think is lacking is a recognition of the size of this problem."

The big problem, Mott says, is that most businesses don't use enough technology to manage their technology. "We are undercapitalized as an industry because we have all these operational costs," he says.

New tech trends such as virtualization, grid computing, and on-demand services may help businesses get a better handle on costs. Server virtualization "will be a major catalyst for attacking energy costs and really making people rethink data center design approaches through the end of the decade," Forrester Research analyst Frank Gillett says.

Autonomic computing--systems that can adapt to changing circumstances, correct mistakes, adjust to shifting workloads, and fix problems without human intervention--has been more hype than reality. Cisco Systems, EDS, HP, IBM, and Sun have programs and products intended to simplify management across the enterprise, but none has come close to a self-managing, self-healing system.

"Autonomic computing is an interesting long-term vision, but it's so far out that it's really hard to argue with," says Shane Robinson, chief strategy and technology officer at HP. "We're more interested in the interim steps leading up to a vision where the focus is on service-oriented architectures and grid computing."

Whether the answers come from from vendors or university research, one thing is clear: Business technology managers must improve the way they manage their data centers and reduce the costs of routine operations and maintenance. That's the only way to shift the spending focus from ongoing maintenance and operations to innovation.