Payback Time: Making Sure ROI Measures Up

When it comes to measuring return on investment of IT projects, Joe Seibert believes in sticking to the numbers. But convincing others hasn't always been easy. When Seibert joined entertainment and media company Viacom Inc. last year as CIO and introduced his method for calculating ROI, it went over about as well as a box-office bomb.

Seibert, who was CIO at CBS Corp. when Viacom acquired the company in May 2000, had developed an IT performance-monitoring process he calls Dashboard, which drew upon his experience as an accountant, software developer, and consultant. But Dashboard's success at Viacom-also parent to Paramount Pictures, Showtime, MTV Networks, and Infiniti Broadcasting-depended on the support of business managers, who were dubious that the process was anything more than busywork. "I smiled when people resisted," says Seibert, figuring others would eventually find value in the results.

Dashboard is now a blockbuster at Viacom. Line managers and divisional CFOs like the way it's proving the value of IT projects at a time when the $25 billion company is closely watching costs, including an IT budget of several hundred million dollars. Dashboard treats IT investments as a portfolio to be planned and managed before and after purchases are made. The process involves synchronizing accounting data from Viacom's J.D. Edwards & Co. system, items such as staff and materials costs, with IT projects tracked by Microsoft Project software. The data, describing the detailed financial impact in terms of cost, timeliness, and anticipated value of each IT project, is presented in an Excel spreadsheet to CFOs on a quarterly basis. It also provides business and IT managers who champion projects with monthly updates on how each investment is performing. "It keeps us focused on the highest-value projects," Seibert says.

Seibert doesn't recall killing a project because of the results, but when Dashboard flags an investment that's not delivering expected ROI, managers can intervene. So far, Viacom's money managers "aren't challenging the value of our investments," he says. Far from it. "MTV Networks has always partnered with IT to ensure that business needs are met in a timely fashion," says Mark Rosenthal, president of MTV Networks. With the new processes, Rosenthal says, he can better prove the value being delivered to his division.

Few companies have a system for tracking ROI as large-scale as Seibert's, although in the frenzy to cut costs and justify every penny spent, many CIOs are returning to tried-and-true metrics for determining ROI. Approaches and results vary widely, but ROI has re-entered the lexicon of senior executives in a big way, and it's putting pressure on IT to prove the validity of projects. Of 200 IT managers who participated in an online InformationWeek Research survey last month, 80% say the importance of measuring return on investment has increased since a year ago.

Traditionally, ROI goes beyond simple payback, or determining when the investment will pay for itself. It uses metrics such as net present value and the internal rate of return-which consider the value of money invested over time and the cost of the company's capital-to compare the cost of implementing an IT project with the financial benefits it provides. Depending on the project, those results can be operating-cost reductions, revenue growth, or both. The benefits may be evident in months or could take years.

It's about time boardrooms increased scrutiny of IT spending, says Howard Rubin, executive VP and research fellow at research firm Meta Group. "Companies need to start treating technology as an asset."

Lowell Robinson, CFO at Hotjobs.com Ltd., which is in the process of merging with Monster.com, says tough times call for tough measures. "We're looking at shorter-term payback rather than long-term payback because of the economic climate," says Robinson, who has the final say on any major expenditure from the company's $12 million IT budget. "You have to balance capital investments with business needs. If it's a cost-reduction project, it's going to have a much higher profile than a revenue-generating project."

Although the resurgence of ROI may be the logical result of today's economic uncertainty, careful cost-benefit analysis of IT projects can be a jolt compared with recent years, when money flowed freely to IT investments, particularly those involving the Internet. Strict adherence to ROI was frequently ignored as businesses worldwide spent a total of $2.2 trillion on IT last year to ensure that they wouldn't be left behind in the technology and Internet revolution, Meta Group says.

Sharp Microelectronics of the Americas, for example, planned to monitor ROI after implementing content-management software from ePrise Corp. less than two years ago, but never got around to it. "We've been so busy trying to roll things out, we haven't had the time and resources to sit down and evaluate how well we've done," says Don LaVallee, the Camas, Wash., company's director of strategic business and IT. LaVallee relies on a gut feeling that the software has paid off.

Measuring IT through soft or intangible benefits has many proponents, but generally, gut feelings just aren't enough anymore. Twelve to 18 months ago, many of the 5,000 customers of Candle Corp.'s consulting service "wanted to quickly add functionality and capability in the drive to get to E-anything," says Mike Caruso, VP of worldwide customer support. But as with Hotjobs, the emphasis now is on smaller projects, shorter cycles, and a rapid return on IT investment. "They're looking for returns in 90 days, six months, two years' time," Caruso says.

Hurwitz Group is also finding that clients are clamoring for more traditional measurements. A year ago, businesses were interested in a soft-dollar payback concept that Hurwitz called return on opportunity. "Then, all of a sudden, everyone cared about ROI," analyst Jasmine Noel says.

ROI also has become a buzzword among technology vendors eager to prove that their products and services are worthy of a chunk of IT budgets. But Nancy Tripp, VP of SunTrust Banks Inc.'s solution center in Atlanta, doesn't trust vendor-provided ROI figures. "Vendors will put the most positive or most aggressive return rate associated with their software," she says. "But as the project manager, I put the most conservative return rate." Among InformationWeek Research survey respondents, 38% say they aren't influenced by vendor ROI claims, while 54% are somewhat influenced. Only 8% are guided by what vendors claim.

Also emerging are new tools intended to simplify ROI calculations, but these may not add up. "People are looking for a silver bullet, and ROI tools aren't the silver bullet," says Rick Davidson, a CIO partner at the Feld Group, a CIO-for-hire consulting firm. Davidson left equipment manufacturer Case New Holland in Racine, Wis., last year after three years as its CIO. "An ROI tool is too abstract; it's not specific to your business," he warns.

That's not to say technology can't help measure ROI. When Seibert was at CBS, he bought ad-tracking software that promised to cut staffing needs at two television stations. After implementing the system, Dashboard flagged an increase in overtime costs instead. CBS discovered that people weren't using the software properly and were working longer hours because of it. CBS had to spend more on training, but ultimately rolled out the ad-tracking software at additional stations because it saved money.

Now, Viacom is developing an extranet that will distribute digitized marketing materials to affiliates at the cost of about $1 million, and Dashboard will monitor its progress. The extranet is expected to save $600,000 a year by eliminating fees paid to suppliers of paper-based marketing materials, paying for itself within two years. As with all IT investments, if the monthly Dashboard reports reveal the extranet is off its projections by more than 10%, managers will re-evaluate it.

Many IT managers do the best they can to show fiscal responsibility, especially in today's economy. Top management is "expecting deliverables at less cost, and they want a return earlier," says Jane Aboyoun, who oversees a $75 million IT budget as CIO of TMP Worldwide Inc., a $1.3 billion advertising and recruitment company that also owns Monster.com. "They want to see benefits within months, not years." It's more difficult to get approval on "a massive project where we can't see results for five years," she says.

That's why Aboyoun now proposes prototype projects before making any significant investments. In the second half of last year, the company built a prototype of a data warehouse that salespeople could access to learn customer buying patterns to boost cross-selling among TMP's various businesses. When the initial prototype, used by a limited number of salespeople, hiked sales almost immediately, the project was expanded earlier this year.

The challenge is trying to balance the need for financial accountability with the rapid pace of change and the reality that some things just can't be easily measured. Some admit they've spent millions on Web-based networks, online ordering systems, customer-relationship-management applications-and still haven't seen what would traditionally be considered a strong ROI. But that's OK.

Businesses won't abandon investments that are demonstrating soft benefits, such as happier customers and a broader reach around the globe, even though project managers can't easily prove immediate cost savings or increased revenue on a spreadsheet. Soft ROI typically falls into the areas of improving customer satisfaction, increased employee productivity, and a better competitive edge, all of which are vital to businesses.

"A lot of what we're doing in IT is being done for the first time, so original estimates and predictions are a dartboard," says John Zarb, CIO at Libbey Inc., a $400 million maker of glassware, flatware, and china. "As long as your company's earnings per share keeps growing, how important is it to go back and study detail?"

Nearly half the companies surveyed by InformationWeek Research agree. They place equal weight on ROI and intangible assumptions when evaluating IT investments. Even though most respondents cite a stronger focus on ROI, only 24% adhere to formal ROI measurements on IT projects. Thirty percent use a combination of formal and informal measurements, and 35% primarily use informal ROI measurements. The other 11% say that ROI measurements don't figure in at all.

Amid the dot-com craze of the late 1990s, the Bank of Montreal spent upward of $69 million on online banking initiatives, yet ROI wasn't a major consideration. "The driving force at that time was more strategic. We needed the right infrastructure to compete," says Charles Piermarini, senior VP for venture creation in the E-business group at the Toronto bank, Canada's fifth largest.

But with the online infrastructure now in place, E-business initiatives must pass muster from both ROI and strategic business perspectives. For instance, through its Harris Bank subsidiary in Chicago, the Bank of Montreal this past year invested about $5 million in eScout, an E-marketplace that lets small and midsize businesses buy nonstrategic supplies and equipment as well as human resources and recruiting services. For the most part, the Bank of Montreal won't make E-business investments unless it believes the value will double every four years and the payback will take about two years, depending on the project. The investment also must exploit the bank's core capabilities. Piermarini says eScout, which can be accessed through Harris Bank's home page, lets the bank cross-sell products, such as loans to finance purchases made through eScout. In addition, eScout strengthens the bank's relationships with commercial customers by helping them run their businesses. "ROI can't be the sole driver for all our investments all of the time," Piermarini says. "We're not a venture-capital group."

Others report similar experiences. Consolidated Freightways Corp. spent up to $100 million annually on IT in recent years and invested millions in a new Web infrastructure from Sun Microsystems and iPlanet about 18 months ago. The Vancouver, Wash., company applied its broad set of standard ROI metrics before the investment: What would be the impact on revenue, business processes, transactions, and associated cost reductions? CIO Martin Larson says the $2.3 billion freight company operates on razor-thin margins and has used very traditional ROI calculations to measure IT investments.

But to some degree, that thinking went out the window when it came to the Internet. "Being in a new environment and a new business model caused us to look at expenditures as a cost of entry because of where we saw great potential and shift in the market," he says. Improved customer relationships play a significant role in the company's Web strategy, yet it's the most difficult to measure. Says Larson: "Everyone said they'd use [online ordering and tracking], but would they actually use it?"

Nearly two years later, Consolidated Freightways monitors its Web site to see how customers use it, but Larson still doesn't see the Web infrastructure delivering what would be considered impressive ROI on a pure revenue or cost perspective. "I can't tell you that for every $1 we've invested, we've returned $1.50," Larson says. Nevertheless, he considers the investment a victory in terms of keeping Consolidated ahead of the technology curve and gaining customer approval. "We would do it all over again," he says.

Even with the best of intentions, ROI forecasts can flop. Late last year, $100 million writing utensils manufacturer Dixon Ticonderoga Co. went online with an ordering system that can save the Heathrow, Fla., company $11.50 every time a customer uses it instead of the paper-oriented system. Based on a customer survey, CIO Garrett Grainger estimated that 50% to 60% of the customer base would use online ordering-but he was wrong. Only 15% to 20% actually use the system, since most mom-and-pop shops decided they're more comfortable with the fax or telephone. Although the project cost $45,000, the estimated ROI proved to be based on faulty assumptions.

For his part, Robert Church also learned a thing or two about ROI since embarking on a CRM strategy three years ago. Church, the IS manager of electronic data interchange at Sierra Health Services Inc. in Las Vegas, compared the downtime of the $1.4 billion company's existing TeleMagic sales system with reliability guarantees offered with a new CRM system from Onyx Software Corp. The old system was down for 20 to 40 hours every month, Church says. "Every day the system was down, we were deferring revenue, postponing a sale," he says. Salespeople set about 10 appointments an hour, and, of those, some 30% result in sales. That translates into $2.2 million in annual revenue that's "postponed opportunity." The Onyx system offered a service-level agreement guaranteeing 99.5% uptime. Since deploying the system in 1999, uptime has been consistently 99.8%, which amounts to only three hours of downtime a year.

Church also determined that his sales representatives were spending about 30 hours a month reporting on their activities and generating sales reports. He expected CRM software to automate some of that. "We said conservatively, at least five of those hours would be spent selling," he says. If the company could give 80 sales representatives an additional five hours every month to sell product, he estimated, sales would increase approximately $800,000 over five years.

Those figures all looked good on paper, but while uptime held up, the five hours of extra sales time didn't. Another wrinkle came shortly after deploying the system: Sierra bought another insurance company. The resulting sales attrition "threw our numbers off," he says. In addition, Sierra had to spend time integrating product lines, training employees, and deploying the newly acquired facility.

On the flip side, the negatives were offset by some unforeseen benefits. With the CRM system, sales employees entered data on prospective clients and competitors. When one of the company's largest competitors went out of business, "within an hour and a half we had 7,500 leads, and we closed sales on about half of those," Church says. Those sales alone paid for the Onyx deployment. "You've got to continually monitor ROI from your systems, because as the business changes, so does the ROI from any given software package," Church says.

Another way to consider IT is as a portfolio of projects. Some organizations, such as PNC Financial Services Group, Snap-on, and the University of Nebraska, find value in a new breed of tools designed to treat IT assets like those found in investment portfolios, much the way Viacom does (see sidebar story, "Portfolio Management Tracks Health Of University Projects").

Yet Viacom's Seibert says managers shouldn't view IT portfolio management as a way to validate the worth of projects, but as a way to align IT with the primary goals of the company. "We don't use it as a method of justifying why we need to spend money," he says. "We use it as a method of knowing where to invest money to keep our company growing. If you want to be a mature technology organization, you need to know you're doing the right thing the right way."

Hard-liners such as Meta Group's Rubin say companies still aren't doing enough to prove their investments' worth. He calls ROI "a justification mechanism rarely followed up on." Most companies aren't continuing to measure ROI for months and even years after a project's implementation, he says. Among InformationWeek Research survey respondents, three-fifths say they continue to measure and monitor ROI after technology is purchased, including 26% continuously and 23% on a weekly basis.

But more extensive post-audits may not be practical for all companies. Libbey CIO Zarb, who oversees a $10 million IT budget, says post-audits are a luxury few companies with his staff or budget can afford. And as Consolidated Freightways and Dixon Ticonderoga found out, it can be difficult to predict what customers and business partners will do.

So how do you measure the ROI of IT? Companies should set some concrete expectations for IT investments before approving them. Also, IT portfolio management is a smart, forward-thinking method for aligning IT with the goals of the business. It offers documentation on why some IT projects are working and guidance about when to cut loose or adjust those that aren't. But ROI approaches also have their limits, and flexibility is key.

Even Seibert, with his strict views of IT portfolio management, says soft ROI can be a strong consideration. If a project is intended to provide a strategic value, then cost and revenue ROI may take a backseat. Consolidated Freightways' Larson says competitors who were the last to get on board with online shipment tracking are paying the price through lower customer satisfaction. "For ROI, we look at the hard and the soft [data], even if our finance guys don't want to hear that," Larson says. "Looking back, we did a lot of things right."

--with Eric Chabrow, Diane Rezendes Khirallah, Jennifer Maselli, and Christopher T. Heun