With little progress by late November, Austin Energy general manager Larry Weiss appealed directly to Palmisano on Nov. 22, 2011. "Instability issues have been substantial and continue to have serious and costly impacts to our business and our customers," Weiss wrote. "In short, the City of Austin cannot properly operate its utilities. We are losing revenue." Weiss said system errors meant Austin Energy couldn't bill apartment residents for water, balance its books, or file audit reports.

"We have tens of thousands of customers who have required one-on-one assistance either in accessing their accounts through the portal or in addressing account difficulties resulting from system instability or error. This results in substantial detriment to customer relationships, our reputation, and work efficiency capability," Weiss said. "My concern is the lack of progress in identifying the root cause and, frankly, the pace and quality of response from IBM."

Shortly afterwards, Austin Energy officials voiced concerns that some of IBM's proposed fixes, such as adding more server capacity, would mean that the utility would have to pay more than it originally planned on the project. "IBM implies there will likely be additional cost to the City as a result of IBM's decision--as part of the recovery effort--to move to more powerful servers," wrote Austin Energy's Kerry Overton to IBM's Chris Lilley on Nov. 23. "We do not support additional cost for the replacement architecture under the fixed price contract."

Soon after, Frank Kern, then head of IBM Global Business Services, wrote to Austin Energy GM Weiss to inform him that Palmisano had handed him Weiss's complaints. "Sam Palmisano has asked me to respond to the concerns raised in your Nov. 22nd letter," Kern wrote on Dec. 8. "I first want to assure you that the IBM team is equipped to handle the issues that typically arise in complex billing solutions like the one at Austin Energy, and we intend to meet IBM's contractual commitments."

Kern outlined five steps IBM would take. They included "improving communications on business impacts caused by known defects," ensuring that all outstanding issues were documented and assigned to the right people, implementing "best practice processes to ensure repeatable success," working more closely with third-party vendors such as Oracle, and "identifying any remaining gaps that are outside the project scope and providing recommendations."

Kern announced his retirement earlier this year as IBM announced management changes under new CEO Ginni Rometty, who succeeded Palmisano on Jan. 1. Rometty tapped former North American sales leader Bridget Van Kralingen to head Global Business Services.

Claypool wrote to Van Kralingen earlier this month, complaining that little had changed since Kern proposed his remediation program: "The City of Austin continues to sustain substantial financial and operational losses resulting from two multi-million dollar procurements in which IBM has failed to provide the fully functional systems required out of three major system integration contracts we have with you."

Claypool was referring to the billing system and also to an inventory management system IBM was supposed to build for the city. "More than a year later, the City is still suffering from serious, continuing issues in getting the system running and using it to conduct routine City of Austin business," Claypool wrote.

As for the billing system, Claypool told Van Kralingen that Austin would hold back $3.8 million in payments to IBM until the situation is righted. "We are extremely disappointed and continue to have serious concerns about the quality of service we have received from IBM to date, which could impact any future contracting procurements," Claypool wrote on Feb. 10 of this year.

In an interview this week with InformationWeek, Claypool said that, despite his harsh messages to IBM, he hasn't given up on resolving things amicably with Big Blue. "They have been working with us to provide a solution that meets the requirements," Claypool said. "They have been responsive. I would characterize it as, in their words, 'incremental response.' We would like to see a faster response."

Claypool said there's no single root cause for all the problems with Austin Energy's billing system. "It's very complex. It's got 73 different interfaces," he said. "The teams have been working very hard through nights and weekends to figure out all of these complexities and integrations, to make them work with all the systems."

Claypool was not Austin Energy's CIO at the time the utility drew up the contract with IBM. In the future, the utility's outsourcing contracts will include much more stringent penalties for non-performance, he said. "The penalties around specific areas like system availability need to be much stronger," he said.

An IBM spokesman would say only that "IBM is working with Austin Energy to resolve the situation as quickly as possible."

It's time to get going on data center automation. The cloud requires automation, and it'll free resources for other priorities. Download InformationWeek's Data Center Automation special supplement now. (Free registration required.)