Stop Software Project Killers

When it comes to enterprise software projects, I'm a pessimist. Most major software projects miss their timelines, don't deliver all their promised functionality, and run over their cost estimates--but not seriously. What scares me are the ones that go very wrong, blowing out their budgets by 400% or more while delivering little or no value. And we've all experienced them.

They can be a job killer, so it's critical for IT leaders to structure them so that they can detect and correct problems--or pull the plug--early on. A common refrain amid runaway software projects is that the senior-most IT officer, juggling too many projects at once, didn't know things were going very wrong until it was too late. Therefore, the first thing to do is make sure your time is spent on the right projects.

During my time at FedEx, every critical enterprise software project was assigned to an officer, typically a VP who was responsible for its success, and that person reported to me as a senior VP. Each one of these high-risk projects also had a program/project manager and a technical lead.

On the highest-risk projects, the program and technical leads reported directly to me instead of to the VP. The objective was to get at least three independent views of how the project was progressing. (The traditional structure of having the program and technical leads report to the officer in charge can lead to information being filtered by the responsible officer.)

[ See 20 Innovative IT Ideas To Steal, for winning advice from this year's InformationWeek 500 honorees. ]

The program lead is important in this structure. On a large project, typically there are numerous project leads working for several other officers. Pulling the plans together, pushing the teams for accurate status reports, and assessing which areas are having trouble is a huge task. Having the program lead report directly to me gave that person the implied authority to challenge information.

Testing the software is yet another opportunity to bring in an independent perspective and reduce risk. Several years ago, I converted an application VP position to a testing VP position and formed an independent test organization. My intention was to put testing on the same footing as development, improve the quality of critical software, and give me another voice in a project. That VP also reported directly to me.

It's also critical to structure the project to reduce technical risks. Bridge engineers don't design and build a bridge and then test it by driving trucks across to see if it falls down. Likewise, don't build or install software before identifying all the technical risks and fixing problems.

When building an integrated database called Customer Fusion at FedEx seven or eight years ago, we came across numerous technical issues that had to be resolved before development could begin safely. Chief among them: Could the messaging layer perform at the required transaction rate?

The technical leader was crucial in identifying the issues, designing and overseeing the test execution, and analyzing the results. With Fusion, the test bed was constructed in the hardware vendor's lab using real data. The plan was to take the messaging layer to the required performance, take down some of the processors to test the failover, and perform other scenarios. The vendor's products failed, and it took several weeks to fix them. Once those performance issues were resolved, however, the project proceeded on plan and was both a technical and business success.

Global CIOs: A Site Just For You Visit InformationWeek's Global CIO -- our online community and information resource for CIOs operating in the global economy.

If only the technical risks were anticipated so well on every enterprise software project. Nearly 15 years ago, during a business consolidation at GeoLogistics, we needed to replace three accounts payable systems with a new one. My business colleagues wanted Oracle. Our technical experts wanted IBM--the DB2 database on AIX, which were our company standard. Oracle insisted that its AP software worked on both DB2 and its database, but our contractor told me that to his knowledge, no one had put Oracle AP on DB2. I missed the warning.

I knew there were significant differences in how Oracle and DB2 handled read and write locking. I didn't know the Oracle programmers didn't know the differences. When we started installing Oracle AP, some modules failed on load because of severe compile errors. It was a technical mess that forced us to keep writing around the Oracle AP problems, which we would have avoided if I had followed the design, test, build model I outlined above.

Throttles, Switches, and Fallbacks--Oh, My!

From this experience, I learned the value of throttles, switches, and fallback processes to minimize implementation risks.

Throttles, which control the rollout of a new process or product, are particularly useful when moving from one piece of software to another that does essentially the same thing. For example, when a company modernizes its pricing system, it needs to carefully design a scale-up plan to compare the output of the new system with the old in live conditions. I have done many transitions of this type, and in every case we found errors in both the new and old software.

Switches turn a process or product off and on and are often combined with throttles. In the pricing system example, new pricing logic will have a switch that turns it on and off. Several times I've seen new systems go live only to then be pulled from the market immediately. It absolutely becomes an IT issue if a marketing mistake screws up thousands of customer invoices and it takes IT six weeks to code a recovery.

Finally, I require a fallback process to be included in every enterprise software project plan.

The application development teams always complain switches, throttles, and back-out processes add considerable cost and time. Most application teams prefer to "fail forward"--solve problems on the fly. But sometimes forcing dev teams to think about these risk management processes leads them to better designs.

Like everyone else who has made a career of building and installing software systems, I have had my share of difficult and failed projects. But none ever failed spectacularly. The key is to structure enterprise software projects to eliminate all the known risks, and then to structure the team to get as many independent views as possible--and then listen to them.

Dr. Larry Tieman has been a senior VP at FedEx, a CIO, or a CTO for the last 20 years. He has worked with some of the great CIOs, including Max Hopper, Charlie Feld, and Rob Carter. He can be reached at [email protected].