DevOps Demands You Fix Twice

My friend and colleague Richard Miller, CTO of genealogical search engine Mocavo, likes to say, "Fix everything twice." When I first heard him say that, I didn't quite understand what he was getting at. Why not just fix a problem properly the first time so it doesn't come up again? He replied that it's just like carpentry -- measure twice, cut once. Richard was starting to mix metaphors, and he lost me.

What Richard's talking about are DevOps and lean startup principles. Organizations today -- especially startups -- are moving too fast to get everything right all the time. Add to that how quickly technology is shifting, and Richard's "fix twice" adage makes perfect sense, even for enterprises.

DevOps is an extension of Agile that involves the rest of the organization in rapid iterative releases, focused on driving increased customer value. Quicker turns on the product mean bringing new capabilities -- presumably features customers are asking for -- to market faster, fostering a competitive business. DevOps uses a more integrated team environment and automation to deliver on accelerated product releases

One result of iterating often is that organizations don't feel the need to build far beyond their scale, because they know that they can come back in the next sprint and up the ante. Compare that to the old model, where you have only a couple of shots to get things right. You can't really go back and attack shortcomings after the fact. Shorter release cycles mean, in general, that you can adjust quickly.

All that said, when product releases really get rolling, some things are going to break. Fortunately, DevOps is about things breaking. If you've taken time to get everything just perfect, you haven't truly adopted the rapid iteration (or lean product development) ethos.

If your organization can build and release 12 times in a year, will you have a better, more useful product than if you had released only at the end of the year? In almost all cases, yes. The organization that can release more quickly will win. It will be able to adjust based on feedback so that it's building the product that most closely matches what customers are looking for. Additionally, it's going to hear about problems from customers continually versus waiting to find out how the product worked, or didn't.

This is where "fix twice" comes in.

In a DevOps culture, IT teams are moving fast. Solutions to problems need to happen just as quickly. An application or site goes down? Put a Band-Aid on the problem and stitch it up later. That's fix number one -- quick and dirty. The goal is to stabilize the issue with the least amount of effort. The team is generally working under pressure and duress. It's difficult for those involved to think about anything other than getting things operational again.

[If you realize that mobile security means more than ensuring users don't download malware-bearing games from the Android store, take our 2014 survey and enter to win a 32 GB Kindle Fire HDX.]

The real magic happens in fix number two. This is where the team goes back, studies the problem, and takes action to fix it permanently -- or as close to permanently as is practical at the time.

A DevOps team under pressure to rectify a major issue doesn't exist in an environment conducive to strategic evaluation and problem solving. By putting a patch on the initial problem, you bought breathing room. Fix two gives the team an opportunity to find the best long-term solution.

People can debate different approaches, spike on the problem, even try a few different tactics before settling in on the right long-term strategy. Ironically, a team that has experienced failure can better evaluate the problem and focus on creating an ideal, longer-term solution than one that is trying to solve a problem in advance.

Perhaps the most significant challenge with this approach is that most organizations don't have the discipline to get to fix number two. Fix number one took the pressure off. As long as the Band-Aid stays stuck, there's little incentive to circle back. With DevOps and IT folks stretched so thin, it's often difficult to pick up our heads and think about longer-term problems.

Unfortunately, with technology, by the second or third time a problem rears its head, the situation is generally pretty dire. It will take a lot more than Band-Aids to keep things together.

The best DevOps organizations don't let the same problems recur. They schedule time during the sprint cycle to permanently address core issues, and as a result have longer-term fixes, which translate into greater stability.

So while Richard likes to quip "fix twice" often, the real message is not just about solving specific problems. It's about building a methodology and inculcating the discipline to execute on that method. DevOps is at the core of this approach, as is the ability to move fast. For IT organizations that want to be the best, fix failures twice.

Female IT leaders attending the InformationWeek Conference can join InformationWeek.com Editor-In-Chief Laurianne McLaughlin and Rebecca Kaul, President of the UPMC Technology Development Center, for a peer networking breakfast. Then join your peers at our Interop Women in Technology Panel & Luncheon for an open forum to discuss how to advance in an IT organization, keep your skills sharp, build fruitful relationships with colleagues, learn effective dispute resolution techniques, and build a mentoring network. Space is limited to 50 participants.