The Perils of Patching: How to Minimize the Risk

It’s every IT pro’s nightmare.

A massive outage at the cloud computing service provider Fastly essentially broke the Internet, taking down Amazon, Reddit, and countless other websites worldwide for more than an hour. Ironically, the outage wasn’t the work of hackers. It was triggered by a bug in a software deployment that was activated when a single customer changed their configuration settings. Fastly quickly restored service, but the massive disruption quickly rippled through the industry.

This disastrous incident shines the spotlight on an issue that is often overlooked: the need for greater rigor in rolling out software updates and ensuring patch management doesn’t accidently introduce a risk.

Software vendors regularly patch their products to address issues and make their programs more usable for customers. The problem is that they don’t always follow strong quality procedures to ensure that updates won’t introduce new, potentially catastrophic, problems. What does this mean for you if you’re one of their customers? It means you must proactively do all you can to protect yourself.

How to Limit Third-Party Risk

Increasingly, enterprises are at the mercy of the software vendor, as evidenced by supply-side attacks involving SolarWinds and Kaseya. From an IT management perspective, that’s both a blessing and a curse. If something goes wrong, you’re powerless (but it’s a relief to know it’s not your fault).

Still, it’s a net loss when it impacts business. The good news is that while much of the update process is beyond the control of the customer, there are some fairly simple things you can do to minimize vendor risk:

Make Sandboxes Part of Your Change Management Strategy

It’s inevitable that mistakes will happen at some point, and that’s why putting a formal change management process in place is invaluable. Recognize that today, updates can contain malicious code. The question becomes, how do you kick the tires to make sure a software update does what it is supposed to?

First of all, make sure you regularly back up your critical IT infrastructure. This way, if a third-party vendor’s bug impacts you, it will be possible to restore your entire IT environment quickly. The ITIL framework offers a good change control mechanism and is a good starting point for most companies looking to implement change management.

Every time you make a change, you should be able to rest easy knowing you can roll it back if something goes wrong. Here are three strategies to consider:

When it comes to software updates and patching, security must be front of mind. It’s key. On the vendor level, you have to feel confident that the software companies you work with will quickly identify issues and can restore operations to pre-patch levels for their customers. On the customer level, it’s incumbent upon all of us to limit supply chain risk and protect our businesses as much as possible. Remember, once an environment is compromised, it’s like dominoes. There can be no end to the issues, and the fallout can be catastrophic. A cautious approach to rolling out updates and software patches may slow things down -- but sometimes, that can be a good thing.