Revisiting Disaster Recovery and Business Continuity

When a major disruption occurs, whether is a hurricane, an earthquake, a war or a pandemic, organizations think about their disaster recovery and business continuity plans. Worse yet, if they actually do find themselves in disasters, they have to start executing their plans.

What organizations want from a DR and business continuity plan is two things: 

Few companies achieve both objectives. This is ok, because no one really wants to be performing a “real life” DR and business continuity exercise ever.

However, the concerning fact for must businesses is that their DR and business continuity plans are neither up to date, nor comprehensive enough.

Five years ago (2015), over half of businesses with 300 or less employees believed that a disaster recovery plan wasn't needed, two-thirds of companies didn't carry business continuity insurance, and only 37% believed a formal disaster recovery plan was needed. Three years later in 2018, 75% of smaller businesses had no formal disaster recovery plan. Large enterprises are better prepared, but even in these enterprises, trying to keep DR plans current as business and circumstances change can be a struggle.

Keeping your plan up to date

We have only to look at the COVID-19 crisis to see how business has changed.

More employees are working remotely. That means more demands are being placed on internet, corporate networks and cloud-based resources -- with a move away from central offices and data centers. We’ve also seen a movement of more IT resources and services to the cloud, and more business activity moving to online e-commerce

These changes in IT and business processes require updates in the DR and business continuity plan, if the plan is to synchronize with IT and business workflows.

Strengthening security and networks

Moves to remote offices and e-commerce require robust internet, network and software security and failover. All are critical areas that historically have been overlooked in DR plans, which have tended to focus on data center hardware and software.

For internet and corporate communications networks, more than one network should be available for overflow. If a network fails, it should able to failover to another network without a service interruption. If outside vendors are used to host networks (e.g., commercial Internet providers), you should have more than one vendor.

Internet and corporate network communications should ideally also route through different geographical zones. This easily facilitates failover should a disaster occur in one geographical area, but not in another.

Security should minimally be two-factor authentication, with data encryption where warranted. Internal networks, especially at the edges of enterprises, should be trusted networks.

Your entire internal and external network topology should be documented in an appendix in your DR plan. This master schematic can be used by personnel in time of a disaster to assist with rerouting and reconfiguration of communications.

It should also be noted that failure to keep the internal and external network schematics updated is one of the major fail points of disaster recovery plans. The DR plan network schematics should be updated every time a new network is added, or a network revision occurs.

Dealing with personnel substitutions and staff issues

I was working with two New Orleans companies when Hurricane Katrina struck the area in 2005. Many companies lost everything. Some that were able to activate wireless communications managed to remain open, but none were prepared for injury and loss of life to key employees.

In one case, a system programmer lost his life. When the company activated its disaster recovery plan, there was someone to take his place, but not at the same skill level, so recovery took longer. More importantly, there was significant grief experienced by staff that impeded productivity. Mental slip-ups occurred that delayed recovery.

What I learned was that you can “get by” with an employee who takes over responsibilities at a lesser skills level -- but it’s  much more difficult to manage grieving employees who are unable to do their best because of the emotional toll that a situation like a teammate’s death is taking on them.

From a disaster recovery standpoint, compensating for less than optimal performance by staff -- should be on the radar. If there is a grief experience that staff is going through, grief counseling and mental health steps should minimally be part of post-disaster protocols.

Coordinating IT and business processes

Best-of-class disaster recovery and business continuation plans cover both IT and the business.Unfortunately, in many companies, the emphasis for recovery is on the IT, and not on business operations.

This is a mistake.

To illustrate, if your company is a financial institution and your core banking system goes offline, the tellers in bank branches still have to transact business with customers. In some cases, manual ledgers of transactions can be maintained and then later posted to systems when systems come back online. In other cases, there are transactions that simply can't be done without the system. The tellers in the business have to know which transactions can be done manually (and how) and which transactions have to wait. Procedures in the DR plan for how to conduct banking during a disaster help them do this and go far to keep customers happy and confident in the bank.

Paying attention to recovering business operations as well as IT matters, because even if you're experiencing problems, customers and stakeholders want to feel that you're in control of the situation and that you'll get it fixed.

Coordinating communications

Continuing with the bank-teller example, it doesn’t do a bank any good if its IT disaster recovery execution is flawless when employees are telling customers, “The system is broken,” or “All our IT is down.”

I know of one bank where this happened. The bank spent the better part of one day trying to calm the press and assure customers. Apparently, a local media outlet had gotten hold of one teller’s comments-about a “broken system.” This caused customers to flock to the bank, asking to withdraw their money.

To prevent this from happening, senior management, the marketing and/or corporate communications department, IT and the end business should have a plan as to who communicates disaster status to whom -- and who is the “single voice” that communicates to the public.These procedures and policies should be written into the DR plan. Without a formal communications plan for disasters, confusion and misinformation can result and can fuel an even more impactive disaster than a system outage once the rumors about a business get rolling.

Work with vendors

There is one final and critical point about disaster recovery and business continuation: Your company must have the cooperation of your vendors in any DR circumstance.

Before you sign with a vendor for any type of product or service, the vendor should be vetted for its disaster recovery plan. Does it have one? How often is the plan tested? Is the plan certified by an outside auditor or agency?

If you are using a cloud services vendor, you should also insist that an annual disaster test be conducted for your applications hosted by the vendor.

If a vendor is reluctant to meet these criteria, you are probably best served by another vendor.

Read more about DR/BC on InformationWeek:

Developing a Continuity Plan for the Post-Coronavirus World

Rewriting Disaster Recovery Plans for the Edge

Ease Disaster Recovery Drama with Communication Plan