Cerner Hosting Outage Raises Doctor, Hospital Concerns - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Healthcare // Electronic Health Records
02:23 PM

Cerner Hosting Outage Raises Doctor, Hospital Concerns

Cerner's national network went down for a day, forcing physicians to resort to handwritten orders.

11 Healthcare-Focused Business Intelligence Tools
11 Healthcare-Focused Business Intelligence Tools
(click image for larger view and for slideshow)
Cerner Corp.'s remote hosting service went down for several hours on Monday, July 23, affecting Cerner's hospital and physician practice clients all over the country. This unusual outage, which reportedly took down the vendor's entire network, raises some new questions about the reliability of cloud-based hosting services.

Responding to an inquiry from InformationWeek Healthcare, Cerner spokesperson Kelli Christman said, "Cerner’s remote-hosted clients experienced unscheduled downtime this week. Our clients all have downtime procedures in place to ensure patient safety. The issue has been resolved and clients are back up and running. A human error caused the outage. As a result, we are reviewing our training protocol and documented work instructions for any improvements that can be made."

Christman did not respond to a question about how many Cerner clients were affected. But HIStalk, a health IT blog, reported that the outage was national and possibly international in scope.

[ Is it time to re-engineer your clinical decision support system? See 10 Innovative Clinical Decision Support Programs. ]

As for the duration of the downtime, HIStalk conveyed reader reports indicating that it had lasted all or a large part of the day. A typical comment: "We have not charted any patients today. Not acceptable from a healthcare leader." Colleen Becket, chairman and co-CEO of Vurbia Technologies, a cloud computing consultancy based in Tampa, Fla., told InformationWeek Healthcare that NCH Healthcare System, which includes two Tampa hospitals, had no access to its Cerner system for six hours. The outage affected both the facilities and NCH's ambulatory-care sites, she said.

"My husband, an ob/gyn physician [at NCH], called to tell me Cerner was down and could I please find out why," said Becket in an e-mail. "There had been no communication from hospital or Cerner with regards to downtime. They resorted back to handwritten orders during the six hours the system was down."

Another blogger on HIStalk said, "Communication was an issue during the downtime as Cerner’s support sites was [sic] down as well. Cerner unable to give an ETA on when systems would be back up. Some sites were given word of possible times, but other sites were left in the dark with no direction. Some sites only knew they were back up when staff started logging back into systems. Issue appears to have something to do with DNS entries being deleted across RHO network and possible Active Directory corruption."

Doug Hires, a partner at Santa Rosa Consulting, based in Bloomfield Hills, Mich., said the outage of the Cerner network was unusual and surprising. If one of the company's data centers had gone down, he noted, that wouldn't have been as big a problem because the data would have been backed up at another data center.

The federal HIPAA law, Hires explained, requires providers to have contingency procedures and backup plans in case their systems go down, so Cerner's clients were probably prepared for the outage. However, he added, it may have been difficult for providers to go back to writing orders on paper without having access to electronic health records containing patient medical histories.

Hires pointed out that service-level agreements (SLAs) with hosting companies include penalties for downtime. He surmised that if most Cerner customers had SLAs, this outage might prove to be very expensive for Cerner.

Whether a hospital or healthcare system operates onsite clinical and financial systems or outsources them to a remote hosting vendor, the results of downtime for end users would be the same. Nevertheless, Hires observed, the Cerner event shows "the potential and vulnerability is there" for an outage involving remotely served applications. This is a big concern for CIOs with regard to mission-critical information.

"The fear in the industry is about control," Hires noted. "If I have my own procedures and I have access to my own physical gear, I'm less concerned." For example, if an onsite server went down, a hospital might be able to swap it out with a redundant server that mirrored the data in real time.

A recent KLAS Research survey found that 55% of respondents already used remote hosting for clinical applications, storage, e-mail, and/or picture archiving and communication systems. Nearly a quarter of these respondents had remotely served electronic health records.

But another group of respondents were hesitant to have mission-critical applications served remotely, partly because of doubts about the reliability and security of cloud computing, KLAS reported.

In this InformationWeek Healthcare virtual event, EHRs: Beyond The Basics, experts will discuss how to improve electronic health record systems. It happens July 31.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Strategist
8/30/2012 | 3:52:37 AM
re: Cerner Hosting Outage Raises Doctor, Hospital Concerns
I've read a lot of service contracts, and I've noticed that unless you go over them very carefully - both for general content and legalese, you may well not notice that vendors of 'cloud' and data services of various types have all sorts of 'outs' clauses hidden away. These tend to be one way - if the client blows it, heavy penalties. If the vendor blows it, OOPS! So Sorry!

My point is simple; these contracts generally are written by and favor the vendor. If you want there to be good penalty clauses, YOU and your attorney need to make sure they are included. High on the list of issues invoking penalties are outages.

I bank I once worked for had a data service used by at least 20,000 people. The contract called for the vendor to add a second site with separate comms and full, instant switchover capability. They never got around to it, but no one went after the vendor. One day, during a blackout, things were going OK - the huge data center on backup power, when a security guard heard an alarm going off, and he JUST HAD TO SILENCE IT. So he went, found the big red button marked Don't Touch Except In Case Of Emergency, and hit it! The alarms went silent, as did every other piece of equipment on the site, as the backup power shut down. Took the vendor over a week to clean up the mess and get fully back into service.
User Rank: Apprentice
7/28/2012 | 1:56:52 PM
re: Cerner Hosting Outage Raises Doctor, Hospital Concerns
This calls for an investigation to determine the number of deaths, injuries, and near misses amidst the mayhem at each hospital that was affected. Cerner should publish the list of hospitals affected for the FDA, CDC, and the Joint Commission. Are electronic medical records all that safe? Doubt it when millions of patients face delays and hospitals are not equipped to handle it.
2021 Outlook: Tackling Cloud Transformation Choices
Joao-Pierre S. Ruth, Senior Writer,  1/4/2021
Enterprise IT Leaders Face Two Paths to AI
Jessica Davis, Senior Editor, Enterprise Apps,  12/23/2020
10 IT Trends to Watch for in 2021
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/22/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you.
Flash Poll