Verizon Cloud Outage Draws Overblown Angst - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Cloud // Infrastructure as a Service
10:45 AM
Charles Babcock
Charles Babcock
Connect Directly

Verizon Cloud Outage Draws Overblown Angst

Verizon Cloud's planned 40-hour outage set off outraged critics, but long-term implications will make the cloud service more stable and more competitive.

Cloud Storage: 8 Ways You're Wasting Money
Cloud Storage: 8 Ways You're Wasting Money
(Click image for larger view and slideshow.)

Verizon Cloud customers were notified that the service was slated to undergo maintenance that would take the service offline starting 1:00 a.m. this past Saturday (Jan. 10) and last for up to 48 hours. In reality, the Verizon Cloud was out of service for about 40 hours over the weekend. After it came back online, Verizon explained that the update's purpose was to enable "seamless upgrades" -- i.e., to prevent the need for downtime when future updates are made.

Verizon's announcement of the planned service outage drew intense flak from analysts last week, who said being down for up to 48 hours was "insanely stupid" for a cloud vendor, according to Rob Enderle, principal of the Enderle Group. IDC's Frank Gens said, "Forty-eight hours seems very excessive." Dan Olds at the Gabriel Group said, "A two-day outage, for any reason, is a very big deal to enterprise cloud customers. I can't recall any cloud outage that is this lengthy."

Well, Amazon's Easter outage in 2011 was closer to three days. I'm just saying... it's not that shrouded in the mists of time. And that was an unplanned, unannounced, and unprepared-for outage, which is different from what Verizon did.

[Does your SLA cover downtime? See Amazon SLAs Didn't Cover Major Outage.]

In the interest of having more than the three cloud suppliers that Microsoft seems to think is the maximum allowed, I harrumphed loudly at the nature of Verizon's customer notice and was glad it did what needed to be done and got it over with.

In the group of big suppliers, Verizon Cloud is the youngest of the Amazon, Google, Microsoft, Rackspace set. It became generally available only in October 2014 after 12 months of beta operation. It represents a fraction of the enterprise cloud customers that Verizon acquired when it purchased Terremark in January 2011. Terremark's customer base represents the bulk of Verizon's Enterprise Cloud customers, who would resent a two-day outage, but they were not affected by the upgrade to Verizon Cloud. They represent the users of a legacy cloud -- as much managed hosting system as self-provisioning cloud -- that Verizon is trying to evolve away from, but wisely has left intact for now.

I disagreed with the redoubtable David Linthicum when he said, "Verizon's enterprise cloud will go offline for as long as two days starting Saturday, Jan. 10" (emphasis added). If anything, the majority of parties affected are consumers, small businesses, and developers, not enterprises. If Verizon actually were cavalierly shutting down enterprise customers, I guess I'd be indignant, too. But the Verizon Enterprise Cloud (primarily the legacy Terremark customer base) was not affected. Nor were Enterprise Cloud Managed Edition and Enterprise Cloud Federal Edition, again part of the Terremark legacy.

In fact, there are some businesses among those customers being shut down, and they were justifiably concerned about being offline. But I suspect they, too, want Verizon to upgrade its cloud operations software, if only to make this sort of maintenance outage unnecessary in the future. That's a primary goal of this upgrade. As I said, by all means, Verizon, get on with it.

Not everyone is as irritated as the foregoing pundits. In a comment on a GigaOm story, "Verizon Cloud warns customers of a 2 day (!) maintenance closure," Carl Brooks of the 451 Group said, "This is a whole bunch of nothing, IMO. I think it's lousy PR on their part but it's not some grand failure of design. I seriously doubt anyone at Verizon is going to mash the big red button on Verizon Cloud until Monday morning. They're telling a small subset of their customers to [get out] while they do some planned maintenance, which is miles better than the usual practices from cloud providers."

In a check with Verizon, spokesman said "less than 10%" of its cloud customers were affected. That probably included managed hosting customers as well -- strictly speaking, not cloud users.

Brooks, in his comment to the GigaOm story, added: "You know what [Amazon Web Services] says to users before it upgrades and reboots a bunch of its platform? Nothing."

Well, it's possible AWS migrates its customers' virtual machines to an availability zone or data center that won't be affected by the maintenance, rather than forcing them to shut down. I haven't heard of AWS staging 48 hours of downtime for a software changeover. So I'd be cautious on that point. The fact that Verizon has had to do so is scarcely a triumph of cloud services marketing.

The commentator that I most appreciated on this issue is the one who sounded the most well-informed on it. Forrester's James Staten, in an email exchange in advance of the outage, told me: "Verizon has been in the IaaS cloud business for several years and with different underlying architectures. Most of these solutions up to this point have been more traditional hosting in design and architecture than IaaS (meaning less self-service, elastic, API controlled, etc.)...

"Their latest, Verizon Cloud, is more the latter but a very new offering. As such there are some elements of the architecture that were not optimized at first release. One such issue is not having the ability to apply firmware changes non-disruptively. This outage will add this capability."

For those of us who want to see viable service providers in the marketplace -- in addition to Amazon, Microsoft, and Google -- this upgrade was the best way for Verizon to get there. The consequences of staging it in this manner are between Verizon and its customers. Post-outage, I'd be surprised to hear from a large group of outraged customers who say they're mad as hell and not going to take it any more (but don't hold your breath).

If Verizon gets to a more competitive position, and customers get Verizon's promised continuous service in the future, then I'm glad they got on with it, and I'll try to restrain my temptation to snipe.

Verizon Cloud offers three types of service.

  • Reserved service, which comes with service level agreements for individuals and companies deploying their workloads to it
  • Elastic services, managed by Verizon and allocated to existing (Reserved) workloads when they need to scale -- and there are resources available for them to do so
  • Research pools, which are basically test-and-dev servers run to a scale as the customer sees fit, as resources allow; no SLAs available with Research Pools

Staten noted: "Of these three resource types, only Reserved is affected by this outage. Granted it is the primary resource type, but most customers have not committed to these resources yet. As you would expect, companies tend to test things out on Research before deploying to Reserved, just as you would in-house by building the app in test & dev, then moving it later to production."

Speaking of SLAs, until recently the best SLA you could get from cloud providers, including Amazon, was a guarantee of 99.9% availability (which was improved over the last two years to 99.95%). Under that earlier standard, up to 21.5 hours of unplanned, unannounced downtime was allowed in any 12-month period, even if all 21.5 hours occurred in one month. The planned Verizon Cloud outage essentially doubled what has been the accepted SLA for unplanned outages

As Staten said in his message: "Given the early state of Verizon Cloud and that the purpose of the upgrade is to prevent these [maintenance downtimes] going forward, I think the concerns being raised are a bit overblown." 

Apply now for the 2015 InformationWeek Elite 100, which recognizes the most innovative users of technology to advance a company's business goals. Winners will be recognized at the InformationWeek Conference, April 27-28, 2015, at the Mandalay Bay in Las Vegas. Application period ends Jan. 16, 2015.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Charlie Babcock
Charlie Babcock,
User Rank: Author
1/13/2015 | 1:22:57 PM
Verizon's entry too tentative
Does anyone remember the ins and outs of what Amazon did three months into general availability of EC2? No, because it was a new service granted a little leeway for shakedown. Part of Verizon's problem is that the standard for cloud computing has risen as it somewhat tentatively stuck its toe in the market. Now it's trying to play catch-up. Future upgrades will occur in background as customers' systems continue running, it says.
Charlie Babcock
Charlie Babcock,
User Rank: Author
1/12/2015 | 7:03:33 PM
Did you hear that, Verizon?
And I fervently hope they don't do it again or I will suffer angst myself. My name will be mud. 
David Wagner
David Wagner,
User Rank: Strategist
1/12/2015 | 6:58:46 PM
Re: Verizon completes upgrade in 40 hours
@Tom- Right. and you have to take them at their word that this update means all other updates won't require this. one assumes as technology moves, it might be required again. That said, planned outages are what they are-- standard practice in IT for decades.
Thomas Claburn
Thomas Claburn,
User Rank: Author
1/12/2015 | 6:34:16 PM
Re: Verizon completes upgrade in 40 hours
I'm inclined to agree that the outage was overblown, but at the same time, it makes me wonder about the viability of cloud services in industries or situations where delays are unacceptable. You couldn't have hospital systems, for example, just shut down for 40 hours without a contingency plan. 
Charlie Babcock
Charlie Babcock,
User Rank: Author
1/12/2015 | 2:33:24 PM
Verizon completes upgrade in 40 hours
The upgrade took longer than the most optimistic predictions, about 40 hours, but less than the 48 allotted. When it was completed, Verizon acknowledged it was staged to eliminate the need for such blackout upgrades in the future:  "All maintenance and upgrades to Verizon Cloud will now happen in the background with no impact to customers," spokesman Kevin King wrote in a blog on the Verizon site this morning.
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Remote Work Tops SF, NYC for Most High-Paying Job Openings
Jessica Davis, Senior Editor, Enterprise Apps,  7/20/2021
Blockchain Gets Real Across Industries
Lisa Morgan, Freelance Writer,  7/22/2021
Seeking a Competitive Edge vs. Chasing Savings in the Cloud
Joao-Pierre S. Ruth, Senior Writer,  7/19/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
Monitoring Critical Cloud Workloads Report
In this report, our experts will discuss how to advance your ability to monitor critical workloads as they move about the various cloud platforms in your company.
Flash Poll