How Did T-Mobile Suddenly Recover Unrecoverable Data?

Microsoft today said it had recovered most of affected Sidekick customers' lost data. But this past weekend, like a doctor issuing a terminal prognosis, T-Mobile told affected customers that their data "almost certainly has been lost." So, what changed between then and now?The statement at T-Mobile's site comes from Roz Ho, corporate VP of "Premium Mobile Experiences, Microsoft Corporation." (Amusingly, there is no mention in the statement of the Microsoft subsidiary, called "Danger," that runs the Sidekick data service under contract with T-Mobile. Wonder if Microsoft's damage control experts have murdered the Danger name and buried it forever?) Ho wrote: "We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data."

This is a complete reversal from the finality of T-Mobile's statement posted on Saturday, in which it told customers that if they were among those who had lost data, forget about getting it back.

In an earlier blog, readers joined in my astonishment that Microsoft wouldn't have back-up systems in place to make data recovery possible. But, now we know there were systems in place. So, why did they originally say they couldn't recover the data? Here are a few ideas:

#1 Microsoft/Danger-um, I mean-Microsoft knew it could recover the data all along, but didn't want to put the time and expense into doing that. Or, T-Mobile didn't want to pay its data services contractor, Microsoft, to put in the time and expense of doing that. But after the big backlash about the outage in the press, decided the initial decision was the wrong one.

-or-

#2 There was a communications error. When T-Mobile issued the statement on Saturday, it didn't really know what was going on, and thus, did not realize the data was recoverable. In that case, the use of such finality in the language-your data "almost certainly has been lost"-was the faux pas.

-or-

#3 Those in charge at the Microsoft (Danger!) data center on Saturday didn't know the data was recoverable. They didn't know, or didn't think about, rebuilding the system "component by component," and thus told T-Mobile to tell its customers to expect the worst.

Those are just a few ideas. Those of you who work in data centers and are more familiar with the details of backup and recovery may have better ones. Feel free to share.

Nonetheless, this situation has highlighted for both businesses and consumers, that if you are going to have your data managed "in the cloud" by another party, don't take anything for granted. Cloud vendors should present you with a specific and solid data recovery plan, and reliable, consistent communications-no matter how many vendors are involved--if customers are affected by system failures.