10 Big Data Migration Mistakes

Transferring data between computer systems or storage formats is never a trivial task, particularly when it involves both structured and unstructured data. The complexity of data-migration jobs means that cost overruns and delays with "go-lives" are all too common, said Arvind Singh, co-founder and CEO of Utopia, a Chicago-based enterprise data solutions provider.

In a phone interview with InformationWeek, Singh outlined 10 common data-migration problems--five pitfalls and five risks--that enterprises should strive to avoid.

Pitfall #1: Failing to engage the lines of business and business users at the outset.
When companies integrate or consolidate multiple systems into one--often after a business merger--they need to identify the right business uses at the outset.

"You need to identify who knows and understands the business data," said Singh. "Who's the subject matter expert in your business? It's certainly not IT or the systems integrator."

In other words, bring the people who'll be using the data into the migration project. After all, they'll be the ones operating the system once it goes live.

2. Absence of data governance policies and organizational structure.
"You've got data being moved from System A to System B, but who owns the governance structure? Who has the rights to create, approve, edit, or remove data from the system?" Singh asked.

[ Now that you have all this data, what do you do with it? Learn 5 Ways To Benefit From Big Data. ]

Other issues that must be resolved: Is your organization set up to manage data? Is there a business process for managing the lifecycle of data? And do you have data stewards in the company?

Pitfall #3: Poor data quality in a legacy system.
Companies often don't realize that an "as-is assessment" is essential before embarking on a data-migration job.

"Understanding the quality of existing data in a legacy system is a huge pitfall that companies often don't spend enough time on," said Singh

Questions to consider: Will the existing data support new users? What is it missing? And what are you planning to do, analysis-wise, that you're not able to do today?

A detailed assessment makes it easier for companies to estimate the amount of work required to migrate legacy data successfully.

Pitfall #4: Neglecting to validate and redefine business rules.
Your company's business and validation rules may not be current.

"It's amazing how little time companies have spent agreeing on a business rule, much less making sure the data complies with the business rule," said Singh. "In other words, you think you have a business rule, but does your existing data match, map, or comply with that rule?"

In addition, auditors need to be sure that data moved from a legacy system to a new system has been validated, especially when a migration involves critical information such as financial, inventory, and payroll data.

Pitfall #5: Failure to validate and test the data-migration process.
Don't save this step for the end. "You really need to make sure that you're validating and testing throughout the process," Singh said.

Questions to consider: How are you going to test the data? Who will test and evaluate it? Who will sign off on it? And who's the ultimate consumer of the data?

This process must be built into the project's lifecycle, but unfortunately companies often "don't spend enough time aligning the data testing, validation, and migration cycles to the project timeline," said Singh. And now for the five big data migration risks:

Risk #1: Employees entrusted with a data-migration project lack industry best-practices experience.
A organization's employees may be very good at what they do, but that doesn't mean they're experts in data management, migration, and governance.

"They are creators and consumers of data, but they're not fully familiar with best practices on tools, processes, services, templates, and accelerators," Singh said.

Risk #2: Your team relies too much on the tools of the job.
This problem is often the result of staff inexperience. A data-migration project often falls in the lap of IT, which may not be properly trained to manage it. A migration tool used improperly can wind up moving bad data. "It's the garbage-in, garbage-out analogy," Singh said.

Your goal, of course, is to transfer data quickly and reliably. What matters is how well you use data-migration tools, and "what accelerators and templates you have that go along with the tools," said Singh.

Risk #3: Cross-object dependencies.
"I can't tell you how many times I've sat in a meeting where (a client) said, 'We just discovered a whole new source of data that we were not even aware needs to be moved,'" Singh said.

Cross-object dependencies often are not discovered until very late in the migration process. A complex project may have 60, 70, or perhaps even 80 different data objects coming in from a hundred or so different applications.

"When we get involved with clients, we look for those missing pieces of data or dependencies," said Singh.

Indeed, cross-object dependencies--and discovering new sources of data late in the process--are major risks that can throw off your migration timeline.

Risk #4: Attempting to go live in one big upload at the end.
This is a recipe for disaster, Singh said, because you're assuming that everything is perfect--that you're going to be able to simply hit a button, and all the data will load flawlessly.

"That's a big risk," he said. "You need a project timeline with multiple, iterative test loads along the way."

Risk #5: Budget overruns due to inadequate scoping or preparation at the start.
This often happens when an organization believes its systems integrator (SI) will take care of these details. Big mistake.

"Most SIs usually don't handle data beyond saying, 'I'll connect the pipes and move the legacy data into a target system,'" Singh said.

"We get called into data-migration projects that are in the realization phase," he said, "and people are saying: 'Look, the data isn't tying together, we're not able to do user testing.'"

This problem, of course, can lead to cost overruns and wrecked timelines.

One of the biggest challenges facing IT today is risk assessment. Risk measurement and impact assessment aren't exact sciences, but there are tools, processes, and principles that can be leveraged to ensure that organizations are well-protected and that senior management is well-informed. In our Measuring Risk: A Security Pro's Guide report, we recommend tools for evaluating security risks and provide some ideas for effectively putting the resulting data into business context. (Free registration required.)