Data Drift Happens: 7 Pesky Problems with People Data - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management
01:10 PM
Matthew Magne, SAS
Matthew Magne, SAS

Data Drift Happens: 7 Pesky Problems with People Data

Making sense of data associated with customers, citizens and other individuals may seem simple, but it's possible that the records don't tell the real story.

People are like snowflakes. No two are alike. But, we all have similar characteristics – also known as personally identifiable information (PII) -- basic elements that organizations track about customers, citizens or any other representation of an individual.

But, people aren’t static entities. Lives change, and that leads to bad, or at least inconsistent, data. The problem manifests itself in marketing efforts that fail to reach the right customers. Or voter rolls filled with citizens who have moved, changed party affiliations or died. Or an insurance company that receives multiple claims about the same individual for the same incident.

Image: Pixabay/3dman
Image: Pixabay/3dman

Luckily (or unluckily, depending on how you look at it), the problem isn’t new. Here are a few reasons why data on individuals has flummoxed organizations for decades.

  1. Movement. According to the US Census Bureau, about one in nine Americans moved over a one year period. As a marketer, imagine more than 10 percent of your customer addresses changing in a single year. That’s happening, and while the percentage fluctuates, it’s a good reason that personal data remains dynamic.
  2. Names. This may seem like a simple one, but in the case of marriages, it’s a change in personal data that happens frequently. What happens when a repeat customer signs up for the same service using a different name? You may create duplicate data if you can’t catch the similarities in other parts of the record. And one of the hardest things to do is to create a single view of the customer across all your systems. All of these factors can lead to fractured, inconsistent views of your customers.
  3. Households. As people move, they often combine one family with another. In the case of a marriage or civil union, that can mean new financial arrangements or legal rights that an older database may not track. Or, if you get a new roommate, maybe there’s just another name on the bill. Any change in this status leads to data elements, like name or address, that need to be reconciled. 
  4. Channels. With the rise of social media, organizations have to track and relate even more channels of interaction. Thirty years ago, it was simply just a phone number and address – what fit into a now-quaint volume called a phone book. Then, organizations could track email, phone and store traffic. Add social media to the lists of channels, and it gets more complex. But, does email go away in favor of Snapchat? Absolutely not. It’s just another channel added to the mix.
  5. Privacy. Customers can get online and order products, manage their accounts and receive timely service. Access is good. However, you have to figure out how to provide these services while avoiding a costly security breach.
  6. Errors. People data can get out-of-hand because people are imperfect. People mistype data that they enter into self-service form. Or, a data entry clerk may misread a handwritten form or simply miss-key an item. Or they type an additional address or other personal information into the notes field where it can’t be sorted or found. All of this introduces the human element – and bad data – into your systems.
  7. Processes. Personal data is the representation of an individual, but it is also the result of successfully executed business processes. If customer service staff isn’t trained correctly, they can input data incorrectly. If new customers are added to the sales force automation solution without a set of instructions designed to check if the customer already exists, you’ll have duplicates in your database. Now what happens if you change one of their addresses? Did you change both records? Which one is the best address? Bad data is often a symptom of a broken or failing process or a manual process that could be automated to ensure you have access to quality, trusted data

So, what’ the solution? First, simply understand that data drift happens. The trick is to deal with it proactively. That means you should keep better track of where your data resides. Create business rules and monitor it constantly to ensure it’s of high quality. Put in place automation where you can. And apply data quality to ensure there are less duplicates and correct addresses, helping you get that consistent view across different systems.

The negative ramifications of mishandling data increase every year with legislation like GDPR in Europe. These regulations could mean huge fines if data is not handled correctly. Beyond that, how can you deliver the best customer experience, if you don’t know your customer’s correct address, last name, or what products they bought both online and offline? It’s more important than ever to have a plan in place to handle your data, automate your broken business processes and apply data quality management to your business.

Every organization has a pool of data on customers, employees, citizens or other types of individuals. Making sense of that data may seem simple, but it’s possible that the records don’t tell the real story. With more data arriving each day – and more options for people to manage data outside of the enterprise IT structure – getting people data right is quickly becoming a must-have.

Matthew Magne, SAS
Matthew Magne, SAS

Matthew Magne is a TEDX speaker, musician, avid Catan player and data professional currently working as a thought leader for SAS in the Big Data Management space. A recovering entrepreneur and coder, Matthew has taught entrepreneurship in prison, launched multi-million dollar big data products, and spoken at numerous conferences on topics such as Big Data Analytics and Transforming Data Professionals into Data Rock Stars.

Matt is a member of the UNCW Marketing Advisory Board, the UNCW Data Science Advisory Board and a mentor at TekMountain, a co-working space and technology incubator.  He received his BS, cum laude, in Computer Engineering at Boston University, has done graduate work in Object Oriented Development and completed his MBA at UNCW.

The InformationWeek community brings together IT practitioners and industry experts with IT advice, education, and opinions. We strive to highlight technology executives and subject matter experts and use their knowledge and experiences to help our audience of IT ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Data Science: How the Pandemic Has Affected 10 Popular Jobs
Cynthia Harvey, Freelance Journalist, InformationWeek,  9/9/2020
The Growing Security Priority for DevOps and Cloud Migration
Joao-Pierre S. Ruth, Senior Writer,  9/3/2020
Dark Side of AI: How to Make Artificial Intelligence Trustworthy
Guest Commentary, Guest Commentary,  9/15/2020
White Papers
Register for InformationWeek Newsletters
2020 State of DevOps Report
2020 State of DevOps Report
Download this report today to learn more about the key tools and technologies being utilized, and how organizations deal with the cultural and process changes that DevOps brings. The report also examines the barriers organizations face, as well as the rewards from DevOps including faster application delivery, higher quality products, and quicker recovery from errors in production.
Current Issue
IT Automation Transforms Network Management
In this special report we will examine the layers of automation and orchestration in IT operations, and how they can provide high availability and greater scale for modern applications and business demands.
Flash Poll