Your organization probably already has more data than it knows what to do with. Yet, it's quite likely you're overlooking, disregarding, unaware of, or unable to access important information that could directly affect analyses and business outcomes.
It doesn't matter what your universe of data is -- enterprise data or a combination of internal and external data sources -- important nuggets of information may be missing.
"Companies are collecting more data, but often struggle with what to do with it," said Dave Hartman, president and founder of technology advisory firm Hartman Executive Advisors. "Data can be extremely overwhelming in its raw form."
[What's hot in data science? Read 10 Programming Languages and Tools Data Scientists Use Now.]
Data silos are one culprit causing blind spots, which result in missed opportunities and can expose your company to risk. Today's companies need cross-functional views of their data in order to serve customers and to compete effectively.
"The untapped gems lie in the cross-links between data sources that relate to customers across channels and derive future value. This includes creating consistent and comprehensive customer profiles across the board," said Dominik Dahlem, lead data scientist at customer intelligence cloud platform vendor Boxever.
Achieving that kind of visibility isn't easy, since it requires organizations to tie together enterprise systems, which can be difficult to do despite the number of APIs available. Businesses also have to consider external data sources, such as social media or weather data, which help provide a complete view of customer behavior or serve to identify overlooked issues that might negatively affect business performance.
Constellation Research vice president and principal analyst Doug Henschen said unused data tends to fall into two broad categories. These are data exhaust and dark data, both of which have been aided by NoSQL technologies. Data exhaust is information an organization is currently not saving. Dark data is information companies may be saving but not exploring.
"Historically, companies used to throw away historical data simply because they couldn't afford to retain it. So they would keep, say, one or two years' worth of data," said Henschen. "Most people keep durable goods, such as beds and furniture, for at least five years. So targeted promotions based on two-year-old buying information may not [enable] deep enough analysis."
Dark data includes semi-structured and unstructured data that are hard to compute against. Although it's been possible to retrieve the metadata, deciphering the meaning has been a one-time, manual task. With modern search, natural language processing, semantic analysis, sentiment analysis, and machine learning, such data is now becoming computable, according to Henschen.
"The text in a CRM comment field can be analyzed and segmented across the entire customer base, rather than just looking at sentiment, one customer at a time," said Henschen. "The same goes for social network comments about your brand and products."
One way to identify missing gaps is to focus on the key metrics the organization wants to improve, which may turn out to be different than the current KPIs. For example, if the goal is to improve customer satisfaction, it's important to understand the cause of customer dissatisfaction, as well as its symptoms.
"The drivers of lower customer satisfaction might be inadequate product features and the symptoms would be lower customer adoption or product sales," said Sanjay Sidwani, SVP of marketing analytics at financial services company Synchrony Financial. "Getting a deep understanding of the drivers requires persistence in analyzing data and slicing it in multiple ways."
What data is your company missing? Once you've reviewed our 12 examples, tell us about your own experiences with hidden data in the comments section below.
[Editor's note: On page 3, the reference to Hartman Executive Advisors was corrected. Also, the quote on page 13 was updated to correct its attribution. It was said by Scott Masker, business systems engineer at MacLean Fogg.]Lisa Morgan is a freelance writer who covers big data and BI for InformationWeek. She has contributed articles, reports, and other types of content to various publications and sites ranging from SD Times to the Economist Intelligent Unit. Frequent areas of coverage include ... View Full Bio