6 Causes Of Big Data Discrepancies

1 of 7

(Image: Geralt via Pixabay)

As the universe of big data continues to explode, organizations struggle to identify and leverage the data that matters most. As companies continue to add more data sources to the mix, the number of potential data sets grows, as do the opportunities for new and different insights. Because data can be combined in so many different ways, there are many possible outcomes.

"People have different choices from the same data set," said Kirk Borne, principal data scientist at Booz Allen Hamilton in an interview. "People will choose different things they think are informative, so the search is to find the most important variables."

As a result, the same data can result in very different interpretations.

"If I run a supernova simulation where the resolution is too low and two supernova scientists analyze that, if one knows the simulation is not a sufficient resolution and the other doesn't, they would come to very different conclusions," said Tony Mezzacappa, chair of theoretical and computation astrophysics at the University of Tennessee, in an interview. "Data completeness is part of data quality. People should understand what the dangers are in extracting conclusions based on such data."

Whether or not data is complete enough may not be obvious until later. For example, cosmic microwave background radiation confirmed the Big Bang theory, at least until the European Space Agency discovered that dust in the universe emits microwaves of its own that can introduce the same kind of polarization.

"That was a very big deal when it was announced. If [cosmic inflation] had been confirmed, it would have spoken volumes about the nature of our universe, its beginning, its evolution, and its end," said Mezzacappa. "Further analysis, and likely further astronomical data collection, will be required to include the effect of dust."

There are inherent uncertainties in algorithms, models, outcomes, and sometimes the data itself that can impact conclusions. Human nature also plays a part. Here, we explain six of the many reasons the same data can result in different interpretations.

1 of 7