IT's 10 Fastest-Growing Paychecks

For many enterprises, big data analytics may seem more like a collection of science experiments rather than a refined discipline that delivers tangible insights. And organizations experimenting with Hadoop and other big data tools may also find their tried-and-true data-management methods aren't producing the results they had hoped for.

The solution? Adding social media collaboration and crowdsourcing to analytics, or so says Oliver Ratzesberger, senior VP of software at Teradata and formerly analytics head at eBay and Sears. Ratzesberger is writing a nonfiction book, tentatively titled The Sentient Enterprise, in collaboration with Northwestern's Kellogg School of Management professor Mohan Sawhney. The book, slated for publication next year, will examine the role that crowdsourcing might play in the future of analytics.

It's a social approach to big data analysis, or "LinkedIn for Analytics," the authors say.

[You may need an extra turkey. See 10 Techies We'd Invite To Thanksgiving.]

Ratzesberger came up with the concept in 2008.

"We started realizing that when you bring more and more people together to work on analytics in a company, the traditional concepts of metadata and documentation -- and 'go ask that person for insights' -- start to break," Ratzesberger told InformationWeek in a phone interview.

For instance, when a company grows ever larger, manages faster and faster data streams, and has 100 or even 1,000 or more analysts, the old ways don't always work.

"Traditional metadata was meant to be done by a centralized team that documents things, so that other people can find it," said Ratzesberger. "But you can't document the world with five, fifty, or five hundred people."

He added: "The problem is, we human beings are not very scalable. If you want us to do twice as much work, it usually means twice as many people. And there's not a single CEO or CIO in the world who has unlimited headcounts."

So here's where the social media approach comes into play. The future of analytics must utilize the wisdom of the crowd, including human interaction via social networks and even gamification, said Ratzesberger.

For instance, if you have hundreds or thousands of people within an enterprise working with data, their interactions will reveal a lot about what data is most relevant to the organization.

Said Ratzesberger: "We started building this concept of 'analytics of analytics,'" which involves hiring a data scientist who builds analytical models of what other analysts and data scientists in the organization are doing with data.

This allows management to better understand key information, such as what new data sources are being pulled in.

"There's another level on top of that," said Ratzesberger. "Not just looking at all the queries and dashboards in use, but [also the] commentary and discussion that goes around with it."

This is where things get social in a big way. First, the system makes it easy for users to search for any known piece of data within the company.

The next step sounds a lot like Facebook, LinkedIn, or Twitter: You "take any interaction with data and allow people to 'like' a data set or visualization, follow an analyst, and basically create networks of interactions," said Ratzesberger.

This social approach shows relationships between data sets.

"I could then say, people who use this data set also look at the following data set," Ratzesberger said. "Or people who use this visualization also look at these things. Or here's a new trending set of visualizations that everybody is talking about."

A Tableau visualization, for instance, could be a starting point for a conversation initiated by a data analyst. "People could respond to it and say, 'You know what, I overlaid another variable in this visualization, and here's what it looks like," said Ratzesberger. "You could see a whole story of data unfolding in front of you."

Just 30% of respondents to our new Big Data and Analytics Survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives? Get the The Trouble With Big Data issue of InformationWeek Tech Digest today. (Free registration required.)