How To Find Strategic Advantage From Big Data

Big Data Talent War: 10 Analytics Job Trends

Big data has become more than the hot buzzword in a technology sector still adapting to the fundamental changes created by the last wave of disruptive technologies--virtualization, cloud computing, mobile computing, and BYOD consumerization.

Whether they're prepared to deal with it or not, even mid-size companies have to face big data, not for the benefits it could offer, but because nearly all corporate data has become "big," according to reports from Ventana Research, Forrester, and other analysts.

Stereotypically big data sets--defined as any data set too large for conventional databases or analytical tools--are inflated with status reports, error messages, activity logs, and other data from the "Internet of Things," according to a seminal report from McKinsey & Co.

Big data is certainly a game changer, but it's at a stage of development in which hype overwhelms reality, according to Dan Vesset, the IDC analyst who was lead author on a March study that predicted the market for big data products and services would grow at 40% a year through 2015.

[ Do you see big data as a pain or an opportunity? Read Does IT Really Care About Big Data?. ]

The rapid growth, though it seems sudden, is actually the result of nearly 30 years' worth of development in analytic software, data warehouse techniques, and best-practice guidelines on how to manage a business based on data rather than intuition, Vesset said.

Helping the growth along is the rush among major vendors to adapt non-big-data databases and software to handle big data, as well as "more than half a billion dollars in venture capital" funding a host of startups, Vesset said.

That doesn't mean either users or vendors really understand the market.

Users are eager to build on reorganizations made to take advantage of ERP, CRM, and other products designed to put data, rather than guesswork, behind critical business decisions. Few top-level executives, however, understand the changes necessary in a company's data-gathering and decision-making processes well enough to make big-data migrations a real priority.

Still, users push their organizations toward big data, even at companies that find little value in data from the Internet of Things, according to Forrester analyst Mike Gualtieri.

Status reports from the inanimate aren't necessary for an app or a decision to qualify as being driven by big data, Gualtieri said. The only thing needed to give small data the impact of big is that the data be demonstrably causative. That is, the data being analyzed has to have a direct cause-and-effect connection with the decision being made. Big data needs to be causative; needs skilled data analysts to build and manage it; needs good-quality predictive-analytic software to boil data down to recommendations; and needs a program, platform, or set of processes that put changes in place based on decisions driven by data, Gualtieri said.

It's good for mid-size companies that all those requirements are available to them as well as to larger companies, but that availability also puts mid-size companies in the position of having to move faster on big data than they'd like, just to keep from being left behind, according to Daniel Castro, senior analyst with Information Technology and Innovation Foundation (ITIF), a Washington, D.C.-area think tank.

Among the problems mid-sized companies have adapting to big data is the kind of non-standard data it includes, primarily unstructured text from end users. That includes commentary or feedback from social networks, location data from mobile devices, forum postings, and other user-generated sources of data.

That data, almost completely excluded from BI or other assistive decision-making products until recently, has become the primary means of customer interaction for most companies, according to McKinsey analyst James Manyika.

The McKinsey study contrasts the traditional reliance on transactional information as a source of intelligence on customers with what it called "exhaust data" within big-data datasets. Exhaust data, like metadata, is content created as a by-product of other online activity. Unlike metadata, exhaust data exists on its own rather than simply as a way to provide context for records stored in traditional relational databases.

12 Top Big Data Analytics Players

Being able to collect more data on customers and shovel it into an analytic engine is only half the capability most companies need to develop, however, according to Larry Kanter, a forensic accountant and president of Kanter Financial Forensics in Dallas.

Good analysis and decision-making depend on the quality of data, not quantity, Kanter said.

Without tools or expertise to eliminate minutiae or select only relevant classes of information, the only alternative most companies would have is to hire forensic accountants or other data experts to spend hours or days filtering big data by hand--a process far more expensive, imprecise, and un-repeatable than using software designed to do the same thing in a fraction of the time, without the risk that perceptual biases will warp the analysis.

Fortunately, according to venture capitalist, author, and tech-entrepreneur Dave Feinleib, there is a rash of big data apps on the way to do just that. Some are designed to gather and analyze data generated by machines; others are designed to collect and contextualize data from Facebook and other social networks.

The fastest-growing categories, however, are those designed for specific vertical markets such as healthcare and finance, and those aimed at collecting and analyzing data for specific departments within the organization, such as marketing, IT, and customer service.

Though many are traditional on-site apps, most are appearing as online apps built on cloud platforms with the computing power to analyze huge data sets as well as the algorithms to find useful knowledge within masses of data, according to Raj De Datta, CEO of BloomReach.

"BDAs [big-data apps] don't just repackage your data in a cool interface or offer productivity improvements in data scalability, they harness the world's data to deliver you a better outcome--like more revenue," according to De Datta.

BDAs, publicly available at subscription prices rather than the full cost of licensing and implementing sophisticated analytics on-premise, could make big-data analysis available to vast numbers of mid-sized companies that lack the will or expertise to build their own tools.

In business it's almost always better to make informed decisions using hard data than to use experience and intuition, but most companies have only begun to shift toward data-driven decision-making, according to a study released last month by Capgemini.

Both enterprises and mid-sized companies have difficulty making that move, Capgemini found.

Of 607 business managers, 90% said they would have made at least one major decision differently during the past three years if they'd had access to better data.

Nevertheless, more than half of respondents said big data is not considered strategic at the highest levels of their companies, despite the quality of decisions that can be made based on it.

That doesn't mean top executives aren't looking for better data with which to make better decisions, according to Capgemini's South East Asia CEO Deepak Nangia.

It only means that the organization's data-handling practices are not prepped to give decision makers the data they need when they need it to make better decisions, which would lead them to see decisions based on big data--and therefore big data itself--as a strategic advantage worth pursuing, Nangia said.

The challenge for big-data advocates--as it is with many promising but early-stage technologies--is to demonstrate quickly the benefit of building complex data sets and analytics to help their companies make better decisions, and get top managers to agree to make the investment even without the heavily analyzed data that might help push them in the right direction, Castro said.

"There's still a lot of convincing to be done," he said.