Who Is Paying For Big Data Projects?

There is an interesting contradiction in data from a new survey of open source big data developers, data specialists, and IT managers: Big data projects designed to give business managers better information with which to make critical decisions are moving forward in many companies, but neither the business units nor IT seem aware of the progress--or be willing to pay for it.

The survey, from open source Hadoop, data integration, and big data analytics firm Jaspersoft found that a surprisingly high 62% of in-house corporate developers are either planning or actively deploying systems to deliver big data analytic functions, whether the projects are labeled "big data" or not. Only a third have a formal budget or any other funding to help them do it, however.

"Given the maturity of the big data market in general, we were expecting it would all be skunk works--projects in a kind of bootstrap prototyping/research phase," said Mike Boyarski, director of product marketing for the company.

JasperSoft's results may have been skewed because many of those it surveyed are highly technical, open source developers accustomed to doing much of the fundamental development work themselves.

Many, for example, are getting ready to deliver big data analytics without purpose-built big data software. Instead they're adapting existing analytics or database management systems to "function at an extreme level," on very large, very diverse, very changeable data sets, Boyarski said.

They're also getting support from forward-thinking business-unit managers interested in the kind of insight deep analysis of customer behavior can provide, but who aren't prepared with the business cases, return-on-investment analyses, or user requirements that might get a big data project budget approved, Boyarski said.

"There is a general awareness about the technology among business managers, but that has not been completely translated into budgeted, planned, clearly identifiable plans to move forward with it," Boyarski said. "They do get that analyzing new data of different varieties and different velocities is important, but not what to do about that. So on the IT side, technologists are doing what it takes to move forward even without extra funding."

[ Don't get sidetracked from the start. See Big Data Projects: 6 Ways To Start Smart. ]

Big data is not yet considered a strategic asset or potentially game-changing advantage, largely because business managers haven't had as much time to learn about or see the advantages in big data as they did with cloud computing, BYOD, and other technologies that were driven more by business units than IT, according to Frank Gillett, VP and principal analyst at analyst company Forrester.

The odd thing is, the dichotomy between wanting the advantages of big data and the will to pay for it may continue for the next several years, despite projections from Forrester and other analyst companies showing sales of big data products will grow as much as 40% per year until 2015.

Business managers aren't opposed to big data, they just don't know enough about the difference between it and traditional analytics to do any long-term planning, according to yet another survey, this time from market researchers at TheInfoPro.

According to InfoPro results, 56% of midsize and larger companies have no plans for any projects involving big data after 2013, even though almost none said they would willingly do without its advantages.

Of 607 business-unit managers surveyed by Capgemini in June, 90% said they would have made at least one decision differently during the past year if they'd had better information about the question--the kind of data they might have gotten from big data analysis.

Even so, more than half of Capgemini's respondents said big data is not on their list of technologies likely to deliver strategic advantages to the corporation.

"We compared our results to some of the other surveys and got very similar results," Boyarski said of JasperSoft's search for answers about big data. "This still seems very much like a bottoms-up initiative," he said. He said grassroots support more often goes to technologies more likely to end up in the hands of end users, not large-scale analytics likely to be available or useful to only a specialized subset of any organization that uses them. The difference in support seems to be related to the issues dealt with by different businesses, he said. Companies that rely heavily on the Web for sales, marketing, or interaction with customers tend to be more savvy about SEO, Web server-log analysis, and other relatively specialized ways to attract or analyze the behavior of customers, Boyarski said.

Big data holds more promise for them because their customer base is large, the value of each transaction is often small, and even a small change in marketing or customer service can be multiplied into a huge bottom-line result.

Companies focused more on selling in the real world--through what used to be called brick-and-mortar operations, or in B2B organizations whose transactions tend to be fewer but more expensive--tend not to see as much benefit from deep analysis of a comparatively small number of transactions.

"E-commerce companies look at customer churn and want to analyze why people are leaving or not buying," Boyarski said. "If they can identify the reason for the churn, they can fix the root cause more easily."

Another reason could be the source, nature, and potential result from the data being analyzed, according to Forrester analyst Mike Gualtieri.

Most of the information in big data analytic sets is still proprietary--from CRM applications, Web server logs, and other machine-to-machine data, not the unstructured text coming from social networks, email, or other sources in human, not machine, language.

Collecting, standardizing, and processing that data for effective analysis using existing tools is a huge challenge--one that could easily put the kibosh on big data plans for companies that see more value in getting opinions straight from the horse's mouth, rather than from analyses of the pages customers view, links they click, and whether or not they ultimately decide to buy after browsing a company's website, he said.

Nearly every information-management tool maker is working on new big data features, add-ons, or partnerships, so as not to miss out on the opportunity to sell into a hot new market, Gillett said. As those modified apps come to market--and more purpose-built big-app analytics and data integration products ship--that may change, he said.

Boyarski had a different theory: It's the timing, not the tools that may be the key to making the IT market's hottest, least-understood new phenomenon out of the category of experimental technology and into the list of those considered worthy of use in production systems.

"Companies that are looking for customer-oriented analytics often also want the tool to give them a real-time response--several seconds to even a couple of minutes. They want to be able to respond quickly to customer needs, which would be a real advantage," Boyarski said. "Most of that data right now is batch-oriented, so the best you can do is make decisions daily or weekly rather than hourly.

"As the requirement for real-time responsiveness goes up--especially online--so does demand for better analytics. We saw a one-to-one correlation between those two things," Boyarski said. "A lot of companies right now are still tackling more basic problems--less than 30% said they're even using all the data they already have to make decisions. Some of that has to be worked out first."

Expertise, automation, and silo busting are all required, say early adopters of private clouds. Also in the new, all-digital Private Clouds: Vision Vs. Reality issue of InformationWeek: How to choose between OpenStack and CloudStack for your private cloud. (Free with registration.)