Hadoop Adoption Poised For Growth: Survey

7 Worst Cloud Compliance Nightmares

Hadoop adoption is past the "kicking the tires" stage and is likely to see enterprise adoption accelerate, according to a survey of more than 2,100 business, IT and C-suite executives from 1,300 companies worldwide.

The survey was conducted by AtScale, a vendor whose products and services aim to help business users visualize the data in Hadoop platforms. The firm partnered with MapR, Tableau, and Cloudera to survey those companies' customers about their Hadoop use. The survey included responses from CIOs and CEOs working at the same companies.

According to AtScale, this is one of the larger Hadoop surveys to date. One was a Barclay's Equity Research Unit forecast that painted a very bullish outlook for Hadoop adoption, Aziza said. The other was a Gartner study from earlier this year that received 125 responses from a group of 284 and offered a relatively muted outlook for Hadoop adoption.

[ Test your container know-how. Read Containers 101: 10 Terms To Know. ]

More than three quarters of respondents (76%) said they plan to do more with Hadoop in the next three months. "We've entered a second phase where people connect Hadoop with business users," said Bruno Aziza, CMO of AtScale. About 77% of respondents are not tire kickers. They are already operating between 10 and 500 Hadoop nodes. And many of them believe they will reap strategic advantage.

In fact, nearly half of respondents (49%) said they obtained some "tangible value" from using Hadoop. Another 45% said they hope they will, while the remaining 6% were pessimistic.

Hadoop adoption is not a slam-dunk guarantee of success. Companies are more likely to reap value from a Hadoop implementation if they are operating more nodes. Among respondents whose companies operate 500 nodes or more, 76% reported getting "some value" out of Hadoop. Conversely, only 33% of respondents whose organizations operate 10 nodes or less said they got "some value" out of their Hadoop installation.

Value is more easily attained if self-service access (SSA) is part of the implementation. Respondents from companies using SSA were 50% more likely to extract value from their Hadoop implementation than those who were not, according to the report.

A lack of skill sets among IT professionals may be the biggest stumbling block facing Hadoop adoption. The one bottleneck identified by the survey is the need for professionals with the skills that enable them to manage a Hadoop implementation. Two thirds of respondents (66%) identified this as their top challenge. Aziza cited a McKinsey & Co. study that showed the need for 140,000 to 190,000 skilled analysts to fill this gap.

Challenges notwithstanding, about half of those surveyed said they are committed to making greater use of Hadoop in the next 12 months, Aziza said. "You've got a year to figure it out. In a year, everybody is going to be on board."

ETL is one bridge leading organizations towards greater Hadoop adoption. ETL, which means extract, transform and load, involves people "taking data from the traditional data source and dumping it into Hadoop," Aziza said. Survey results showed 74% of respondents using Hadoop for ETL.

The next step in the transition takes Hadoop from the realm of data scientists to business users. Data scientists "focus on fewer and more complicated problems the enterprise sees," Aziza noted. The challenge is getting the data in front of business users and analysts.

Self-Serving Access (SSA) is what does this. Nearly two thirds (61%) of respondents believe SSA will allow them to obtain value from a Hadoop implementation. Here, "obtaining value" does not mean cost cutting. Rather, it's about achieving increases in revenue because the business side is making better decisions with the data they have.

Finding the insight that yields the gain is like finding a needle in a haystack. But in the world of big data, making a bigger haystack works to the user's advantage. "More data means better algorithms," Aziza said. "The more data I can capture, the better the correlations will be."

Ultimately, all paths lead toward achieving business results for the enterprise. In the end, sales and marketing people need to know they are spending their dollars wisely to spur sales. Using Hadoop to get there appears to be gaining traction.