Big Data Moves From Hype To Reality, CompTIA Finds

14 Ways IoT Will Change Big Data And Business Forever

Companies are moving to the next stage with big data -- past the hype and into broader adoption. But new challenges are on the horizon, such as how to master all that data. Those are a couple of several key findings in a recent Computing Technology Industry Association (CompTIA) study.

The industry organization surveyed 402 IT professionals for the report released in December and found that 72% of respondents said big data projects had exceeded their expectations, and about 75% said their businesses would be stronger if they could harness all of their data.

Companies "are trying to consolidate data and understand the structure of their data," said Seth Robinson, senior director for technical analysis at CompTIA, in an interview with InformationWeek. They are finding this out the hard way, because lax data policies in the past allowed departments and work units to "do their own thing" in IT, resulting in data silos. "Companies have to look at data differently now."

Data silos are a problem for 42% of IT professionals responding to the survey, up from 29% two years ago.

More Data Than Ever

Plus, more data is being generated than ever before. The CompTIA report cites another study by Cisco that forecast 2016's IP traffic passing the 1 zettabyte threshold. That is 1 billion TB of data. That number will double again in three years. Helping to drive the data glut, the price of storage has dropped from $1,000 per GB in 1995 to a few pennies today. So, squirreling away 100TB of data is not going to be a big deal, especially now that companies can rent storage in the cloud to suit needs.

[Algorithms are changing the world. But where did they start? Read Algorithms: Turning Data Into Products.]

How Organizations Use Data

Yet 63% of survey respondents said they rely on data for day-to-day operations, 61% said that they are more sensitive to data privacy issues, and 59% said that they want to use data to better understand their customers.

Respondents are evenly split when asked how and where they seek improvement in their operations. For example, 40% said they are doing well with real-time analysis, while 42% seek improvement. A full 39% of respondents want to be able to search all of their data across departmental lines, while 37% say they want to improve their ability to do so.

In 2015, 51% of all respondents said they were undertaking big data projects, while another 36% said they were planning to do so. This is up from 2013, when 42% of respondents said they were undertaking big data projects, and 46% said they were planning them.

Despite the desire to reap the fullest benefit from data, companies undertaking big data projects are not suffering setbacks. "They are not running into headaches or hidden costs," Robinson said. Companies may be meeting modest goals. "The metric may not be that obvious," he said. "Coming up with the right metric is part of being data savvy."

Skills Gap

Still, there is a skills gap that needs to be addressed. A total of 49% of respondents said they have sufficient data skills in-house to take care of big data projects. But 40% admit they are "moderately deficient." The biggest skills gaps are real-time analytics (reported by 42%), relational databases (41%), data security (36%), predictive analysis (35%), distributed storage (33%), and data mining (30%).

"Part of this is technical," Robinson observed. The tools needed to handle big data are new, so there may be insufficient experience to make full use of their potential. There is also an informal skill set needed as well, since data analysts need to "speak" business as well as understand the technology, he added.

Enterprises may turn to outsourcing to help fill this knowledge gap, but only if they can find the right skills there. "We heard companies saying when they go outside, they can't find the knowledge or the skills," Robinson said.

The report noted that third parties are still concentrating their expertise on component services and not end-to-end solutions.

Still, component expertise can become a building block toward providing a big data fix. Data backup, for instance, could lead to "having a conversation about the data," Robinson said, while installing a firewall can be a step toward greater data security.

"People have gotten past thinking it (big data) is hype," Robinson said. Companies are building on what they have learned, which in turn can lead them to more comprehensive big data solutions, he added. "We'll see that play out over this year."

**Elite 100 2016: DEADLINE EXTENDED TO JAN. 15, 2016** There's still time to be a part of the prestigious InformationWeek Elite 100! Submit your company's application by Jan. 15, 2016. You'll find instructions and a submission form here: InformationWeek's Elite 100 2016.