Big Data Squeezes Legacy IT Spending: IDC

Big data really isn't new. Data has been getting bigger for years, and companies have been finding ways to glean insights from their growing stockpiles of data for years. But it seems that calling it by a new buzzword is what it took to move it front and center for high level executives. It worked for cloud computing, didn't it?

Big data analyses helped Xerox slash employee turnover in its call centers by revealing a set of criteria predicting how long a new employee would stay. The criteria were simple enough that the main part of Xerox call-center candidate interviews is now done by software, rather than humans, according to the Wall Street Journal.

Predictive analytics have also helped companies discover other incentives that could boost retention. The finding that pay rate isn't a key factor can save companies' money by reducing the average size of incentive raises, according to a study by HR consultancy Mercer, a unit of Marsh & McLennan. Most companies--95%--look so closely at pay rates that they have little attention to spare for other factors.

Predictive modeling--a classic big data function--is far more accurate at identifying the reasons employees leave, but only 43% of companies polled by Mercer use any predictive modeling at all. Methods Mercer describes as less sophisticated analyses are far more popular. For example, 95% of companies use benchmarks from analysts, professional associations, or other external sources, while 90% use internal benchmarks. Only 64% use mathematical models that simulate specific personal situations to predict individual decisions--as researchers in sociology and psychology often do.

[ Big data is all the rage. Executives Push Big Data Projects, Not Sure Why. ]

The ability to collect and analyze data is not new, but it is exploding due to increases in computing hardware, the capacity of high-volume database-management systems such as Hadoop, and analytics that came first from the open source community.

"Analytics is top of mind for corporations" these days, according to IDC BI analyst Dan Vessett.

The overall market for business analytics--of which big data is one segment--will grow to more than $57 billion by 2016. "Driven by the attention-grabbing headlines for big data over three decades of evolutionary and revolutionary developments in technology, the business analytics software market has crossed the chasm into the mainstream mass market," Vessett said.

In 2011, IDC's 2011 Digital Universe Study predicted sales of big data analytics would grow at nearly 40% per year through 2015. This is seven times the average speed of other products in the same broad category, according to a March report by Vessett and Benjamin Woo, VP of storage systems at IDC.

By comparison, the 6.8% per year overall that global IT spending will grow--according to IDC predictions--is thin soup.

Making it thinner is the popularity and sales growth of technologies IDC predicts will become the four pillars supporting the next major era of corporate computing: mobile devices, cloud computing, big data, and social networking.

Together the four, heavily consumerized technologies will account for as much as 20% of total IT spending, reshaping the IT universe by adding new devices while starving legacy systems, routing every dollar toward new technology rather than maintaining older systems, according to IDC's predictions.

As a concept, big data woke end users to the potential for marketers to nano-target ever-narrower slices of the consumer market, and define corporate strategies according to accurate measures of a business environment, rather than broad approximations, according to Woo.

Three quarters of companies with more than 500 employees plan to invest in business analytics during the next 12 months, according to IDC, but very few seem to understand why. Only 38% said unequivocally that previous BI investments had paid off, while 15% said it hadn't, and 11% didn't know.

Thirty-six percent said they don't know how to measure the benefits of BI, while 48% said they couldn't even guess how long it would take a BI project to pay back the cost of using it.

Some of the reason is the fault of traditional methods and products, which were too inflexible, too limited in the data they could consider, and too lacking in both detailed analysis and predictive abilities to make their benefits clear. Most companies can make high-level, strategic decisions using analyses that approximate the answers they need, according to Vesset's analysis. The next step up in sophistication requires better data and processes, streamlined to make analysis timely and actionable, that allow companies to improve their troubleshooting, operational efficiency, and performance in near-real time, rather than on a monthly or quarterly basis.

The most sophisticated level of analytical capability is the potential to make day-to-day tactical decisions based on data rather than instinct, according to the report.

That level of analytical ability, however, is comparatively rare because it requires high levels of automation in the gathering and dissemination of data, identification of patterns, real-time performance monitoring, and extra touches that make abstruse analytics accessible to non-technical business users.

Creating those abilities in companies that don't have them takes more than just the addition of sophisticated data-handling software such as Hadoop or other open source big data analytics.

Making even the basics of big data work is easier with help from other hot-trending technologies--cloud networks that provide capacity on demand and data from mobile devices or social networks that offer a look at the real-world use of a company's products or services, data that may be difficult to get using traditional methods, Woo said.

It is the combination of these major technologies that makes them the pillars of the next generation of computing, according to Woo and Vesset.

Demand for combinations of these technologies will also drive development of other technologies--ultra high-capacity storage media; high-speed, high-volume messaging; and even higher-capacity broadband networks to connect big data, big clouds, and distributed applications, for example, according to IDC's March report from Woo and Vessett.

Because each of those technologies also depends on a reliable, high-speed, secure infrastructure, the four pillars of the new world of IT will not only drive development of consumer technology. They will drive reinforcement, acceleration, and refinement of corporate IT infrastructures as well, the two wrote.

The future of IT, in other words, relies as much on updating the technology of yesterday as on inventing something new. Not coincidentally, it is the difficulty in taking advantage of any of the them that will drive most companies to develop the skills and infrastructure they need to make all work together and quantify the benefit of having done so, according to experts in predicting the progress of analytics as well as technology.

Many enterprises are building data warehouses to centralize the ever-increasing information flowing through their organizations into useful repositories. This makes good business sense, but it opens up a slew of concerns from a security standpoint. IT professionals can apply many of the same security best practices used with databases, but there are new lessons to be learned, as well. Download our Securing The Data Warehouse report. (Free registration required.)