Big Data Fails: How to Avoid Them - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management
08:06 AM
Connect Directly

Big Data Fails: How to Avoid Them

What are the key factors that ensure the success of a big data project, and what are the factors that can contribute to failure? EMC's Global Services Big Data chief shares his perspective, gained from working with customers in the field.

Big Data: 6 Real-Life Business Cases
Big Data: 6 Real-Life Business Cases
(Click image for larger view and slideshow.)

Here's how the implementation of big data initiatives typically goes at big companies these days: An organization deploys Hadoop, hires some data scientists, and waits for magic to happen. But the magic doesn't happen.

That's the scenario described by Bill Schmarzo, CTO of EMC's Global Services big data practice, and author of the book Big Data -- Understanding How Data Powers Big Business. Schmarzo also teaches the course Big Data MBA at the University of San Francisco.

Schmarzo's group at EMC doesn't sell products. Rather, it helps large corporate customers figure out what to do with their big data investments. He's seen many stalled big data initiatives. His group comes in when the stall happens. What's causing all these stalls?

"Organizations don't need a big data strategy as much as they need a business strategy to do with big data," Schmarzo told InformationWeek during an interview at Strata + Hadoop World in New York City this week. 

[Looking for more about what happened at Strata + Hadoop? Read Cloudera Boosts Hadoop Portfolio With Security Data Update, Offerings.]

When Schmarzo's group starts working with a customer organization, the first step is to figure out what the business is trying to accomplish in the next six to nine months.

Bill Schmarzo, CTO, EMC Global Services Big Data

(Image: Bill Schmarzo via LinkedIn)

Bill Schmarzo, CTO, EMC Global Services Big Data

(Image: Bill Schmarzo via LinkedIn)

"We have users that have these hunches about how things work because they've been living in a business environment for so long," he said. "But what are the business decisions they are trying to drive off of that?"

For example, Schmarzo tells the story of a hospital in a major city that saw an increase in injuries after a local team's sporting events. The people who worked there recognized the correlation. But how could they measure it and use the information to make business decisions? Schmarzo said that an investigation of the data showed that injuries went up by 27% on game day, and that quantification of the injuries was something the hospital could use to figure out how much it needed to augment its ER nursing staff for game days.

In another example, Schmarzo's team helped a casino that was trying to understand how to get more lifetime value from casino visitors. The company typically offered high-roller rooms to visitors who gambled a lot in one day. A big data analysis examined what would happen if the casino offered those high-roller rooms to other visitors. The result was that other visitors who were offered high-roller rooms ended up staying even longer and spending more than the high rollers did. They were potentially worth more to the casino in the long run.

These are a few of the success stories Schmarzo has seen in the EMC practice. But there are still plenty of challenges for organizations implementing big data initiatives. And the biggest one is cultural, not technological.

Schmarzo said his group sees the most success with midmarket companies. And there's another factor that seems to drive the success of these projects.

The companies that run into the most trouble are those in which data is in silos, and the thinking about that data is also in silos. For instance, in a banking company there may be a checking account silo and a mortgage silo, and the owners of each group aren't accustomed to thinking about the whole customer who consumes both services.

(Image: NorthernStock/iStockphoto)

(Image: NorthernStock/iStockphoto)

Companies that can get past that limitation in their thinking are more likely to be successful with their big data initiatives.

And that example also shows an important factor in successful big data initiatives – collaboration among groups who may not normally collaborate with each other. It relies on team members with different areas of expertise working well together.

"The places where we are seeing success is where the business people and the IT people like each other," Schmarzo said.

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Moderator
10/13/2015 | 6:18:26 PM
HPCC Systems
Jessica, very informative approach when tackling Big Data. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it's computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quickly and simply. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. Their built-in Machine Learning Library and Matrix processing algorithms can assist with business intelligence and predictive analytics. More at
User Rank: Moderator
10/5/2015 | 1:37:29 PM
Business case
It all needs to start with a business case. Without a solid business case, a big data strategy will struggle to ever gain the stakeholder support needed to find success. This is especially true if the organization has any hopes of taking big data beyond the "project" classification into a true way of operating. Peter Fretty, IDG blogger working on behalf of SAS
User Rank: Ninja
10/4/2015 | 2:37:10 AM
interesting to know
interesting to know, thank you for sharing... I learn a lot, thank you...
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
Where Cloud Spending Might Grow in 2021 and Post-Pandemic
Joao-Pierre S. Ruth, Senior Writer,  11/19/2020
The Ever-Expanding List of C-Level Technology Positions
Cynthia Harvey, Freelance Journalist, InformationWeek,  11/10/2020
White Papers
Register for InformationWeek Newsletters
The State of Cloud Computing - Fall 2020
The State of Cloud Computing - Fall 2020
Download this report to compare how cloud usage and spending patterns have changed in 2020, and how respondents think they'll evolve over the next two years.
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Flash Poll