Big Data's Surprising Uses: From Lady Gaga To CIA

If I've learned one thing in 17 years of doing business analytics, including designing systems for companies such as Home Depot, Best Buy, Panera, Coke, P&G, General Mills and Kraft, it's that collecting lots of data is easy. Applying analytics to gain insight into customers' needs and wants? That's hard when you have data flowing in from all directions -- social media, blogs, forums, smartphone applications, surveys and other unstructured digital channels.

In my experience, it's common for senior management to acknowledge that big data can provide meaningful and actionable insights. But when it comes to actually mining this data for insights, many companies don't know where to start or focus on the wrong things and get bogged down.

Here are five basic principles for getting unstuck. Many of them might seem simple, but done in order, this system delivers game-changing results.

1. Define Your Scope.

Every business faces challenges such as growth, retention and costs. It's important for senior leadership to identify the most pressing problems. Then IT can focus in on few key areas where analytics can help identify root causes. Say sales are down in a particular product line.

[ What big data tools do you wish you had? Read 5 Big Wishes For Big Data Deployments. ]

Once the business has decided this is what needs attention, set realistic goals in areas where you can achieve measurable success over six to 12 months. The objectives could be to reduce customer attrition by 2%, increase margins by 3%, or increase coupon redemption rate by 5%.

2. Identify A Business Sponsor.

Find a champion who can communicate effectively with both senior leadership and technology/analytical employees while spanning organizational hierarchies and departmental gaps. This person will play a central role in controlling and communicating findings for future analytical projects. If that seems like a special breed, you're right, and many firms still struggle in selecting their analytics leader. Hint: The key is less about analytics or management experience than the art of removing clutter and plucking out the actionable insights.

3. Lose The "Everything Or Nothing" Mentality.

Don't wait until you have the perfect data warehouse to do analytics. Take the leap, start with a proof-of-concept project, and measure initial results within three to six months. I've worked with companies that thought they should have all the data and attributes ready to start an analytics project, which is not correct. Learning and applying small concepts in the right direction based on the data is better than guessing at the answers until the data warehouse is ready.

For example, in one financial services firm where I consulted, we spent three months scratching through huge piles of raw data to understand why revenue margins were getting squeezed, even though volumes were growing. Using data-mining techniques, we found the root cause was attrition of high-margin customers. The management immediately took retention measures, which halted the margin decline within six months. Otherwise, they would have waited another year for the complete data, during which time they would have lost plenty of customers.

4. Balance Speed With Accuracy.

Many companies spent the last decade building massive warehouses to host big data but have yet to see any return on that investment. Best case, a typical large data warehouse implementation can take upward of a year before any business user will see any viable report coming out of it. During this time, an organization can easily collect another billion data points while encountering new business challenges. My advice: Take raw data from a specific, focused business area and then apply analytics to that one area. Pulling actionable insights from the data to understand "Why?" and "What next?" can shorten this time to weeks rather than months.

For example, in another of my projects, we found 30 out of 500,000 customers who were at high risk of leaving. That's a small number, but the loss of those customers would have meant a loss of about $4 million dollars. If the company had waited for a completed data warehouse implementation, this insight would have been missed. The diagram below explains how a data warehouse implementation (top section) and analytical development (bottom section) can work in parallel and simultaneously provide a real-time flow of intelligent data from analytics into data warehouse.

Big Data, Smarter, Better

5. Data Visualization Is The Key.

It's typical for an analyst who has been working on a project for more than two months to show all the frequency or statistical results with a presentation deck consisting of hundreds of slides. Stop! A few charts with great data visualization are worth 1,000 slides. Actionable visualizations such as Price or Attrition Alerts can help sales teams better engage with customers instead of analyzing a plethora of reports. The key: reports should be easy to understand as well as recommend the next actionable step for business leaders. Condensing piles of data to just a few charts is a balancing act of art and science. The visualization should narrate what the next short-term actions should be in order to improve the business outcome.

Distribution of Margins

Shankar is a managing partner and head of the Retail practice at Kalvin, a Cincinnati based data-mining analytics firm. During his 17+ years of business analytics experience, Shankar has designed and delivered large scale data-mining customer analytics for variety of large firms including Kroger, Home Depot, Best Buy, Limited Brands, Panera, Red Roof Inn, Coke, P&G, General Mills and Kraft.