How to Get Your Big Data Journey Back on Track - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management // Big Data Analytics
09:30 AM

How to Get Your Big Data Journey Back on Track

Moving your big data initiative from proof of concept to production calls for mapping out some key steps.

Big data has been in the hype cycle for a while and Gartner last year identified that the topic is approaching the trough of disillusionment. Factually, many companies are still early on in their adoption of the related architecture patterns and technologies.

Many are building proof of concept systems or have unplanned emerging technological adoption. The challenge they face is the confusing technology landscape and the lack of long standing patterns. Everything appears to be in flux. Nevertheless, some technology savvy companies like Twitter and LinkedIn succeeded in extracting tremendous value and even setting standards like the Lambda Architecture or developing technologies like Kafka.

The challenge for the current adopters is to minimize risk and maximize value. Understanding how many successful big data journeys are structured and how to converge towards such a path best does that. From my experience at Big Data Partnership, the former is distinctively easier than the latter unfortunately. This is especially true once change management and internal organizational dynamics have to be taken into account.

In a greenfield situation the challenge is merely to take a business first and technology second approachand execute along a set of sensible steps:

  • Business goals. They should be up to date and clear.
  • Discover your strategy. How can big data support the business goals?
  • Discover your roadmap and use cases. Plan for the long term and identify short term use cases that are achievable, design an accompanying architecture and products.
  • Develop your use cases. Build proof of concepts to demonstrate value, viability, and solicit buy-in.
  • Deploy your architecture. Productionize your use cases with low risk, employing the learning from developing the proof of concepts.
  • Support and train. Focus on your business by employing professional services, vendors, and support where feasible, and train your staff along the above mentioned path to bring them with you on the big data journey.

The real life is messy and not many have the opportunity to start with a blank slate or resources and time necessary to do so. More often than not some steps are missed. For example:

  1. Business goals are years old, implicit, and expected to not have changed despite the competitive landscape having moved along.
  2. Strategy, surprisingly, for some is an afterthought once someone broke with the accepted technology stack to achieve something unachievable with the traditional stack.
  3. Roadmaps and use cases emerge ad hoc on a need basis and are not aligned with a vision risking investments’ future value.
  4. Architecture is bolted on legacy soft and hardware leading to brittle, inefficient, and unreliable outcomes.

All of these bring the risk to misleadingly conclude big data may be immature or pure hype when often the project was set up for failure. This can reset internal buy-in for years, in the meanwhile competitors take on the lead on new products and insights.

The solution to projects that did not take an ideal planned sequential approach to exploring and adopting big data is to identify the blind spots and address them by priority. Generally this is a top-down approach ensuring goals and strategies are up to date, clear, and aligned first. Next an ideal roadmap with immediate practical use cases and long-term vision should be established.

Then the current work, architecture, or proof of concept has to be put into perspective in this roadmap. At this point painful decisions may have to be made if it turns out that the current work needs realignment. Or it may be that it does fit, and the exercise reveals a clear short-term plan and vision of how to continue the big data journey. At the very least it provides a tool to communicate internally the journey and its benefits to stakeholders for resource allocation and buy-in.

Where are you in your big data journey, and what roadblocks have you encountered?

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
10 Cyberattacks on the Rise During the Pandemic
Cynthia Harvey, Freelance Journalist, InformationWeek,  6/24/2020
IT Trade Shows Go Virtual: Your 2020 List of Events
Jessica Davis, Senior Editor, Enterprise Apps,  5/29/2020
Study: Cloud Migration Gaining Momentum
John Edwards, Technology Journalist & Author,  6/22/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
Key to Cloud Success: The Right Management
This IT Trend highlights some of the steps IT teams can take to keep their cloud environments running in a safe, efficient manner.
Flash Poll