Big Data Meets Trough Of Disillusionment: Gartner - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management // Software Platforms
08:06 AM
Connect Directly

Big Data Meets Trough Of Disillusionment: Gartner

Gartner says cloud, in-memory databases are headed into Hype Cycle "trough of disillusionment." Productivity expected within two to five years.

Cloud computing and in-memory databases, two darlings of the big data movement, have passed the "peak of inflated expectations" and are headed into the "trough of disillusionment," according to Gartner's Hype Cycle for Big Data, 2013.

Hype Cycles are Gartner's way of communicating the degree of hyperbole versus productivity associated with emerging technologies, with the peak and trough being followed by a "slope of enlightenment" and progressing to "plateau or productivity" as technologies mature. In the case of cloud and in-memory databases, the plateau is two to five years away, according to research discussed at last month's Gartner Symposium.

In the case of cloud, compute capacity and storage in the cloud are thought of by many as cost-saving options, but "organizations are a little disappointed about perceived cost versus reality," Gartner analyst Nick Heudecker told InformationWeek. "We're starting to see some pullback with organizations realizing that they may not be saving that much money."

That could be because moving anything approaching big data scale into the cloud involves storage and data-movement costs that can add up. Heudecker cited the case of the streaming service Spotify, which recently opted to move its cloud deployment on-premises.

[ Looking for the inside edge on big data breakthroughs? See What's The Best Path To Big Data Insight? ]

"Cloud makes sense when you can't predict your load or demand, but once you have a very predictable workload and you can easily estimate what kind of resources you're going to need, the cloud may not be as attractive," Heudecker said.

Spotify's example matches the experience of Rob Rosen, big data field lead at Pentaho, who said he often sees cloud as the place where big data practitioners start. "The cloud is a great place to do prototyping and early architecture work, but once they succeed and start to scale up, they start to see monthly bills that make them say, 'hey, wait a minute; maybe it's time to bring this in house,'" Rosen said.

On the in-memory database front, Heudecker said the technology has captured a lot of attention, but now questions are emerging as people look beyond the touted benefits of fast performance, simplified infrastructure, and potentially smaller data footprints and power requirements. The questions are popping up about support for disaster recovery and high availability, availability of database tooling, and the impact on application development teams.

"Right now in-memory technology is adolescent," said Heudecker. "We forecast that it will be two to five years before it reaches the plateau of productivity -- perhaps longer, depending on the database."

SAP's Hana platform is perhaps the best known in-memory database platform, but others include the Aerospike database, in use for several years mostly in online advertising; IBM's BLU acceleration feature for DB2, introduced earlier this year; and Microsoft SQL Server 2014, an update set for release in the first half of next year that will include an In-Memory Online Transaction Processing option that's now in public beta. Oracle preannounced an in-memory option for Oracle database, but beta and GA release dates have not been announced.

Other technologies plotted on Gartner's Hype Cycle for Big Data include Hadoop SQL interfaces and the Internet of Things now headed for the Peak; social analytics and Hadoop distributions headed into the Trough; and social media monitoring and predictive analytics now on the Plateau. One final observation from Heudecker is that the Big Data Hype Cycle is going to be particularly dynamic.

"There are a large number of technologies on this Hype Cycle, and the majority of them are clustered around the Peak," he said. "That means we're going to see a lot of change over the next two to five years."

Emerging software tools now make analytics feasible -- and cost-effective -- for most companies. Also in the Brave The Big Data Wave issue of InformationWeek: Have doubts about NoSQL consistency? Meet Kyle Kingsbury's Call Me Maybe project. (Free registration required.)

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
3/31/2015 | 3:23:30 AM
Trough of disillusionment
I think that the idea that ALL technology has this trough is misleading.  However; This time definitely comes where the behaviour moves beyond raw hype.  And we do seem to like a bit of hype (Think IOT!)


In the #bigdata world that time (what they call the trough) has clearly come - and for me as an experienced Pentaho consultant the time is that of genuine implementation. People are now doing REAL bigdata projects, not POCs and extracting real value from the tools. The tools have matured (Hadoop and Pentaho), are now very usable and do a good job.


Obviously not everyone needs it.  But for those that do now is a far more interesting time in bigdata than it was a year ago.


Even if we take that chart as truth, there's only growth to come!
User Rank: Apprentice
2/2/2014 | 3:03:04 PM
Data Strategy is the Answer
You're absolutely right about Big Data being in the "trough of disillusionment". It is, and for good reason: people have been burned by the solutions they've tried.

What's been lacking up to this point is a cohesive data strategy. There needs to be a plan for asking questions, getting answers from the data, and then empowering executives to make data-driven decisions based on that.

I wrote an entire blog post detailing the data strategy that I've seen be successful, you can find it here:

I'm fairly confident that once organizations put a strategy into place to use data effectively that Big Data will be lifted out of the trough once again.
User Rank: Apprentice
11/19/2013 | 11:16:49 AM
Dear Dough Henschen:

I truly enjoyed reading your article on the growing awareness of In-Memory databases. The growth of In-Memory computing and the associated news is music to Altibase's ears! 

Altibase has been perfecting the science of In-Memory databases for nearly 15 years with over 500 enterprise clients and thousands of mission critical deployments.  

Altibase is a major player in the In-Memory DB space as indicated in Gartner's Magic Quadrant for Operational DBMSs 2013.

Please take a look at (see its case studies at as well as comparison docs at Moreover Garter accurately indicated that Altibase's products have  "Broad use case applicability" and "Maturity" in its Magic Quadrant for Operational Database Management Systems.

Please take a look and reach out to me with any quesitons. And I would love to see Altibase included in your upcoming news.


Jung Nahm


User Rank: Apprentice
11/18/2013 | 1:24:55 PM
Good News?!?
Great blog, I had the same feeling in January, maybe I was wrong/early at the time, but, here are some of my fears :

D. Henschen
D. Henschen,
User Rank: Author
11/18/2013 | 10:30:23 AM
SAP: Time to emphasize real-world in-memory gains
SAP is the self-proclaimed leader on in-memory technology, but it has to show it's leading out there in the real world. Let's start hearing more concrete customer success stories and less hype about unspecified "never-before-possible applications" unmeasured "dramatically faster performance" and unverifiable "transformation without disruption." I'm talking to customers and hearing lots of good things, but there are realities (like the cost of new servers) and tradeoffs (like rework required to change apps and analytics) that SAP doesn't acknowledge.

Customers need to know they will get to that "plateau of productivity." Hype will deepen the dissolusionment while real-world case examples will help establish realistic expectations.
10 Ways to Transition Traditional IT Talent to Cloud Talent
Lisa Morgan, Freelance Writer,  11/23/2020
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
Can Low Code Measure Up to Tomorrow's Programming Demands?
Joao-Pierre S. Ruth, Senior Writer,  11/16/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Flash Poll