Surround the Warehouse: Prediction for 2009

The data warehouse has been positioned as the sole source of analytical data in organizations, but that is changing. Rather than trying to remodel the data warehouse to accommodate fresher and more detailed operational data (near real-time activity in operational systems, process logs, etc.), these data sources will operate in parallel (or horizontally, whichever word you like) as complementary feeds to analytics. It takes too long and is too expensive to expand the data warehouse concept to do this.

BI tools like Microstrategy have to retool to be able to query multiple sources to satisfy a single query (they are doing that in the upcoming release 9, I believe). All of the other BI vendors will do the same.However, like BAM/CEP, how does all of this real-time capture deal with the problem of data quality? The promoters of the concept aren't being very forthcoming about that, but IBM, Informatica, Expressor, Talend, etc. are addressing it.

There will be a lot of talk about this and the use of unstructured content, too, but I don't see that happening in '09. Things are too shaky and organizations will be focusing on costs. There will be funding for perceived experts and leaders.

So I think the significance of this trend will be a move toward a broader data warehouse concept that includes data sources that are only connected to the data warehouse through some sort of metadata - hopefully something more semantic, such as OWL. This is very different from an EII-like composite, which still requires persistent views and hand-coded SQL.

I only expect babysteps in '09, but it is the beginning of a trend. I've been barking about decisions, not just analytics, for a couple years. Tom Davenport has begun to do the same, and also I noticed the other day that Claudia Imhoff has moved her topic from Operational BI to Decision Intelligence.

There will be a lot of noise about the Cloud and x-aaS (SaaS, PaaS, DaaS, etc.), but I don't see that gaining as much actual revenue as it will airtime. Oracle did sell $100m in packaged analytical apps last year, but not in the on-demand space.

There is always a lot to write about when looking at the early-adopter space, but when you look at the whole BI landscape, things change very slowly. One of my former clients has thousands of reports (almost all written and maintained by a centralized IT group) written in a BI tool that is versions behind, against a shaky data warehouse on Oracle two versions behind. They're not going to change anytime soon. We talked about a migration plan a few years ago, but they haven't started. The only thing that will get them off a dime is something like an organizational Katrina.

Many organizations are in the same boat. They may see the potential utility of revivifying their data warehouse strategy, but they just don't know how to do it. So despite all of the innovative and useful things the industry comes up with, the data warehouse legacy is like a big boat anchor.

And that's why I see the "surround strategy" of augmenting BI with operational data while leaving the data warehouse in place as the best opportunity right now.The data warehouse has been positioned as the sole source of analytical data in organizations, but that is changing. Rather than trying to remodel the data warehouse to accommodate fresher and more detailed data (operational systems, process logs, etc.), these data sources will operate in parallel as complementary feeds to analytics. It takes too long and is too expensive to expand the data warehouse concept to do this.